The Accuracy and Business Value of a Fracta Machine Learning Drinking Water Main Condition Assessment

Posted by Doug Hatler on Mar 26, 2019 4:39:27 PM

The Fracta Machine Learning and GIS approach to condition assessment are new.  Many utilities, public utility commissions and consulting engineers still view this as a “black box,” with the primary concern being accuracy.  The same question repeatedly comes up in any discussion about the use of Fracta and Machine Learning in pipeline condition assessment, “How do you know the prediction is accurate?”  Let us explain why.

Balanced Accuracy for LOF Predictions

The data that comes out of a Machine Learning model is only as accurate as the data that goes into the model (i.e., “garbage in, garbage out”).  Fracta uses a Supervised Machine Learning model with input variables (x) and an output variable (Y).  The model uses an algorithm to learn the mapping function from the input to the output, Y = f(x).  The goal is to approximate the mapping function so well that for new input data (x) the algorithm can predict the output variables (Y) for that data. It is supervised learning because the process of an algorithm learning from the training dataset is similar to a teacher supervising the learning process.  By using a training data set the correct answers are known, the algorithm iteratively makes predictions on the training data and is corrected by the teacher. Learning stops when the algorithm achieves an acceptable level of performance.  Generally, 80% of the historical data are used to train and 20% are used to validate the Machine Learning model.

Assessing a water distribution system with a Machine Learning model requires an understanding of both failures and non-failures. True Positive Rate (TPR) measures the proportion of correctly identified real positives. In the Fracta solution, that means correctly identifying the high probability failures.  True Negative Rate (TNR) measures the proportion of correctly identified actual negatives. This methodology focuses on correctly identifying the segments that have low LOF. The accuracy is a balance between high LOF and low LOF results. Because the training and validation of the model are based on 80% of the data, the maximum Balanced Accuracy that it can achieve is 80%. 

Figure 1 illustrates the concept of Balanced Accuracy for a medium water utility.  The Machine Learning model accurately predicted 79.23% of actual main breaks captured.  79.23% out of 80% is a 99% accuracy.

True Positive Rate and True Negative Rate of Actual Water Main Breaks  Picture2

Figure 1: Case Study:  Accuracy for a Large Utility (3,395 miles) Dataset 2013-2017

Age-Based Model Versus the Fracta Machine Learning Model

Artificial Intelligence/Machine Learning leverages a water utility’s collected data and combines it with over 1,000 different variables to create a more accurate predictive model for calculating the probability of water pipe segment failure. Comparing these two types of models reveal fascinating results, as explained in the analysis for a medium-sized utility.

Fracta ML Model  Age-based Model for a Medium Utility

Figures 1 and 2: Comparison of the Fracta ML Model vs. the Age-based Model for a Medium Utility (847 miles) 2013-2017

The worst 2% of water main pipes, according to the Fracta Machine Learning (ML) model, captured 10.9% of the historical breaks during a time-shifted study, where part of the utility data is held back, and the ML model predicts those “missing” years.

There are 14.8 miles out of the 847 miles (2%) identified in this dataset as being the worst, or highest risk, water pipes using the Fracta ML model.

The combined break and age-based model focused on 2.4% of the network. So, the Fracta ML model (1.9%) was 21% more effective in identifying pipe breaks over the age-based model (2.4%).

  • 9% worst ML model miles captured 10.9 percent of breaks at 14.8 miles
  • 4 miles in a break/age-based model to capture the same amount: 18.7 miles
  • Replacement cost difference: 4 @ $1M /mile or $4M

In this case, if the utility replaced every pipe from the Fracta ML model versus the break/age-based model, they would replace 4 miles less and would have saved $4M ($1M per mile for direct replacement costs) by not replacing pipes that still have good performance - regardless of their anticipated service life.

If the Fracta ML methodology were used to replace every mile to capture 10.9% of breaks, the utility would have been able to eliminate the same number of breaks using 4 miles less pipe at the cost of $1M / mile. 

This utility would need to review or assess 0.5% more of their pipe distribution system, or approximately 4 miles, to find the same percentage of breaks as the Fracta ML model. The Fracta ML model is 21% more effective in identifying pipe breaks over the break/aged-based model.

We can also look at this from a straight replacement strategy. If a utility wanted to replace 2% of the network, no matter the model, how much of the historical breaks are covered, or, how many potential breaks were avoided?

When using the Fracta model to replace 2%, we have captured 11% of the historical breaks or potential future breaks.  If we were to use the traditional break and age-based model, a 2% replacement would have only found just under 10% of the breaks, allowing for other high-risk pipes to fail. This data set included 889 breaks in the addressed. Fracta identified eight more critical breaks than the traditional break and age-based model.  Taking $15,000 as an average cost per break, we have now avoided or saved $120,000. 

Summary

A condition assessment as a fundamental part of asset management assumes that materials or infrastructure components deteriorate, with the goal of gathering information to predict the need for repair, rehabilitation, or replacement.  Advances in information technology and data science provide a fast, accurate and affordable approach to condition assessment with Machine Learning and GIS. Machine Learning brings the next generation of cost savings to water utilities. 

Topics: Capital Planning, Leak Detection, Condition Assessment

Subscribe for more news & updates from Fracta