Ma-halanobis distance was calculated on the validation data and selected an optimal threshold by maximum f1-score for detecting anomalies. Then, on the test data, everything was calculated again and anomalies were detected using the threshold set on the validation data.
3.5 Evaluation
The correct interpretation of the results depends on the choice of evaluation of the algorithms. The choice of the wrong method can lead to very sad consequences.
In the anomaly detection problem, for example, if you take the method that is usually used for balanced classes in classification, you can greatly err in the results and mislead the scientific community. Evaluation, usually used in classification, shows the accuracy of a particular class. However, such an evaluation only makes sense if the number of elements in each class is balanced. Otherwise, for example in the anomaly detection, in which the number of elements of one class is about one percent of the number of elements of another class, this will lead to the fact that the accuracy of your algorithm will always be about one hundred percent.
3.5.1 Metrics
The metrics used to evaluate the implemented algorithms were chosen taking into account the particularity of the task and data. The confusion matrix is a table of four elements: true positive (TP), false negative (FN), false positive (FP), and true negative (TN). Usually, it is used when there are only two classes. True positive and true negative shows the number of correctly classified elements of the positive and negative classes. False negative and false positive indicates the number of algorithm errors. This matrix is best suited for this task. Using it, the following metrics were calculated:
• True Positive Rate (RT P) also known as recall shows the ability of the algorithm to detect leakage when it exists.
RT P = T P T P +F N
• True Negative Rate (RT N) also known as specificity shows the ability of the algorithm to avoid false alarms when there is no leak.
RT N = T N T N+F P
• F1-score - a frequently used metric when there are imbalanced classes in the data. It balances precision and recall of classifiers.
Ftp,f p = 2T P
2T P +F P +F N
3. Realisation
3.6 Comparing methods
For a more accurate result, all algorithms were evaluated on a test dataset, which consisted of eight scenarios without anomalies and two with anomalies. As ex-pected, the benchmark showed the worst result, the classification recurrent neural network showed a little bit better result. One-class SVM and Isolation forest showed a very good result, while not taking into account the time component. In the first place, as expected, was lstm with Mahalanobis distance. It is also worth noting that proper data separation and sampling have a huge role in the result.
All results are shown in the table below.
Detectors Ftp,f p (%) RT N (%) RT P (%)
MNF 17.66 48.74 71.58
One-class SVM 28.92 68.98 82.28
Isolation forest 37.44 84.25 68.26
LSTM 18.86 50.44 69.24
LSTM + Mahalanobis 57.70 97.56 64.58
Table 3.1: Detectors and their score
Leak detection example using lstm with Mahalanobis using water pressure (Figure 3.7).
Figure 3.5: Mahalanobis distance based on pressure 24
3.6. Comparing methods
Figure 3.6: Actual test leaks
Figure 3.7: Predicted test leaks
Chapter 4
Conclusion
In this work, several methods for detecting anomalies and their comparison were shown. As practice has shown, one of the most difficult stages was the correct interpretation of the data as well as the creation of a dataset using LeakDB.
Although the results did not turn out to be as good as expected, however, all the algorithms work better than the benchmark. All the goals set at the beginning of the work were fulfilled. In the future, to improve the results in this problem, it is worth trying to use all the data and not just some of it. A longer training of neural networks and different architectures can also help. It can be also tried using convolutional neural networks, as well as autoencoders. Expanding the dataset using real data also can be helpful. It is worth warning that on real data, these algorithms in this configuration may not work, because in the real world many factors cannot be simulated. Also in the real world, there is a human factor in front of which almost any algorithm is powerless.
Bibliography
[1] Chandola, V.; Banerjee, A.; et al. Anomaly Detection: A Survey. ACM Comput. Surv., volume 41, no. 3, July 2009, ISSN 0360-0300, doi:
10.1145/1541880.1541882. Available from: https://doi.org/10.1145/
1541880.1541882
[2] Chen, Z.; Brown, E. N. State space model. Scholarpedia, volume 8, no. 3, 2013: p. 30868, doi:10.4249/scholarpedia.30868, revision #189565.
[3] Box, G. E. P.; Jenkins, G. Time Series Analysis, Forecasting and Control.
USA: Holden-Day, Inc., 1990, ISBN 0816211043.
[4] Winters, P. R. Forecasting Sales by Exponentially Weighted Moving Av-erages. Manage. Sci., volume 6, no. 3, Apr. 1960: p. 324–342, ISSN 0025-1909, doi:10.1287/mnsc.6.3.324. Available from: https://doi.org/
10.1287/mnsc.6.3.324
[5] Alpaydin, E. Introduction to Machine Learning. The MIT Press, second edi-tion, 2010, ISBN 026201243X.
[6] Rousseeuw, P. J.; Driessen, K. V. A Fast Algorithm for the Minimum Co-variance Determinant Estimator. Technometrics, volume 41, no. 3, Aug.
1999: p. 212–223, ISSN 0040-1706, doi:10.2307/1270566. Available from:
https://doi.org/10.2307/1270566
[7] Dadi, H.; Venkatesh, P.; et al. Tracking Multiple Moving Objects Using Gaussian Mixture Model. International Journal of Soft Computing and En-gineering (IJSCE), volume 3, May 2013: pp. 114–119, ISSN 2231-2307.
[8] Moon, T. The expectation-maximization algorithm.Signal Processing Mag-azine, IEEE, volume 13, 12 1996: pp. 47 – 60, doi:10.1109/79.543975.
[9] Sch¨olkopf, B.; Williamson, R.; et al. Support Vector Method for Novelty Detection. In Proceedings of the 12th International Conference on Neural
Bibliography
Information Processing Systems, NIPS’99, Cambridge, MA, USA: MIT Press, 1999, p. 582–588.
[10] Liu, F. T.; Ting, K. M.; et al. Isolation Forest. InProceedings of the 2008 Eighth IEEE International Conference on Data Mining, ICDM ’08, USA: IEEE Computer Society, 2008, ISBN 9780769535029, p. 413–422, doi:10.1109/
ICDM.2008.17. Available from: https://doi.org/10.1109/ICDM.2008.17 [11] LeCun, Y.; Bengio, Y.; et al. Deep Learning.Nature, volume 521, 05 2015:
pp. 436–44, doi:10.1038/nature14539.
[12] Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Com-put., volume 9, no. 8, Nov. 1997: p. 1735–1780, ISSN 0899-7667, doi:
10.1162/neco.1997.9.8.1735. Available from: https://doi.org/10.1162/
neco.1997.9.8.1735
[13] Fukushima, K. Neocognitron. Scholarpedia, volume 2, 01 2007: p. 1717, doi:10.4249/scholarpedia.1717.
[14] Wen, T.; Keyes, R. Time Series Anomaly Detection Using Convolutional Neural Networks and Transfer Learning. 2019,1905.13628.
[15] Abadi, M.; Barham, P.; et al. TensorFlow: A System for Large-Scale Ma-chine Learning. InProceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, OSDI’16, USA: USENIX Association, 2016, ISBN 9781931971331, p. 265–283.
[16] Chollet, F.; et al. Keras.https://keras.io, 2015.
[17] Pedregosa, F.; Varoquaux, G.; et al. Scikit-Learn: Machine Learning in Python.J. Mach. Learn. Res., volume 12, no. null, Nov. 2011: p. 2825–2830, ISSN 1532-4435.
[18] LeakDB : A benchmark dataset for leakage diagnosis in water distribution networks, Zenodo, July 2018, doi:10.5281/zenodo.1313116. Available from:
https://doi.org/10.5281/zenodo.1313116
[19] McKinney, W.; et al. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference, volume 445, Austin, TX, 2010, pp. 56–61, doi:10.25080/Majora-92bf1922-00a.
30
Appendix A
Acronyms
ARIMA Autoregressive integrated moving average CNN Convolutional neural network
LeakDB Leakage diagnosis benchmark LSTM Long short-term memory RNN Recurrent neural network SSM State space model SVM Support vector machine
Appendix B
Contents of enclosed CD
readme.txt ...the file with CD contents description src...the directory of source codes impl...implementation sources thesis ...the directory of LATEX source codes of the thesis text ...the thesis text directory thesis.pdf ...the thesis text in PDF format