Are there emotions in communication between applications?
Good, standardized and sufficiently descriptive error expressions are essential in the development of business-critical applications. Application errors can arise when applications are miscoded, overloaded, lack hardware resources or are even unreachable. In large and complex IT systems, where several different applications or services are involved in the process, error reporting can quickly become unclear and difficult to manage. The reasons vary from the incremental development of individual parts of the system to the lack of standardised error reporting. However, errors are the key information that we need to improve a system. When errors are incorrectly recorded and communicated to other systems, it is very difficult to find the reasons for their occurrence. If we redirect all the communication between the different systems into a big data management system, we can quickly see that the resulting log records containing errors have a more negative connotation, whereas in the records that do not contain errors we find a more positive connotation. Although it is difficult to attribute emotional states to the apps, the positivity and negativity in the text of the communication between the apps can help to find real errors. In this way, we can, more as a joke than for real, associate a negative or positive sentiment with the current mood of the apps.
How to detect undetected system errors?
The classification of errors is handled by the system whose design is presented here. The presented example is a practical solution that uses about 200 interconnected applications in 4 different environments (production, pre-production and two test environments).
The system was developed using an error classifier and sentiment analysis of SOAP web service responses to give operators a better overview of detected errors. The presented solution is language-independent and works even with mixed languages (tested on a mixture of Slovenian and English), which we have used on the system of one our clients. Wondering how it works? Read on.
How did we develop this successful system?
What is easy for end-users is usually complicated behind the scenes. This is also true for the fault classification system, which we have developed in several segments. The system works by analysing the contents of messages, as different web services use different record formats and structures, which means they are not suitable for easy parsing, and even the use of search strings does not solve the challenge. We have therefore designed a system that does not depend on the structure of the answer, but rather prioritises answers between those with and without error based on sentiment analysis using machine learning. In other words, we have designed a system that detects the valence in the communication between applications, i.e. those that have some negative phrases in their feedback messages. There are several steps to make this happen, but we won't bother you too much with mathematics and programming terms. In short, the solution involves machine learning and several different algorithms that parse, analyse, translate and reinterpret the data from the system, and they do it all extremely quickly.
But what do the results show?
For the machine learning purposes, we used a manually labelled collection of selected errors, which we retrieved from a central data collection system. This collection was the basis for building an algorithm that efficiently checks and classifies the answers from the system with high accuracy. After final validation, we achieved an average weighted recall of 0.977 and an average weighted accuracy of 0.977 based on the labelled answer set and with the help of our algorithm, which in other words means that the system is extremely efficient and most importantly very accurate. What about speed? We processed 750,000 responses from different web services and applications in real time, and additionally detected 718 unique errors, which we manually reviewed and confirmed were responses containing data about different errors.
Are your apps hiding undetected errors?
Our system enables fast and efficient operation. The system can help detect errors in complex and intertwined IT systems, regardless of structure and language, and can be implemented in a company's existing central system. As machine learning is involved, better performance and constant improvement in accuracy requires maximising the amount of data available, which is usually the case for larger companies. Our experimentation is just beginning, and in the next phases we will also test the possibility of using an unsupervised approach to machine learning, which would be another major step forward for us and our clients.