Gaining run-time insights into your distributed IT landscape

For organizations that build their distributed system on a (SOA) service architecture, monitoring would typically be assigned to the Enterprise Service Bus (ESB). Organizations coming from an API centric approach would use their API Gateway. In many cases, the fact that these products offer a monitoring solution would be one of the reasons to invest in them. But here also lies the first problem. As organizations realize they will have to combine different architectural styles (use APIs to expose service functionality, use (micro)services to implement API functionality) they are now faced with 2 monitoring solutions and the question on how to integrate them.

Furthermore, as monitoring is only a small part of the functionality offered by these products, they have some shortcomings.

First, in many cases, the user must use the same general-purpose interface to access the monitoring dashboards and information as any other user of the platform. Given the fact that this is also the interface used by the technical staff to perform their duties (deployment, general management, …) the user experience for these less technical users is far from ideal as there are exposed to complexities that do not relate to their tasks. Also, most of the implementations offer very little support for customization. The solution comes with a set of standard views and reports but it is very difficult to adapt or extend these views.

Second, in order to adequately monitor the assets deployed on these platforms, logging must be activated. These logs usually create a huge amount of data that is not very well handled in these out-of-the-box tools. Processing the log information usually slows down the rest of the system, making it unusable for the other users.

Finally, as this logging is limited to the activities happening on the platform (ESB, API Gateway) they only provide information on what is happening on the platform, so no end-to-end view is possible. A feature that is highly desirable in a distributed IT landscape.

The solution to this problem is rather simple, aggregate all the logging information into one single flexible analytics stack that will analyze and visualize all gathered data. Elastic Stack (formerly known as ELK-stack) is an example of such a solution. Architecturally the setup is very simple (see picture) all log information is push to the analytics stack that will handle the monitoring functionality.

To get the log information into Elastic stack, Logstash is used. The sole purpose of Logstash is to facilitate the task of gathering and transforming log data, and this in a near real-time fashion. Once the data is processed, it is forwarded to Elasticsearch, the search engine component that is built to handle large amounts of data and allows the user to perform analytics on the gathered data. As an interface, the user will use Kibana, the visualization component, which provides the functionality to easily create the different visualizations (dashboards)

By using this platform, we circumvented the afore mentioned shortcomings:

• As all log information is pushed to the Elastic stack by design, Elastic stack is able to create an integrated view and the user is only faced with one interface;
• Since the tooling is aimed at data analysis and visualization it provides a much cleaner user interface and more customization options, and;
• Once the log data is collected in the Elastic stack it can be deleted in the source systems (ESB, API Gateway, Application Servers, …). This will decrease the load on these systems and make them more responsive.

We recently had the opportunity to implement such a solution at a customer. Given the fact the client already had an Elastic-stack operational and we only needed to integrate the log information from their ESB (Webmethods), the implementation was straightforward.

We pushed the log information from the Webmethods ESB to the Elastic stack. We then asked for a list of questions they wanted to answer with the monitoring dashboards. Based on these questions we created some general-purpose visualizations (Line chart showing the evolution of the average response time, Pie chart showing who called a particular service…).

It is important to note that the platform automatically handles the time aspect of these visualizations. So, with the above examples you are able to see the evolution of the average response time in the last hour as well as the last month (provided you have the necessary data).

Once those basic visualizations were done, we created ‘filter lists’. These were simple tables listing all different services, the different operations and the different clients(consumers). In its own these tables provided little added value but linked together with the visualizations they provided a rich user experience.

By selecting an item in these tables, the Elastic platform would automatically only return the data linked to the selection you made. As a consequence, you were now able to, for example, visualize the average response time of operation x on service y for client A for the last day/week/month. All this, by simply selecting the right service, operation, client in the corresponding filter list. Once the dashboards were operational it is important for service providers to start using them.

So, we provided training sessions. This training encompassed more than just monitoring. But it gave us the opportunity to show them how the dashboards could help them in their day-to-day work.