Hystrix Dashboard with Turbine – Dynamic Service Discovery and Monitor Aggregated Streams

For the last couple of days I have been working on a solution that fits on microservice architecture to monitor real time streams and make service discovery via document based configuration. Finally we planned to implement Netflix’s Hystrix Dashboard for visualization and Turbine for log aggregation. 

In this article I will try to put all the details in one place for this solution to work, we will discuss the below areas

• Download and Configuration of Turbine 
• Troubleshoot errors for Turbine
• Installation and viewing dashboard with Hystrix

In a distributed architecture, undoubtedly there will be a chance to fail some of the service dependencies. Hystrix is a java based library that gives you a way to control the traffic between these services with latency and fault tolerance logic. 

Turbine is an open source tool for aggregating streams of Server-Sent Event (SSE) into a single stream. Both these tools were open sourced by Netflix in Dec,2011. 

Let’s start by installing Turbine on a local server or on any Public Cloud Provider. I will be using a Linux based system for this. 

Download and Configuration of Turbine 

Java8 or later is required and must be configured on the system for Turbine to Work. Turbine setup is available in-term of war which I will be using with Tomcat to run the application. You can also clone the git repo and use gradle to build and run the application.

Step 1: Download and Configure Tomcat Version 9

Download Tomcat from the below link and extract Tomcat War and perform the required configuration for the application server to work. 
Download Tomcat War 

Step 2: Download Turbine war 

Download and Place Turbine war inside $CATALINA_HOME/webapps 
Download Turbine War 
Start tomcat for the deployment to complete.

If you do not want to run turbine on tomcat, you can run turbine with gradlew as well with the below commands

Step 3: Configure Turbine

We will be using document based configuration for this article. Turbine also works well with the Eureka plugin for service discovery. We need to configure the below items in the configuration file.

• Cluster name
• Cluster Stream url
• Cluster hostname and port 

Turbine configuration file will be found on the below path

$CATALINA_HOME/webapps/turbine-web-1.0.0/WEB-INF/classes/config.properties

Look for this line in the configuration file to add your cluster name. For multiple clusters, the names need to be specified as comma separated values. (No “ “  are required)

Now we will specify the path to access the stream for a cluster

Specify the cluster details for turbine to get the streams 

A global configuration can also be specified with port number and path for all streams

Global Config: turbine.instanceUrlSuffix=:8123/hystrix.stream

That’s all the configuration you need to do and keep the rest as it is, save the file and exit. 

Please make the URL for your cluster in same format and keep it somewhere as we will be needing this url in hystrix dashboard to view the graphs. To verify the stream data, start tomcat and browse the url with the stream path you specified in the configuration file. For example, stream for ‘ myFirstCluster ’ can be seen with below url 

localhost:<tomcat port>/turbine.stream?cluster=myFirstCluster

If you can see the data in the browser HURRAY !!! We have successfully completed this part of configuration. If data is not coming then check the below points

• Tomcat catalina logs, if there is any deployment failure
  tail -f $CATLINA_HOME/logs/catalina.out

• Check on which port tomcat is listening (By default tomcat listens on port 8080 )
  netstat -tunlp | grep tomcat  

• Check if the application is sending data on the required port
  tcpdump -A port 8123  (This is to check for myFirstCluster, Please change the port accordingly )

• Check the port mapping and firewall configurations if you are using any cloud provider or keeping this application under any proxy server.

Installation and viewing dashboard with Hystrix

We can download a war file of Hystrix to run with Tomcat and this can also be built and run with Gradle. This time we will use gradle to show how it works. Hystrix by default works on port 7979.

Step 1: Configure Hystrix Dashboard

$ git clone https://github.com/Netflix/Hystrix.git

$ cd Hystrix/hystrix-dashboard

$ ./gradlew appRun

> Building > :appRun > Running at http://localhost:7979/hystrix-dashboard

How to change a port and bind Hystrix dashboard with an IP ?

Go to hystrix-dashboard/ and open this file build.properties and scroll till bottom. Change the host and port in this below section in the file. 

gretty {

    host = “10.10.10.10”

    httpPort = 7979

    servletContainer = ‘jetty9’

}

A jar file is also available which can be downloaded and run with jar 

https://github.com/kennedyoliveira/standalone-hystrix-dashboard

java -jar -DserverPort=8080 -DbindAddress=192.168.1.100 standalone-hystrix-dashboard-{VERSION}-all.jar

Once the hystrix dashboard starts, please visit the url http://localhost:7979/hystrix-dashboard (change localhost with IP if needed) and you can see the dashboard up and running. 

Now you need to add your streams to see the graphs, for this enter the url we got after turbine setup, give a name to your dashboard for that cluster and click on ‘ Add Stream ’ . You can add multiple streams at a time. Once the stream url is added in the list click on ‘ Monitor Streams ’ to view the graphs. 

Cheers !!

If you face any difficulty during this setup or if there is any ambiguity in the steps mentioned here, please drop a comment in the comment section. 

If you want us to build this solution for you or for your company please visit us on www.hybridskill.com . 

References:

https://github.com/Netflix/Turbine

https://github.com/Netflix-Skunkworks/hystrix-dashboardhttps://netflixtechblog.com/hystrix-dashboard-turbine-stream-aggregator-60985a2e51df

• Hystrix Dashboard
• Hystrix with Turbine
• Turbine aggregator