We were learning about Filebeat, a lightweight approach to shipping log files from our team during a demo of the latest version of Streamdata.io. The open source solution from Elastic allows us to ship all the log files across all of the virtual machines we use to scale Streamdata.io for our customers. Anytime we fire up a new instance to scale our data streaming solution, Filebeat is there to ship the log files for the web server to a central location where we can analyze and report upon them.
Streamdata.io uses Filebeat in conjunction with Logstash to store everything, and Kibana to help us analyze and visualize them via a variety of dashboards and custom queries. Filebeat allows us to diagnose problems for customers in real time, drilling down into individual, or groups of calls made to Streamdata.io and via our APIs. Giving us full access to the data stored within each log file, and allows us to analyze and report on a variety of dimensions with a single click using the Kibana dashboard. Which makes log analysis something anyone can do, even our business-focused employees who wouldn’t normally get their hands dirty with log files.
Using Kibana we are working to deploy useful visualizations accessible through a variety of dashboards, keeping the staff at Streamdta.io in tune with every part of our operations. We are also working to define, rollout, and embed more visualizations into the portal for our end-users, providing them with more real-time data on their own usage. So that our customers can analyze, visualize, and report on the APIs they are proxying, and the real-time data streams and subscriptions they are developing with our solution. Eventually, maybe this is something we can extend for our customers, allowing them to extend this awareness to their own subscribers.
With Filebeat, Logstash, and Kibana in place for the core Streamdata.io platform, we are looking at how we can also use for the Streamdata.io API Gallery. Helping us understand how we can report upon traffic across the hundreds of machine-readable repositories we use to drive the API gallery, and help us better understand how users are searching, browsing, and forking different API collections. Filebeat is proving to be an extremely valuable resource for us to understand our platform, by shipping and centralizing our log files, and slicing and dicing the valuable information they contain so that we can make more informed decisions on a day to day basis.
