We are playing around with what is possible when it comes to streaming data from the Stack Exchange API, which powers the popular Stack Overflow website for techies. We want to understand how you can turn the existing web APIs into streams using Streamdata.io, for delivery to dashboards and other applications. We are also learning how to train machine learning models using the streams that we create using the Stack Exchange API, helping make sense of what is happening on the platform at scale. One thing you need to translate existing web APIs into streaming channels, is a set of relevant topics, depending on what mechanisms the API provides. The most relevant parameters you can pass to the Stack Exchange API, are the tags, which reflects the top areas of technology developers are talking about on the platform. Using the Stack Exchange Tag API we pulled the top 30 tags being discussed:
[
{
“count”: 1595517,
“name”: “javascript”
},
{
“count”: 1399832,
“name”: “java”
},
{
“count”: 1198207,
“name”: “c#”
},
{
“count”: 1186605,
“name”: “php”
},
{
“count”: 1092090,
“name”: “android”
},
{
“count”: 933115,
“name”: “python”
},
{
“count”: 906022,
“name”: “jquery”
},
{
“count”: 740908,
“name”: “html”
},
{
“count”: 565053,
“name”: “c++”
},
{
“count”: 558666,
“name”: “ios”
},
{
“count”: 529833,
“name”: “css”
},
{
“count”: 513940,
“name”: “mysql”
},
{
“count”: 435315,
“name”: “sql”
},
{
“count”: 331996,
“name”: “asp.net”
},
{
“count”: 290550,
“name”: “ruby-on-rails”
},
{
“count”: 283552,
“name”: “objective-c”
},
{
“count”: 275445,
“name”: “c”
},
{
“count”: 266763,
“name”: “.net”
},
{
“count”: 260687,
“name”: “arrays”
},
{
“count”: 251034,
“name”: “angularjs”
},
{
“count”: 232862,
“name”: “r”
},
{
“count”: 231116,
“name”: “json”
},
{
“count”: 229471,
“name”: “sql-server”
},
{
“count”: 221087,
“name”: “node.js”
},
{
“count”: 218923,
“name”: “iphone”
},
{
“count”: 193859,
“name”: “ruby”
},
{
“count”: 188809,
“name”: “swift”
},
{
“count”: 186428,
“name”: “regex”
},
{
“count”: 185541,
“name”: “ajax”
},
{
“count”: 171403,
“name”: “xml”
}
]
We are turning this JSON list of tech tags and turning into topic based streams using Streamdata.io. Allowing us to turn on a streaming channel of questions and answers on each topic and publish to the browser, or use to train machine learning models. We are interested in enabling folks to light up dashboards and streams of relevant questions in their browser, but we are also looking to make sense of what is being said at scale on Stack Overflow using machine learning models. Building on top of what is available with the Stack Exchange API, and augmenting and enriching it using Streamdata.io.
It is helpful to publish these as a JSON collection, which allows these topics to be programmatically turned into streams. We will be using these topics to further refine the streams, and once we get a chance to evaluate them individually, we will be publishing other topics and filters for further refining or aggregating streams by. Many of the web APIs we come across like the Stack Exchange API have endpoints available for pulling these tags or topics, which can easily become the programmatic layer for defining the event-driven landscape that exists across any platform. Which represents a pretty huge opportunity for delivering all these changing event types as real-time streams.
Follow us on social