Setup External Webhook Subscriptions For Stack Overflow Using Server-Sent Events (SSE)

The Stack Exchange API doesn’t provide you with webhooks for subscribing to changing events that happen via their network of sites, including the popular developer destination Stack Overflow. We saw this as an opportunity to demonstrate how you can setup external webhooks for the API using Server-Sent Events (SSE). Demonstrating how anyone can turn on webhook subscriptions for any external API, using real-time, event-driven infrastructure like SSE.

For this example, we are using PHP, but we are just initiating a long-running HTTP call, and posting incremental updates using another HTTP call via POST, something that could be accomplished in any language. The Stack Exchange API doesn’t require authentication by default, reducing friction at integration time, making it a pretty easy API to work with. When you combine the friction-less integration with the amount of content available on the platform, you have the makings for a pretty rich set of real-time streams, and topical webhook subscriptions.

In our working example, all you need is a Streamdata.io account, and key, which takes about 3 minutes to setup, and the following PHP code will let you turn on a stream, and post the results to any URL using a webook:

// Set Streamdata Token
$streamdata_token = '';

// API URL for Stack Exchange Tagged Kafaka
$api_url = 'https://api.stackexchange.com/2.2/questions?order=desc&sort=activity&site=stackoverflow&tagged=kafka';

// Stream URL for Streamdata.io proxying Stack Exchange API
$stream_url = 'https://streamdata.motwin.net/' . $api_url . '&X-Sd-Token=' . $streamdata_token;

// Initiate Long Running API Call (Stream)
$ch = curl_init($stream_url);
curl_setopt($ch, CURLOPT_WRITEFUNCTION, 'streamFunction');
$result = curl_exec($ch);
curl_close($ch);

// This Function Runs With Each Update
function streamFunction($ch, $data)
{

$bytes = strlen($data);
static $buf = '';
$buf .= $data;

while(1)
{

$pos = strpos($buf, "\n");
if($pos === false){ break; }
$data = substr($buf, 0, $pos+1);
$buf = substr($buf, $pos+1);

if(strlen($data)>25)
{

            // Stream Contents  
            $results = str_replace("data:","",$data);

            // Post to our webhook
            $webhook_url = 'https://example.com/webhook/subscriber';

            $headers = [
                'Content-type: application/json'
            ];      

            $http = curl_init();

            curl_setopt($http, CURLOPT_HTTPHEADER, $webhook_url);
            curl_setopt($http,CURLOPT_URL, $webhook_url);
            curl_setopt($http,CURLOPT_POST, count($results));
            curl_setopt($http,CURLOPT_POSTFIELDS, $results);

            $webhook_response = curl_exec($http);
            $webhook_http_status = curl_getinfo($http, CURLINFO_HTTP_CODE);

            curl_close($http);

}
}

return $bytes;

}

In this example, we are just posting the incremental updates to a single webhook subscription URL, but in the real world you will probably be looking up many subscriptions to the same topic, and POSTing the results to all webhook subscription. We are using a POST to publish the results, but you could easily send via email, SMS, or other push notification, going beyond a basic webhook. The goal is to demonstrate how any developer can set up a stream from any API, and turn that into a series of webhook subscriptions–even if the platform provider doesn’t provide webhooks.

Not all API providers have begun investing in their event-driven architecture, and do not offer webhooks, streaming APIs, and other real-time options. This example shows that we do not have to wait for platforms to begin investing in their event-driven architecture, and we can start building it from the outside-in, hopefully demonstrating to each platform the potential of webhooks, streaming APIs, and event-driven subscriptions to data that is changing via the platform. Once we demonstrate the benefits, eventually they might just begin baking some of these features directly into their platform and offering event-driven features alongside their existing request and response infrastructure.