Streams

Become a King with an HelloWorld Kong plugin

| Axway |

This blog post was written with Kong version 0.4.2-xxx. Since then, Kong has made lots of modifications. On top of this Kong plugin article, it may be wise to have a look at their new documentation.

You can also check our other article Streaming Kong API Endpoint Activity.

What is Kong?

Kong is an open-source API proxy based on NGINX, which aims to « secure, manage and extend APIs and Microservices » thanks to plugins oriented architecture. Ok, but what does it actually do?

Well, it helps to pass from a « messy » architecture (left side) to a cleaner architecture (right side):

Architecture without and with Kong

Kong offers some nice plugins, including:

– OAuth 2:0 Authentication
– SSL
– CORS
– Transformation (Request or Response)
– Rate Limiting
– Request Size Limiting

All of these Kong plugins give you the power to transform the requests and responses, to control the access of APIs, to log and measure the APIs calls, etc. Impressive, isn’t it? But what if there is a feature you are missing?

Hey! You can develop your own plugin!

Welcome to our tutorial:  « How to develop a Kong “Hello World” Kong plugin? »!

Let’s develop a “Hello World” plugin

Let’s set up a dev environment!

Kong relies on Cassandra as a database and is developed in Lua. There are several ways to settle your environment:

– either you install Lua and Cassandra on your machine (potentially thanks to Docker images)
– or install a Vagrant image of Kong + Cassandra

I was tempted by the first installation, but finally, I’ve switched rapidly to the second one and must admit I am quite happy with this solution. No need to bother with both installing Cassandra and installing Lua (Lua, Luarocks, etc.).

All of these are installed and already pre-configured for the development. The only thing you need is to clone the Kong repository and plug it to your Vagrant image. Kong’s team has done a great job!

Ok, you also need to install Vagrant and VirtualBox first. If you are under Linux, it shouldn’t be a pain, anyway. If you are under Win… ouch, I have no idea but in any cases, all my wishes with you; we never know… 🙂

The readme for the Vagrant image of Kong is well written (short but with all the command lines to execute. All that I love :)!  I will just give you the link and let you prepare your environment before jumping into the development: https://github.com/Mashape/kong-vagrant.

Note: at the time of writing, git was missing in the latest image of Kong-vagrant and I got an error like this:

vagrant@precise64:/kong$ sudo make dev

Missing dependencies for kong:
lua-cassandra ~> 0.3.5-0

Using https://luarocks.org/lua-cassandra-0.3.5-0.rockspec... switching to 'build' mode
sh: 1: git: not found

Error: LuaRocks 2.2.2 bug (please report at https://github.com/keplerproject/luarocks/issues).
/usr/local/share/lua/5.1/luarocks/fetch/git.lua:19: attempt to index local 'version_string' (a nil value)
stack traceback:
/usr/local/share/lua/5.1/luarocks/fetch/git.lua:19: in function 'git_can_clone_by_tag'
/usr/local/share/lua/5.1/luarocks/fetch/git.lua:63: in function 'fetch_sources'
/usr/local/share/lua/5.1/luarocks/build.lua:207: in function 'do_build'
/usr/local/share/lua/5.1/luarocks/build.lua:412: in function 'run'
/usr/local/share/lua/5.1/luarocks/deps.lua:487: in function 'fulfill_dependencies'
/usr/local/share/lua/5.1/luarocks/build.lua:181: in function 'build_rockspec'
/usr/local/share/lua/5.1/luarocks/make.lua:82: in function </usr/local/share/lua/5.1/luarocks/make.lua:51>
[C]: in function 'xpcall'
/usr/local/share/lua/5.1/luarocks/command_line.lua:208: in function 'run_command'
/usr/local/bin/luarocks:33: in main chunk
[C]: at 0x004049c0
make: *** [install] Error 9

Installing git and re-running the previous command fixed the issue:

vagrant@precise64:/kong$ sudo apt-get install git

Now, the Hello World plugin!

The idea is to develop a plugin which will add an Hello-World header to the response of an API call: you add an API of your choice to Kong, add the HelloWorld plugin to this API and when you try to access this API through Kong, you should get a response with an Hello-World header. Nice, isn’t it?

Here is a basic scenario:

# We add mockbin.com to the APIs managed by Kong (mockbin.com is a sample API provided by Mashape)
curl -i -X POST --url http://localhost:8001/apis/ --data 'name=mockbin' --data 'target_url=http://mockbin.com/' --data 'public_dns=mockbin.com'

# We configure the APIs to use our helloworld plugin
curl -i -X POST --url http://localhost:8001/apis/mockbin/plugins/ --data 'name=helloworld'

# We test our API and check that we get our famous header
curl -i -X GET --url http://localhost:8000/ --header 'Host: mockbin.com'

HTTP/1.1 200 OK
Date: Tue, 18 Aug 2015 14:28:35 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Access-Control-Allow-Origin: *
Hello-World: Hello World!!!
Set-Cookie: __cfduid=d6da0fec67be9a0f2138f3a966f5b0ce71439908114; expires=Wed, 17-Aug-16 14:28:34 GMT; path=/; domain=.mockbin.com; HttpOnly
Etag: W/"WjyUny1hiU0eFTCRGSBgnQ=="
Vary: Accept-Encoding
Via: kong/0.4.2
Server: cloudflare-nginx
CF-RAY: 217e4e5578fb1043-CDG

... lots of html blahblah...

Got it? So, let’s start!
In the rest of the article, we assume that $KONG_PATH is the root of the Kong repository you’ve cloned.

First, you need to create at least 3 files under $KONG_PATH/kong/plugins/helloworld:

kong/
├── ...
├── kong/
│    ├── ...
│    └── plugins/
│         ├── ...
│         └── helloworld/
│               ├── access.lua
│               ├── handler.lua
│               └── schema.lua
└── ...

– access.lua: which will contain the code logic that does the job (i.e. adding the header « Hello-World » to the response)
– handler.lua: which will declare our plugin « HelloWorld »
– schema.lua: which will contain the configuration schema of our plugin. This schema will define the different configuration parameters supported by our plugin and their type (string, array, number, etc.). This schema will be used by Kong to store our plugin configuration into Cassandra and validate any update of the configuration

Let’s start from the least interesting to the most.

Edit the schema.lua with this code:

return {
  no_consumer = true,
  fields = {
    say_hello = { type = "boolean", default = true }
  }
}

This tells Kong that our Kong plugin does not handle consumers (see Kong’s doc) and has one boolean configuration parameter: say_hello, whose default value is true.

Then, declare the « HelloWorld » plugin in the handler.lua:

local BasePlugin = require "kong.plugins.base_plugin"
local access = require "kong.plugins.helloworld.access"

local HelloWorldHandler = BasePlugin:extend()

function HelloWorldHandler:new()
  HelloWorldHandler.super.new(self, "helloworld")
end

function HelloWorldHandler:access(conf)
  HelloWorldHandler.super.access(self)
  access.execute(conf)
end

return HelloWorldHandler

So, we have created an HelloWorldHandler which inherits from the BasePlugin which is the basic « class » of a Kong plugin and whose name / id is “helloworld” (cf the new function). The BasePlugin defines several functions that can be overwritten:

– init_worker()
– certificate()
– access()
– header_filter()
– body_filter()
– log()

These functions (at least, access(), header_filter(), body_filter()) are in somehow hooks: they will be called by Kong in a certain order when your Kong plugin will be executed.

Have a look to the access(.) function too. In this function, we execute the super.execute() function and then call the execute() function with the access object as parameter. This object is the one we will declare in the access.lua. What is worth noting is the fact that we pass a conf object to this function. The conf object holds the configuration of our plugin,  and this configuration is stored in Cassandra.

Now, let’s dive in to the most interesting part: the access.lua. In this file, we declare the logic of your plugin, and ours is quite complicated:

local _M = {}

function _M.execute(conf)
  if conf.say_hello then
    ngx.log(ngx.ERR, "============ Hello World! ============")
    ngx.header["Hello-World"] = "Hello World!!!"
  else
    ngx.log(ngx.ERR, "============ Bye World! ============")
    ngx.header["Hello-World"] = "Bye World!!!"
  end
end

return _M

I’ve used the notation/conventions from the other plugins. You declare a local object _M, some functions attached to it and return this object.
Note the ngx object: it’s a global object you can use there in any functions without passing it as argument.
The ngx.header[.] enables to set a new header to the http response. Depending on the value of say_hello of the plugin configuration, we will either return an Hello-World header with the value “Hello World!!!” or “Bye World!!!”.
I’ve also added a log to check that the plugin has been executed. I’ve set the log level of the message to ERROR just to be sure the log entry will be recorded in some log files. I’ve assumed Kong was at least configured to display error messages … and it is 🙂 .

Let’s register our Kong plugin into the Kong platform

To do so, edit the file kong-xxx-yyy.rockspec (at the root of the project). At the time of writing, I’ve used the version 0.4.2.1 of Kong, so mine is kong-0.4.2-1.rockspec.
All you have to do is to declare our plugin lua files in the module section:

package = "kong"
version = "0.4.2-1"
supported_platforms = {"linux", "macosx"}
...
build = {
  type = "builtin",
  modules = {
    ...

    ["kong.plugins.ip_restriction.handler"] = "kong/plugins/ip_restriction/handler.lua",
    ["kong.plugins.ip_restriction.init_worker"] = "kong/plugins/ip_restriction/init_worker.lua",
    ["kong.plugins.ip_restriction.access"] = "kong/plugins/ip_restriction/access.lua",
    ["kong.plugins.ip_restriction.schema"] = "kong/plugins/ip_restriction/schema.lua",

    ["kong.plugins.helloworld.handler"] = "kong/plugins/helloworld/handler.lua",
    ["kong.plugins.helloworld.access"] = "kong/plugins/helloworld/access.lua",
    ["kong.plugins.helloworld.schema"] = "kong/plugins/helloworld/schema.lua",

    ["kong.api.app"] = "kong/api/app.lua",
    ["kong.api.crud_helpers"] = "kong/api/crud_helpers.lua",
    ["kong.api.route_helpers"] = "kong/api/route_helpers.lua",

    ...
  }
  ...
}

Let’s configure the kong.yml

It’s the file used to generate the Kong configuration file at runtime kong_DEVELOPMENT.yml (this file is generated when performing a sudo make dev in the shell of the Vagrant image). Edit the $KONG_PATH/kong.yml and add the HelloWorld plugin in the plugins_available section:

## Available plugins on this server
plugins_available:
  - ssl
  - keyauth
  - basicauth
  - oauth2
  - ratelimiting
  - tcplog
  - udplog
  - filelog
  - httplog
  - cors
  - request_transformer
  - response_transformer
  - requestsizelimiting
  - ip_restriction
  - mashape-analytics
  - helloworld

  ...

Let’s unit test!

Ok. Now, we have our plugin settled. It’s time to test it! In TDD, we should have written the test first, but, you may not practicing TDD and this is a short tutorial, so please, be indulgent!

First, you need to create a helloworld folder in the $KONG_PATH/spec/plugins directory and create an access_spec.lua in it:

kong/
├── ...
├── spec/
│    ├── ...
│    └── plugins/
│         ├── ...
│         └── helloworld/
│               └── access_spec.lua
└── ...

Edit the access_spec.lua as follow:

local spec_helper = require "spec.spec_helpers"
local http_client = require "kong.tools.http_client"

local STUB_GET_URL = spec_helper.STUB_GET_URL
local STUB_POST_URL = spec_helper.STUB_POST_URL

describe("HelloWorld Plugin", function()

  setup(function()
    spec_helper.prepare_db()
    spec_helper.insert_fixtures {
      api = {
        {name = "tests helloworld 1", public_dns = "helloworld1.com", target_url = "http://mockbin.com"},
        {name = "tests helloworld 2", public_dns = "helloworld2.com", target_url = "http://mockbin.com"}
      },
      consumer = {
      },
      plugin_configuration = {
        {name = "helloworld", value = {say_hello = true }, __api = 1},
        {name = "helloworld", value = {say_hello = false }, __api = 2},
      }
    }

    spec_helper.start_kong()
  end)

  teardown(function()
    spec_helper.stop_kong()
  end)

  describe("Response", function()
     it("should return an Hello-World header with Hello World!!! value when say_hello is true", function()
      local _, status, headers = http_client.get(STUB_GET_URL, {}, {host = "helloworld1.com"})
      assert.are.equal(200, status)
      assert.are.same("Hello World!!!", headers["hello-world"])
    end)

    it("should return an Hello-World header with Bye World!!! value when say_hello is false", function()
      local _, status, headers = http_client.get(STUB_GET_URL, {}, {host = "helloworld2.com"})
      assert.are.equal(200, status)
      assert.are.same("Bye World!!!", headers["hello-world"])
    end)
  end)
end)

Kong comes with some predefined tooling to test a plugin. As you can see, we declare some fixtures. This fixtures configures the targeted APIs, the plugin itself, the consumers (if there are) and so on. Then, we write our tests. Classical. Not much left to say…

Then, execute the tests of the plugins by running in the shell of the Vagrant image:

vagrant@precise64:/kong$ sudo make test-plugins

This should execute the plugin’s tests. If you wish to execute other tests, please have a look at the $KONG_PATH/Makefile.

Tip 1: if you wish to execute sudo make test, add – helloworld in the plugins_available of the expected result in the tests of statics_spec.lua. Otherwise, the tests of this spec will fail.

Tip 2: testing all the plugins may take time. I’ve added in the Makefile the target:

test-myplugin:
@busted -v spec/plugins/helloworld

and then I run

vagrant@precise64:/kong$ sudo make test-myplugin

Let’s try!

We need first to compile our project. So, let’s connect to our Vagrant image (see https://github.com/Mashape/kong-vagrant) and follow the instructions of the README. At the time of writing, you have to execute:

cd kong-vagrant/
KONG_PATH=$PWD/../kong vagrant up
vagrant ssh
vagrant@precise64:~$ cd /kong
vagrant@precise64:/kong$ sudo make dev
vagrant@precise64:/kong$ kong start -c kong_DEVELOPMENT.yml

Right. We are ready to try our amazing plugin!

Open another terminal and execute:

curl -i -X POST --url http://localhost:8001/apis/ --data 'name=mockbin' --data 'target_url=http://mockbin.com/' --data 'public_dns=mockbin.com'

HTTP/1.1 201 Created
Date: Wed, 19 Aug 2015 16:05:07 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *
Server: kong/0.4.2
curl -i -X POST --url http://localhost:8001/apis/mockbin/plugins/ --data 'name=helloworld'

HTTP/1.1 201 Created
Date: Wed, 19 Aug 2015 16:05:18 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *
Server: kong/0.4.2
curl -i -X GET --url http://localhost:8000/ --header 'Host: mockbin.com'

HTTP/1.1 200 OK
Date: Wed, 19 Aug 2015 16:05:27 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Access-Control-Allow-Origin: *
Hello-World: Hello World!!!
Set-Cookie: __cfduid=d6da0fec67be9a0f2138f3a966f5b0ce71439908114; expires=Wed, 17-Aug-16 14:28:34 GMT; path=/; domain=.mockbin.com; HttpOnly
Etag: W/"WjyUny1hiU0eFTCRGSBgnQ=="
Vary: Accept-Encoding
Via: kong/0.4.2
Server: cloudflare-nginx
CF-RAY: 217e4e5578fb1043-CDG

... lots of html blabla...

Let’s check the logs too. Go to the shell of the Vagrant image:

vagrant@precise64:/kong$ more nginx_tmp/logs/error.log

You should see somewhere a graceful trace:

2015/08/18 06:50:47 [error] 4613#0: *7 [lua] access.lua:6: execute(): ============ Hello World! ============, client: 10.0.2.2, server: _, request: "GET / HTTP/1.1", host: "mockbin.com"

That’s it! Wonderful, isn’t it?
Well, ok. That was a basic HelloWorld tutorial. But this gives you the basic insights to start coding a Kong plugin and become a King… The sources are under the helloworld-plugin branch of our GitHub repo.

I hope you have enjoyed this short tutorial on building a Kong plugin and I hope you will be able to answer this question:

So, who's a happy king?

Takeaways

Takeaway 1

You can try to update the configuration of our plugin to get the “Bye World!” value:

curl -i -X GET --url http://localhost:8001/apis/mockbin/plugins
HTTP/1.1 200 OK
Date: Wed, 19 Aug 2015 17:29:39 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *
Server: kong/0.4.2

{"data":[{"api_id":"7f6108e3-59f8-4af7-ca52-6eb588d7a8e1","id":"5cb8e33e-5d02-4afa-ca4a-616a165df246","value":{"say_hello":true},"enabled":true,"name":"helloworld","created_at":1440001042000}]}

curl -i -X PATCH --url http://localhost:8001/apis/mockbin/plugins/5cb8e33e-5d02-4afa-ca4a-616a165df246 -d 'value.say_hello=false'
HTTP/1.1 200 OK
Date: Wed, 19 Aug 2015 17:30:14 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *
Server: kong/0.4.2

{"api_id":"7f6108e3-59f8-4af7-ca52-6eb588d7a8e1","id":"5cb8e33e-5d02-4afa-ca4a-616a165df246","value":{"say_hello":false},"enabled":true,"name":"helloworld","created_at":1440001042000}

curl -i -X GET --url http://localhost:8000/ --header 'Host: mockbin.com'
HTTP/1.1 200 OK
Date: Wed, 19 Aug 2015 17:30:53 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Hello-World: Bye World!!!
Set-Cookie: __cfduid=dca65729395f775c6550cf1864101a9401440005452; expires=Thu, 18-Aug-16 17:30:52 GMT; path=/; domain=.mockbin.com; HttpOnly
Etag: W/"WjyUny1hiU0eFTCRGSBgnQ=="
Vary: Accept-Encoding
Via: kong/0.4.2
Server: cloudflare-nginx
CF-RAY: 218796c07aee04a9-CDG

... lots of html blabla...

Takeaway 2

If you have more tricky things to do, I advise you to have a look at the source code of the different ready-to-use Kong plugins. You may need additional operations like creating a new table in Cassandra. That’s often the case for plugins that play with authorizations or security. In this case, you may be interested in the JWT pull-request of jasonmotylinski-dowjones. This PR gives a fairly good idea of the files you need to create in such a case.

Takeaway 3

Not all APIs are accessible from the outside. Let’s say you want to test Kong with an API you are developing on your local machine. Here is a configuration for the Kong VagrantFile. Under the other config.vm.network declarations, add:

# see http://stackoverflow.com/questions/16244601/vagrant-reverse-port-forwarding
  config.vm.network :private_network, ip: "192.168.50.4"

This configures the Vagrant image to access its host with the ip 192.168.50.1 (yes, we declare 192.168.50.4 in the file, but the host is accessible at 192.168.50.1. See this StackOverflow topic for more details). If you would like to give a specific name to this ip, then update your /etc/hosts in the Vagrant image:

192.168.50.1    myamazingapi.io

and in your host machine:

127.0.0.1localhost myamazingapi.io

Read about navigating the new streaming API landscape.

Click Here
**Original source: streamdata.io blog