Arrow Web Scraping

Web scraping is a technique used to extract information from websites. For mobile apps, it should be considered a last resort. Instead, try to get access to the underlying data via a documented REST web service API. However, if you find that a REST or SOAP API is not available, you may need to use web scraping in order to get the data you need.

If you are going web scrape, then don’t do it in the mobile app. Use a microservices platform, like Arrow. By implementing the screen scraping in an Arrow middle tier server, when the website changes you can change your scraping algorithm without needing to publish a new version of your mobile app.

This blog post will show a simple example of using Arrow Builder to build an API that utilizes web scraping.

The basic steps are:

  1. Create an Arrow project
  2. Install the x-ray npm
  3. Create a custom API (i.e. not based on a connector or model)
  4. Follow the documentation at x-ray

Create an Arrow project

Execute the following command from the command line:

appc new

Install the x-ray npm

In your project folder, execute the following command from the command line:

sudo npm install x-ray

Create a custom API

In the project /api/ folder, create a new file. My simple example, geta.js, is shown below:

var Arrow = require(‘arrow’);
var Xray = require(‘x-ray’);
var x = Xray();

var GetA = Arrow.API.extend({
    group: ‘xrayapis’,
    path: ‘/api/geta’,
    method: ‘GET’,
    description: ‘this is an api that shows how to web scrape using x-ray npm’,
    parameters: {},
    action: function (req, resp, next) {
        x(‘’, [‘a’])(function(err, a) {
            if(err) {
                resp.send({“error”: “cannot reach url”});
            } else {
                resp.send({data: a});

module.exports = GetA;

Follow the documentation at x-ray

In my example above, I am extracting all ‘a’ tags on the web site using the following:

x('', ['a'])

When I run this API, it returns the following:

  "data": [
    "More »",
    "Web History",
    "Sign in",
    "Get Google Chrome",
    "Advanced search",
    "Language tools",
    "Advertising Programs",
    "Business Solutions",
    "About Google",


In this example, we saw how we can leverage the x-ray npm and Arrow to perform web scraping and expose the data as a mobile-optimized REST API. Furthermore, when the web site changes, you can modify your scraping logic in the Arrow API and avoid the need to re-publish a new version of the mobile app.

Previous articleChanging the World: Waste No Food To Hit One Million Meals Delivered!
Next articleWhat’s driving application development in the era of digital business?
Principal Presales Architect II – Leor Brenman has over 12 years’ experience in Enterprise Software and more than 30 years working in the tech industry for companies like Vaultus, Antenna, Appcelerator and Axway. He has a strong technology background and specializes in mobile, API, integration and enterprise applications. Leor has a BS and MS in Electrical Engineering from Rutgers University in NJ, USA.


    • Thanks abada.

      X-Ray is created by the author of cheerio and has cheerio as a dependency and I use request all the time for making web service calls from node. Cleaner than Node’s built in http request.


Please enter your comment!
Please enter your name here