WebCake

Node comes with some pretty awesome features, something that’s been well documented and widely accepted.  These days even some long-time Java devs that I know are pretty excited at how quickly something can be done server-side with Node, and the ES6/2015 features are making a lot of the old-guard naysayers come around.

That being said, one thing I’ve recently been a little disappointed with is the lack of ease with which HTTP requests are written.  It should be noted that this is entirely a result of being spoiled by frameworks like Angular, which make HTTP requests almost too easy. Still, compared to the incredibly simple fs and process modules, the http module takes a lot more thought.

With that in mind, I thought I’d show you how to build an HTTP request client as a Node module, so that you can add it to whatever projects you like. Note that this will be written entirely with built-in Node functionality. It might be easier to do this with installable open-source modules or NPM packages; for me, the motivation was learning how to do it, and sticking with something that works.

A Simple Server

Before we jump into building our request module, let’s first build a simple server to listen for requests, and send a response. This will allow us to demo our HTTP requests. It’s incredibly easy in Node, as it is a backend framework and was built with APIs in mind. If you’re not interested in starting up a listening server, feel free to skip down to the next section.

Wherever it is that you like to write files, create the following file:

// node-server.js

const
    http = require('http');

http
    .createServer((req, res) => res.end(JSON.stringify({
        data: 'Cake, and grief counseling, will be available at the conclusion of the test.'
    })))
    .listen(8000, () => console.log('Listening on port 8000'));

This is a super-simple server file, which will listen at http://localhost:8000 for any incoming requests, and will respond with a simple object. Nothing crazy here. Note that we can use any HTTP verb to hit this endpoint and it will work.

To start your server, navigate in a command line to the directory where your new node-server.js lives, and run:

$ node node-server.js

You should see a console log that says Listening on port 8000, and the process should remain running. If, for whatever reason, you feel the need to close the process, you can hit Ctrl+C to kill it; but for our purposes you can feel free to leave it running.

Sending Requests

Node handles requests in an event-based fashion, meaning that when we create a Node HTTP client, we’re actually creating a short-term event handler that will listen for triggers for the duration of a request cycle. Those of you who can remember the old XHR status checks, they represented the stages of the request lifecycle.

The options object

When we make HTTP requests in Node, we use the http.request() method, which takes two arguments. The second is a response handler, which will discuss in a bit, while the first is a set of config options, passed in as an object. This will be where we direct our request client to make requests to a certain server, path, port, and with a certain content type. You can feel free to configure this however you want, but for now we’ll just go with the basics. It will end up looking something like this:

{
    hostname: 'localhost',
    path: '/',
    port: '8000',
    method: 'GET',
    headers: {
        content-type: 'application/json'
    }
}

If you wanted to hard-code your options object, you can feel free to base it off of that. But hard-coding is super lame. So instead we’ll abstract the construction of our options object. Start a new file called http-request.js, and add the following:

// http-request.js
'use strict';

function Options(method, hostname, path, port) {
    this.hostname = hostname;
    this.path = path;
    this.port = port;
    this.method = method;
}

Options.prototype.headers = {
    'content-type': 'application/json'
}

This will allow us to pass in some data to our constructor in order to end up with a fresh new options object. If you want to make sure it works, you can add this to the bottom:

console.log(new Options('GET', 'localhost', '/', 8000));

Try running the file by navigating to its directory in your command line, and then entering:

$ node http-request.js

You should see Options { hostname: 'localhost', path: '/', port: 8000, method: 'GET' } printed out in the console when the file runs.

The request object

At this point we’ll move on to actually building our reqest, now that we have a configuration process squared away. In this step, we’ll build our module export as a function that returns (implicitly, using an arrow function) a promise that is resolved once the communication between server and client has ended. During the lifecycle of the connection, we’ll listen for data events to be received by the client so that we can add them to our asynchronous return.

In your http-request.js file, add the following to begin the process of setting up a return object. Personally I like to keep business logic at the top, and private functions close to the bottom, so I’m going to add this block between the 'use strict'; declaration, and the Options constructor we just built. You can add yours wherever you like.

// http-request.js

const
    http = require('http');

module.exports = config => new Promise((resolve, reject) => {
    // our http logic will go here...
});

Dynamically building options

So at this point, we’re exporting a function as a module, which takes one argument – a config object. We’ll use that config object to build our options:

module.exports = config => new Promise((resolve, reject) => {
    const
        opts = new Options(config.method, config.hostname, config.path, config.port),
        // more http logic will go here...
});

Already, we can see that our config will hold a lot of data. That shouldn’t be a problem since we can build application configurations to do just that; or if, for example, you’re building an Express app as middleware, you can build config objects based on whatever data you’re receiving in incoming HTTP requests before injecting necessary data to the HTTP request module.

In total, at the very least, our config in this exaple will include the following data:

{
    method: 'string'
    hostname: 'string',
    path: 'string',
    port: 123,
    body: {}
}

So far all of these have been used with the exception of the body field, which we’ll use when we actually send our request to the server.

Building the response handler

Now we finally get to looking at our request object. We’ll create the event handler by invoking the http.request() method, which takes two arguments. The first is an object containing options to direct the HTTP request, while the second is a callback for handling events based on the server-side response. Let’s invoke the http.request() method below the new options object:

const
    opts = new Options(config.method, config.hostname, config.path, config.port),
    req = http.request(opts, res => {
        // we'll build onto our response handler here...
    });

We’ve set up an object saved on the req constant, which will emit events based on the progress of our request lifecycle. That could include connect, abort, error, and upgrade, as well as any other HTTP events that might happen. You can find the full list on the Node docs. It will also be responsible for writing data to the request, and indicating once it has finished constructing the request and is preparing for a response. Our shiny new opts sets up our configuration for all the important details.

We’ve also set up a callback that takes a response object, res, as a parameter. That’ll be our response event handler, which we’ll use to build our return data as it comes in.

Let’s deal with that response handler:

req = http.request(opts, res => {
    let data = '';
    res.setEncoding('utf8');
    res.on('data', chunk => data += chunk);
    res.on('end', () => resolve(JSON.parse(data)));
});

We might have only added four lines, but they do quite a bit. First we set up an empty variable to start storing data as it comes in. Remember, JSON sent over the wire is always sent as a string, so when we add it to our varaible, we’ll be adding bits of string content. Next, we’ve set up a character encoding. You never know what format data will come in, so we have to make sure it’s readable for humans, and not stored as a giant Buffer that only machines can understand.

The last two are the most important. We set up an event handler on to fire on the data event – an event triggered when data is received as part of the response. Notice that it’s specifically set up to handle as many data events as necessary. This is super freaking important, because in the world of HTTP, you can never be sure that data will arrive all at the same time. Sometimes data will arrive in multiple packets, especially if there’s a lot of data being transferred, or if the network in use is extremely busy. So when that data event is triggered, its data packet is added to the local data variable.

FInally, once the response has indicated that its communication is complete and the cycle has ended, the end event is triggered, and we fire a callback that resolves the asynchronous promise with the contents of the local data variable. Notice that we have to parse said contents, since they’re still in the form of a string.

Completing the request cycle

And now, we utimately come to executing the request, as well as indicating to the request even handler that everything we plan to do has been done, and it can move on to listening for responses. Below the block defining req as a local constant, add the following two lines:

req.write(JSON.stringify(config.body ? config.body : null));
req.end();

The write() method will send the request, using the configurations and event handling plan that we’ve just defined. We’re using a ternary to determine if there was a body field in the config, and if not we set it to null. We have to JSON.stringify() our request body since, as mentioned above, data sent over the wire has to be a string. The end() method will let our HTTP request cycle know that nothing more is coming from the client, and that from here on out it’s up to the server to do its thing.

The advantage to using a ternary in determining if the body field was set is that we can then go on to use this module for GET and POST requests alike, or any other request that may or may not send data.

Testing Our Module

At the end, your HTTP request client module should look something like this:

'use strict';

const
    http = require('http');

module.exports = config => new Promise((resolve, reject) => {
    const
        opts = new Options(config.method, config.hostname, config.path, config.port),
        req = http.request(opts, res => {
            let data = '';
            res.setEncoding('utf8');
            res.on('data', chunk => data += chunk);
            res.on('end', () => resolve(JSON.parse(data)));
        });
    req.write(JSON.stringify(config.body ? config.body : null));
    req.end();
});

function Options(method, hostname, path, port) {
  this.hostname = hostname;
    this.path = path;
    this.port = port;
    this.method = method;
}

Options.prototype.headers = {
    'content-type': 'application/json'
}

Get requests

Now we’ll set up a super simple test to make sure it’s working. If it’s not still running, run your simple Node server so that it’s listening on http://localhost:8000. In a separate file in the same directory as your http-request.js where your new module lives, add a file called test-http.js with the following contents:

// test-http.js

'use strict';

const
    request = require('./http-request'),
    config = {
        method: 'GET',
        hostname: 'localhost',
        path: '/',
        port: 8000
    };

request(config).then(res => {
    console.log('success');
    console.log(res);
}, err => {
    console.log('error');
    console.log(err);
});

This will import our module, run a request according to the configured options, and console log either the response, or an error if one is thrown. You can run that file by navigating to its directory in the command line, and typing the following:

$ node test-http.js

You should see the following response:

success
{ data: 'Cake, and grief counseling, will be available at the conclusion of the test.' }

Post requests

Let’s change it up to make sure this module also sends data. Change your config to match the following object:

config = {
    method: 'POST',
    hostname: 'localhost',
    path: '/',
    port: 8000,
    body: {
        cake: false
    }
};

If you run your test-http.js file again, you should get the same results. You can also ensure that POST data is being received by adding a data event-handler to your simple server:

// simple-server.js

const
    http = require('http');

http
    .createServer((req, res) => {

    req.on('data', function(chunk) {
            // this will listen for data events,
            // the same way that the client listens for
            // data events in the response handler
            console.log("Received body data:");
            console.log(chunk.toString());
        });
        return res.end(JSON.stringify({
            data: 'Cake, and grief counseling, will be available at the conclusion of the test.'
        }))
    })
    .listen(8000, () => console.log('Listening on port 8000'));

Remember to restart your server to see changes applied. Once you do, you should now see the request body logged in the console every time you run a POST using the test-http.js file.

Wrapping Up

That’s it, really. Feel free to add this to whatever projects you’re working on. If you have any thoughts or questions, feel free to let me know in the comments below.

Leave a Reply