Speed up your workflow with a Gulp plugin

Discover how you can build a plugin with Gulp and extend it to meet your specific needs


Speed up your workflow with a Gulp plugin

We should have some ways of connecting programs like [a] garden hose – screw in another segment when it becomes necessary to massage data in another way.” This was written on a memo by Doug McIlroy in 1964 and it’s still true today.

The principle makes sense when applied to a build system, which is a series of automated commands that are run either each time you save a file or before you push to a production environment. Gulp can be used to run tasks like image optimisation as well as CSS and JS minification. It’s similar to Grunt but prides itself on its ease of use and efficiency.

We’re going to write a Gulpfile and a plugin. The plugin will scan a CSS file for images and convert those image references to Base64. Base64 is a way to encode images within a file without linking to it externally. We’ll then use Mocha to test our plugin. There’s already a Gulp plugin that does this, but the one we’ll build bears no resemblance to it.


Create a Gulpfile

To use Gulp we first need a file of instructions called ‘gulpfile.js’. This defines the process and order that our build process will follow. Ours is going to convert our images and then minify the CSS with the gulp-minify-css plugin. Here we’re requiring the modules to use and specifying where our CSS files are.

001    var gulp = require(‘gulp’),
002        gulpBase64 = require(‘./index.js’),
003        minifyCSS = require(‘gulp-minify-css’),
004        paths = {
005        css: ‘client/css/**/*.css’
006    };

Write a task

To specify a Gulp task we pass a string (the ID of the task) and a callback function to run when that task is invoked. Gulp handles passing the files to our plugin through piping. Pipes are a powerful way to chain together different bits of functionality to perform complex operations.

001    gulp.task(‘css’, function() {
002        return gulp.src(paths.css)
003            .pipe(gulpBase64())
004            .pipe(minifyCSS())
005            .pipe(gulp.dest(‘build/css’));
006    });

Running Gulp

We then specify the default tasks to run when the Gulp file is run, in our case it’s simple because we only have one. In your Terminal navigate to the folder and simply run ‘gulp’ to run your Gulpfile. Gulp then displays output as each task is run, letting you know the duration of each task. You can run tasks individually too, with ‘$ gulp css’, for example.

001    gulp.task(‘default’, [‘css’]);
002    $ gulp

Plugin dependencies

We’re going to write our own plugin that will analyse a CSS file, find all the images, check to see if they’re under a certain size and then convert them to Base64. First we’re requiring all of the modules that we’ll use (on the disc these are in a package.json file, so you can run ‘npm install’ to grab them all).

001    var through = require(‘through2’),
002        gutil = require(‘gulp-util’),
003        fs = require(‘fs’),
004        q = require(‘q’),
005        path = require(‘path’),
006        PluginError = gutil.PluginError;

Const and config

We use the ES6 keyword ‘const’ to define the plugin name; ‘const’ tells our program that this value won’t change (it’ll be constant). We then make a ‘config’ object that will store any default configuration details. We currently only have one, the maximum size (in kilobytes) that our images should be. We’ve used kilobytes instead of bytes to make it more usable.

001    const PLUGIN_NAME = ‘gulp-base64’;
002    var config = {
003        limit: 8
004    };

Module export

We point ‘module.exports’ to the main function in our plugin; this is what is exposed when we require this module in other files. Because our configuration object is basic we replace it if ‘options’ exists (ideally you’d do some more rigorous checking!). ‘through’ is a wrapper for streams – Gulp pipes a file to us and ‘through’ handles receiving it.

001    function base64 (options) {
002        config = options || config;
003        return through.obj(function(file, encode, callback) {
004        });
005    }
006    module.exports = base64;

Check null files

Part of the Gulp plugin guidelines states that if a file’s contents can’t be read then we shouldn’t throw an error, we should just pass it through. ‘through’ includes some convenience methods like ‘isNull’ to help us with this. Later on we’ll write a test to ensure this behaviour never changes.

001    if (file.isNull()) {
002        self.push(file);
003        return callback();
004    }

Throwing errors

Files can come in two types: a buffer or a stream. It’s recommended that plugins support both types but if it doesn’t then we emit an error through the ‘gulp-util’ PluginError helper. Plugins shouldn’t throw their own errors, they should always use this method and preface errors with the plugin name.

001    if (file.isStream()) {
002        new PluginError(PLUGIN_NAME, ‘This plugin does not     support streams’);
003    }

Buffer checking

We’re going to be handling buffers in our plugin, so the rest of this program will be within this statement. A buffer is like an array that points to bits of memory to form a whole. This makes them very efficient but something that most JavaScript developers don’t usually have to deal with; thankfully Node makes it all relatively straightforward.

001    if (file.isBuffer()) {
002        // next step
003    }

Convert buffer

A file’s contents are stored as a buffer; to convert this into something that we can read, we call ‘toString’ so that we can treat it as we would any other string. We set our initial starting position, create an empty array to push found images into, and cache the number of images (here we’re just looking for the frequency of ‘url’).

001    var css = file.contents.toString(),
002        lastIndex = 0,
003        images = [],
004        numberOfImages = css.match(/url/g) ? css.match(/    url/g).length : 0;

Find image paths

To extract the path we look for ‘url’ from the last index until there are no ‘url’ matches left. This approach has one main limitation, that it requires images to be wrapped in single quotes. If the path isn’t already converted to Base64 then we push it to our ‘images’ array to process later.

001    for (var i = 0; i < numberOfImages; i++) {
002        var start = css.indexOf(‘url’, lastIndex) + 5,
003            end = css.indexOf(‘’’, start),
004            imgPath = css.substring(start, end);
005        lastIndex = start;
006        if (imgPath.indexOf(‘base64’) === -1)
007            images.push(imgPath);
008    }

Making a promise

‘readFiles’ is quite complicated because we now want to loop through each of our found image paths, read their contents asynchronously, and pass it back only when all the files have been processed. To achieve this we’re going to use a popular promise library called Q. If there are no images then we immediately resolve the promise.

001    var readFiles = function () {
002        var deferred = q.defer();
003        var numberOfProcessedImages = 0;
004        if (!images.length) {
005             deferred.resolve();
006        }
007        // next step
008    };

Reading file stats

Within the loop we create a closure to maintain the value of ‘i’. ‘fs.stat’ is a Node filesystem method for getting file stats, we do this because we want to know the size of the file without loading the entirety of it into memory. If the file is not found then we’ll show a message in red to let the user know.

001    for (var i = 0; i < images.length; i++) {
002        (function (i) {
003            fs.stat(images[i], function (err, data) {
004                numberOfProcessedImages++;
005                if (err) {
006                    gutil.log(gutil.colors.yellow
(‘”’ + images[i] + ‘” was not found.’));
007                } else {
008                    // next step
009                }
010            });
011        )(i);
012    }

Create read stream

If the file is small enough then we’ll create a read stream. No matter what size the user specifies, it must be below 32kb otherwise IE 8 won’t display it. Each chunk of data that we receive is pushed to an array, which will contain all of the chunks, so we’ll eventually have the entire file.

001    if (data.size < config.limit * 1024 && data.size 
< 32768) {
002        var all = [];
003        fs.createReadStream(images[i])
004            .pipe(through.obj(function (chunk, enc, cb) {
005                all.push(chunk);
006                cb();
007        }))
008    }

Convert to Base64

To join all of the buffers together we must use Buffer’s ‘concat’ method. To convert this image data to Base64 we simply call ‘toString’, passing the encoding that we’d like. We then find where the image is in ‘css’ and replace it with the encoded version. If we have processed all of the images, then we resolve the promise at this point too.

001    .on(‘data’, function (data) {
002        //push chunk of data to array
003        all.push(data);
004    })
005    .on(‘end’, function () { //once all of file read
006        var base64 = Buffer.concat(all).toString(‘base64’);
007        var ext = path.extname(images[i]);
008        css = css.replace(images[i], ‘data:image
/’ + ext + ‘;base64,’ + base64);
009        if (numberOfImages === numberOfProcessedImages) {
010            deferred.resolve();
011        }
012    })

Print to console

If our image file’s size is too large then we print a message to the console to inform the user. We could add a verbose flag to our plugin to allow users more control over what they see; part of the Gulp philosophy is code over configuration – ‘gulp-util’ includes ways to easily colour-code messages.

001    } else {
002        gutil.log(‘”’ + images[i] + ‘
” was too large to encode.’));
003    }

Return promise

At the end of ‘readFiles’ we return a promise to let the plugin know not to carry on with the other Gulp processes while we run our asynchronous file reads. This is important as otherwise our plugin will carry on potentially writing to a file when other plugins are trying to do the same.

001    return deferred.promise;

Call callback

Once all of the files are read and the promise is resolved the ‘then’ callback is called. To write the new contents to our CSS file we can convert our ‘css’ string back to a buffer and simply assign ‘file.contents’ to it. We then return the callback to let ‘through’ know to carry on.

001    readFiles().then(function () {
002        file.contents = new Buffer(css);
003        self.push(file);
004        return callback();
005    });

Test dependencies

Plugins must be testable, so we write a couple of tests to check our plugin is working. We’re going to use a testing framework called Mocha. Create a folder called ‘test’ and a file called ‘test.js’, then require our dependencies.

001    var assert = require(‘assert’),
002        gutil = require(‘gulp-util’),
003        gulpBase64 = require(‘../index’),
004        fs = require(‘fs’),
005        es = require(‘event-stream’),
006        path = require(‘path’);

Describe test

We describe our module and then functionality within that module. If we supported streaming this would allow us to test in both buffer and stream modes, or whichever major blocks of functionality you decide to test. ‘describe’ is for grouping tests that make sense to put together and you can nest these as deep as you want.

001    describe(‘gulp-base64’, function() {
002        describe(‘in buffer mode’, function() {
003        });
004    });

Create fake file

Our first test makes sure that we’re generating the correct output. The more human-readable your tests are the better, but because we’re dealing with files there’s quite a lot of setup to do beforehand. We’re using the ‘File’ helper in ‘gulp-util’ to create a fake file from the contents of another.

001    it(‘should encode images to base64 and generate a 
css file’, function(done) {
002        var filename = path.join(__dirname, ‘/fixtures/
003        var input = new gutil.File({
004            base: path.dirname(filename),
005            path: filename,
006            contents: new Buffer(fs.readFileSync(filename))
007        });
008        //next step
009    });

Assert contents

Our plugin returns a ‘through’ stream that we can listen to. We use Node’s ‘assert’ capability to ensure the file that is returned from our plugin matches the expected output. ‘styles.css’ within the fixtures folder is a static file that has the image already encoded. When the test has completed we call ‘done’ to fully complete the test.

001    var stream = gulpBase64();
002    stream.on(‘data’, function (newFile) {
003        var file = fs.readFileSync(path.join(__dirname, ‘
/fixtures/styles.css’), ‘utf8’).toString();
004        assert.equal(String(newFile.contents), file);
005        done();
006    });

Write to stream

We’ve added the data event listener but not started writing the fake input file to our plugin. We do this by writing it to the stream with the ‘write’ method, which passes the data to the stream, which triggers the ‘data’ event.

001    vstream.write(input);

Pass null files

Our second test will make sure that our plugin adheres to the Gulp guidelines, which states that files with no contents are just passed along. We can test this by creating a file with no contents and test to see if the data event is triggered. If the file has been processed then we can pass it on.

001    it(‘should ignore files with no content’, function
(done) {
002        var stream = gulpBase64(),
003            n = 0,
004            filename = path.join(__dirname, ‘/fixtures
005    });

Further assertions

We want to ensure three things: that the files are equal, that the contents are ‘null’ and that this has only happened once. The structure for these tests comes from the ‘gulp-cat’ plugin. The Gulp authors encourage learning from other plugins to see how they test but you don’t (and shouldn’t) have to use Gulp within your tests.

001    stream.on(‘data’, function(file) {
002        assert.equal(file.path, filename);
003        assert.equal(file.contents, null);
004        n++;
005        assert.equal(n, 1);
006        done();
007    });

Writing and ending

In the same way we did before, we’re writing a fake file to the stream – only this time we are going to set the contents to ‘null’. Finally, we close the stream with ‘end’, which ensures that no more data can be written to the stream, so the contents should stay as ‘null’.

001    stream.write(new gutil.File({
002        path: filename,
003        contents: null
004    }));
005    stream.end();

Run tests

We can run the tests from Terminal with the ‘mocha’ command. You’ll have to have it installed through npm – if you haven’t then you can install it with ‘$ npm install mocha’. Mocha will then tell you if a test has passed or failed as well as a stack trace if any errors are encountered.

001    $ mocha

Improve your web design skills with Web Designer. Download issues direct from Great Digital Mags or from the ImagineShop