By
We should have some ways of connecting programs like [a] garden hose – screw in another segment when it becomes necessary to massage data in another way.” This was written on a memo by Doug McIlroy in 1964 and it’s still true today.
The principle makes sense when applied to a build system, which is a series of automated commands that are run either each time you save a file or before you push to a production environment. Gulp can be used to run tasks like image optimisation as well as CSS and JS minification. It’s similar to Grunt but prides itself on its ease of use and efficiency.
We’re going to write a Gulpfile and a plugin. The plugin will scan a CSS file for images and convert those image references to Base64. Base64 is a way to encode images within a file without linking to it externally. We’ll then use Mocha to test our plugin. There’s already a Gulp plugin that does this, but the one we’ll build bears no resemblance to it.
Create a Gulpfile
To use Gulp we first need a file of instructions called ‘gulpfile.js’. This defines the process and order that our build process will follow. Ours is going to convert our images and then minify the CSS with the gulp-minify-css plugin. Here we’re requiring the modules to use and specifying where our CSS files are.
001 var gulp = require(‘gulp’),
002 gulpBase64 = require(‘./index.js’),
003 minifyCSS = require(‘gulp-minify-css’),
004 paths = {
005 css: ‘client/css/**/*.css’
006 };
007
Write a task
To specify a Gulp task we pass a string (the ID of the task) and a callback function to run when that task is invoked. Gulp handles passing the files to our plugin through piping. Pipes are a powerful way to chain together different bits of functionality to perform complex operations.
001 gulp.task(‘css’, function() {
002 return gulp.src(paths.css)
003 .pipe(gulpBase64())
004 .pipe(minifyCSS())
005 .pipe(gulp.dest(‘build/css’));
006 });
007
Running Gulp
We then specify the default tasks to run when the Gulp file is run, in our case it’s simple because we only have one. In your Terminal navigate to the folder and simply run ‘gulp’ to run your Gulpfile. Gulp then displays output as each task is run, letting you know the duration of each task. You can run tasks individually too, with ‘$ gulp css’, for example.
001 gulp.task(‘default’, [‘css’]); 002 $ gulp
Plugin dependencies
We’re going to write our own plugin that will analyse a CSS file, find all the images, check to see if they’re under a certain size and then convert them to Base64. First we’re requiring all of the modules that we’ll use (on the disc these are in a package.json file, so you can run ‘npm install’ to grab them all).
001 var through = require(‘through2’), 002 gutil = require(‘gulp-util’), 003 fs = require(‘fs’), 004 q = require(‘q’), 005 path = require(‘path’), 006 PluginError = gutil.PluginError;
Const and config
We use the ES6 keyword ‘const’ to define the plugin name; ‘const’ tells our program that this value won’t change (it’ll be constant). We then make a ‘config’ object that will store any default configuration details. We currently only have one, the maximum size (in kilobytes) that our images should be. We’ve used kilobytes instead of bytes to make it more usable.
001 const PLUGIN_NAME = ‘gulp-base64’;
002 var config = {
003 limit: 8
004 };
Module export
We point ‘module.exports’ to the main function in our plugin; this is what is exposed when we require this module in other files. Because our configuration object is basic we replace it if ‘options’ exists (ideally you’d do some more rigorous checking!). ‘through’ is a wrapper for streams – Gulp pipes a file to us and ‘through’ handles receiving it.
001 function base64 (options) {
002 config = options || config;
003 return through.obj(function(file, encode, callback) {
004 });
005 }
006 module.exports = base64;
007
Check null files
Part of the Gulp plugin guidelines states that if a file’s contents can’t be read then we shouldn’t throw an error, we should just pass it through. ‘through’ includes some convenience methods like ‘isNull’ to help us with this. Later on we’ll write a test to ensure this behaviour never changes.
001 if (file.isNull()) {
002 self.push(file);
003 return callback();
004 }
Throwing errors
Files can come in two types: a buffer or a stream. It’s recommended that plugins support both types but if it doesn’t then we emit an error through the ‘gulp-util’ PluginError helper. Plugins shouldn’t throw their own errors, they should always use this method and preface errors with the plugin name.
001 if (file.isStream()) {
002 new PluginError(PLUGIN_NAME, ‘This plugin does not support streams’);
003 }
Buffer checking
We’re going to be handling buffers in our plugin, so the rest of this program will be within this statement. A buffer is like an array that points to bits of memory to form a whole. This makes them very efficient but something that most JavaScript developers don’t usually have to deal with; thankfully Node makes it all relatively straightforward.
001 if (file.isBuffer()) {
002 // next step
003 }
Convert buffer
A file’s contents are stored as a buffer; to convert this into something that we can read, we call ‘toString’ so that we can treat it as we would any other string. We set our initial starting position, create an empty array to push found images into, and cache the number of images (here we’re just looking for the frequency of ‘url’).
001 var css = file.contents.toString(), 002 lastIndex = 0, 003 images = [], 004 numberOfImages = css.match(/url/g) ? css.match(/ url/g).length : 0; 005
Find image paths
To extract the path we look for ‘url’ from the last index until there are no ‘url’ matches left. This approach has one main limitation, that it requires images to be wrapped in single quotes. If the path isn’t already converted to Base64 then we push it to our ‘images’ array to process later.
001 for (var i = 0; i < numberOfImages; i++) {
002 var start = css.indexOf(‘url’, lastIndex) + 5,
003 end = css.indexOf(‘’’, start),
004 imgPath = css.substring(start, end);
005 lastIndex = start;
006 if (imgPath.indexOf(‘base64’) === -1)
007 images.push(imgPath);
008 }
Making a promise
‘readFiles’ is quite complicated because we now want to loop through each of our found image paths, read their contents asynchronously, and pass it back only when all the files have been processed. To achieve this we’re going to use a popular promise library called Q. If there are no images then we immediately resolve the promise.
001 var readFiles = function () {
002 var deferred = q.defer();
003 var numberOfProcessedImages = 0;
004 if (!images.length) {
005 deferred.resolve();
006 }
007 // next step
008 };
Reading file stats
Within the loop we create a closure to maintain the value of ‘i’. ‘fs.stat’ is a Node filesystem method for getting file stats, we do this because we want to know the size of the file without loading the entirety of it into memory. If the file is not found then we’ll show a message in red to let the user know.
001 for (var i = 0; i < images.length; i++) {
002 (function (i) {
003 fs.stat(images[i], function (err, data) {
004 numberOfProcessedImages++;
005 if (err) {
006 gutil.log(gutil.colors.yellow
(‘”’ + images[i] + ‘” was not found.’));
007 } else {
008 // next step
009 }
010 });
011 )(i);
012 }
Create read stream
If the file is small enough then we’ll create a read stream. No matter what size the user specifies, it must be below 32kb otherwise IE 8 won’t display it. Each chunk of data that we receive is pushed to an array, which will contain all of the chunks, so we’ll eventually have the entire file.
001 if (data.size < config.limit * 1024 && data.size
< 32768) {
002 var all = [];
003 fs.createReadStream(images[i])
004 .pipe(through.obj(function (chunk, enc, cb) {
005 all.push(chunk);
006 cb();
007 }))
008 }
Convert to Base64
To join all of the buffers together we must use Buffer’s ‘concat’ method. To convert this image data to Base64 we simply call ‘toString’, passing the encoding that we’d like. We then find where the image is in ‘css’ and replace it with the encoded version. If we have processed all of the images, then we resolve the promise at this point too.
001 .on(‘data’, function (data) {
002 //push chunk of data to array
003 all.push(data);
004 })
005 .on(‘end’, function () { //once all of file read
006 var base64 = Buffer.concat(all).toString(‘base64’);
007 var ext = path.extname(images[i]);
008 css = css.replace(images[i], ‘data:image
/’ + ext + ‘;base64,’ + base64);
009 if (numberOfImages === numberOfProcessedImages) {
010 deferred.resolve();
011 }
012 })
Print to console
If our image file’s size is too large then we print a message to the console to inform the user. We could add a verbose flag to our plugin to allow users more control over what they see; part of the Gulp philosophy is code over configuration – ‘gulp-util’ includes ways to easily colour-code messages.
001 } else {
002 gutil.log(gutil.colors.blue(‘”’ + images[i] + ‘
” was too large to encode.’));
003 }
Return promise
At the end of ‘readFiles’ we return a promise to let the plugin know not to carry on with the other Gulp processes while we run our asynchronous file reads. This is important as otherwise our plugin will carry on potentially writing to a file when other plugins are trying to do the same.
001 return deferred.promise;
Call callback
Once all of the files are read and the promise is resolved the ‘then’ callback is called. To write the new contents to our CSS file we can convert our ‘css’ string back to a buffer and simply assign ‘file.contents’ to it. We then return the callback to let ‘through’ know to carry on.
001 readFiles().then(function () {
002 file.contents = new Buffer(css);
003 self.push(file);
004 return callback();
005 });
Test dependencies
Plugins must be testable, so we write a couple of tests to check our plugin is working. We’re going to use a testing framework called Mocha. Create a folder called ‘test’ and a file called ‘test.js’, then require our dependencies.
001 var assert = require(‘assert’), 002 gutil = require(‘gulp-util’), 003 gulpBase64 = require(‘../index’), 004 fs = require(‘fs’), 005 es = require(‘event-stream’), 006 path = require(‘path’);
Describe test
We describe our module and then functionality within that module. If we supported streaming this would allow us to test in both buffer and stream modes, or whichever major blocks of functionality you decide to test. ‘describe’ is for grouping tests that make sense to put together and you can nest these as deep as you want.
001 describe(‘gulp-base64’, function() {
002 describe(‘in buffer mode’, function() {
003 });
004 });
Create fake file
Our first test makes sure that we’re generating the correct output. The more human-readable your tests are the better, but because we’re dealing with files there’s quite a lot of setup to do beforehand. We’re using the ‘File’ helper in ‘gulp-util’ to create a fake file from the contents of another.
001 it(‘should encode images to base64 and generate a
css file’, function(done) {
002 var filename = path.join(__dirname, ‘/fixtures/
styles-pre.css’);
003 var input = new gutil.File({
004 base: path.dirname(filename),
005 path: filename,
006 contents: new Buffer(fs.readFileSync(filename))
007 });
008 //next step
009 });
Assert contents
Our plugin returns a ‘through’ stream that we can listen to. We use Node’s ‘assert’ capability to ensure the file that is returned from our plugin matches the expected output. ‘styles.css’ within the fixtures folder is a static file that has the image already encoded. When the test has completed we call ‘done’ to fully complete the test.
001 var stream = gulpBase64();
002 stream.on(‘data’, function (newFile) {
003 var file = fs.readFileSync(path.join(__dirname, ‘
/fixtures/styles.css’), ‘utf8’).toString();
004 assert.equal(String(newFile.contents), file);
005 done();
006 });
Write to stream
We’ve added the data event listener but not started writing the fake input file to our plugin. We do this by writing it to the stream with the ‘write’ method, which passes the data to the stream, which triggers the ‘data’ event.
001 vstream.write(input);
Pass null files
Our second test will make sure that our plugin adheres to the Gulp guidelines, which states that files with no contents are just passed along. We can test this by creating a file with no contents and test to see if the data event is triggered. If the file has been processed then we can pass it on.
001 it(‘should ignore files with no content’, function
(done) {
002 var stream = gulpBase64(),
003 n = 0,
004 filename = path.join(__dirname, ‘/fixtures
/styles-pre.css’);
005 });
Further assertions
We want to ensure three things: that the files are equal, that the contents are ‘null’ and that this has only happened once. The structure for these tests comes from the ‘gulp-cat’ plugin. The Gulp authors encourage learning from other plugins to see how they test but you don’t (and shouldn’t) have to use Gulp within your tests.
001 stream.on(‘data’, function(file) {
002 assert.equal(file.path, filename);
003 assert.equal(file.contents, null);
004 n++;
005 assert.equal(n, 1);
006 done();
007 });
Writing and ending
In the same way we did before, we’re writing a fake file to the stream – only this time we are going to set the contents to ‘null’. Finally, we close the stream with ‘end’, which ensures that no more data can be written to the stream, so the contents should stay as ‘null’.
001 stream.write(new gutil.File({
002 path: filename,
003 contents: null
004 }));
005 stream.end();
006
Run tests
We can run the tests from Terminal with the ‘mocha’ command. You’ll have to have it installed through npm – if you haven’t then you can install it with ‘$ npm install mocha’. Mocha will then tell you if a test has passed or failed as well as a stack trace if any errors are encountered.
001 $ mocha
