Why gulp might not be the answer.

July 25, 2014 • hack

Gulp, the streaming build system, has been around for a bit now and it is incredible to see how fast its ecosystem has been growing. If there is a build task you want to use, chances are that there is already a plugin readily waiting for you. We also contributed a few modules (gulp-gm, gulp-image-resize, amd-optimize). Before Gulp the preferred build system of many frontend devs was Grunt.

Grunt is a task runner with an ecosystem as big as Gulp’s. The tasks (or plugins) would read some files from the filesystem, process them and output them. This works great for simple build processes but becomes cumbersome as soon as you have multiple sequential build steps. Commonly, the solution was to save intermediate files in a .tmp folder and have the following tasks read them from there. This is obviously a leaky abstraction, because you’ll end up configuring a set of tasks that seem independent but actually aren’t. You would need to be careful how to lay out the files in the directory structure. And even on recent hardware it’s still considerably slower to write and read from hard disk instead of keeping everything in memory.

Gulp solves these issues in a very clever way. First, it’s designed to be a composition of sequential build steps. Give it a source, set up your pipeline and specify a destination folder. That’s a much better abstraction. Second, it follows the UNIX/node philosophy of small modules that have a very focussed set of features. Gulp itself supplies a standard way of reading and writing files. Between both endpoints of the pipeline small modules can transform the stream of files; do stuff like compiling CoffeeScript files, minifying CSS, compressing images or uploading files. But every step is truly independent of the others. Third, Gulp makes it very simple to leverage large parts of the node ecosystem. Writing a wrapper for an exisiting module is as easy as loading the through2 module and calling a function.

However, there is still a conceptual problem that Gulp has yet to address. Many build steps are not 1:1 (one file in, one file out) but rather n:1 or 1:n. Just think of a LESS compiler that wants to include @imports or a RequireJS optimizer that traces all dependencies of a module and concatenates them to one. Gulp’s streaming is made to process individual files. Of course, plugins can either cache some streamed files or use node’s fs module directly. But that breaks the system somewhat. For example, the popular browserify module loads and bundles the modules itself. However, if you’re working with CoffeeScript you’ll need to rely on browserify to do the compiliation step instead of Gulp.

var gulp = require('gulp');
var browserify = require('browserify');
var source = require('vinyl-source-stream');

gulp.task('scripts', function() {
  return browserify({
    entries: ['./app/js/main.coffee']
    extensions: ['.coffee', '.js']
  })
      // Compiles CoffeeScript files on-the-fly
      .transform('coffeeify')
      .bundle()
      // Convert browserify output to Gulp stream
      .pipe(source('bundle.js'))
      // Write to disk
      .pipe(gulp.dest('./build/'));
});

Tools like Broccoli or Metalsmith offer a file-tree-based approach to solve these problems. Basically, you read in some files or directories, and then apply a chain of transformations on the whole tree. Conceptually, this is as awesome as it gets, because it allows you to do single-file compilations as well as module aggregations. Broccoli makes extensive use of caching. Every build step caches its output. This may seem like a cheap performance trick. But it is actually incredibly powerful, because it makes rebuilds extremely fast. Rebuilds become so fast that whenever a file in your project changes you can trigger a full rebuild. Over are the days when you had to specify different build rules for watch and build. It almost feels like a functional-reactive build system, which is awesome!

On the downside, Broccoli is not nearly as mature as Gulp. It lacks a comparable-sized ecosystem of plugins and community support. Furthermore, Gulp’s API is also much more intuitive and easier to reason about (see example Brocfile). Like in the Grunt days, Broccoli uses the file system for intermediate file results which feels a bit outdated.

var filterCoffeeScript = require('broccoli-coffee');
var concat = require('broccoli-concat');

// Read file tree from 'app' directory and compile
// all included CoffeeScript files.
scripts = filterCoffeeScript('app', {
  bare: true
})

// Concatenate all script files.
scripts = concat(scripts, {
  inputFiles: ['**/*.js'],
  outputFile: '/app.js'
})

module.exports = scripts;

Anyhow, the search for the perfect build system continues. But there are some strong contenders out there already. My dream setup would be something with the file-tree concepts of Broccoli and the ecosystem and simple syntax of Gulp.

Title photo by Antti T. Nissinen.

by Norman Rzepka


Related posts