NPM-Only Challenge! Making A Development Environment

Posted on Sep 9, 2021 by Alexander Morse.

A few days ago, I found myself going through the familiar motions of setting up a simple development environment in npm. It was the same one I'd made dozens of times before, with Sass preprocessing and a auto-reload feature for live development in the browser. It's simple enough that I don't even bother with a project template — I just type out two or three lines in my terminal to get everything working properly.

This time, though, I had a thought: how much can I get out of a development environment created using only the npm CLI? Usually, there's a point where I hit a wall of complexity, and I need to switch to using Gulp or Webpack or something else with a configuration file. But suppose I was stubborn, and refused to leave the command line for any reason. What features could I implement?

With that question firmly in mind, I got to work on what might be the dumbest side-project I've yet to undertake.

The Challenge

We begin with a directory. In this directory, there is a single src directory, containing a small handful of files:

| - cmd-only-challenge
    | - src
        | - index.html
        | - index.js
        | - styles.scss
        | - shoulder-cat.jpg

index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link rel="stylesheet" href="styles.css" />
    <title>A Cat on a Shoulder</title>
  </head>
  <body>
    <div class="wrapper">
      <h1>Shoulder Cat!</h1>
      <div>
        <img src="shoulder-cat.jpg" alt="A cat on a shoulder" />
        <p>
          Want to see a cat on a shoulder? Then you've come to the right place.
        </p>
      </div>

      <div id="js-check">This page's Javascript has not loaded.</div>
    </div>
    <script src="index.js"></script>
  </body>
</html>

index.js

document.querySelector("#js-check").innerHTML =
  "This page's Javascript <b>has</b> loaded.";

styles.scss

body {
  background-color: lightblue;
}

img {
  float: left;
  max-width: 500px;
}

.wrapper {
  max-width: 1200px;
  margin: 0 auto;
}

shoulder-cat.jpg

a kitten sitting on a shoulder

Given this simple starting project, the challenge is as follows:

Requirements

Deliver a processed version of each file to a dist directory that is sibling to the src directory.
Have the processed files work together as intended. The scss file should have been rendered into normal CSS, and the Javascript file should be loaded by the webpage.
Set up a live-reloading development server on the dist directory, such that if we make a change to anything in src, we automatically see the change in the browser.

Rules

The development environment must be set up using the npm CLI only. Notably, this means no switching over to a text editor, even a terminal-based one like Vim. Any command we run on the command line should start with npm or npx.
No writing custom code. While we might be tempted to use something like cat to create a configuration file or Node script, this is against the spirit of the challenge. Any direction we give should be in the form of CLI arguments.

With our parameters firmly in place, let the insanity begin.

Setting Up The npm Project

Right out of the gate, I'm going to be a little more particular than usual with what version of npm I'm using. I'll be going through this article using npm 7, which is packaged with Node 16.x.

Why do we need that specific version? Actually, npm 7 is the only reason my strategy works at all. It comes with a new command that previous versions don't have: npm set-script, which allows us to define package scripts using the CLI. Since we can't edit package.json in an editor, we'll need to use this if we want to use scripts.

(Note: Technically, we could use the json package, which allows for editing JSON files on the command line. If for some reason you want to follow along on this insane experiment, and don't have npm 7 installed, you could use that, or just allow yourself to edit package.json in an editor. I'll be using set-script for the sake of simplicity.)

With that out of the way, we can initialize our npm project in the usual lazy way:

npm init -y

Processing Source Files

Now that our project is initialized, let's start by focusing on how we want to process each file type. We'll do this in a straightforward, naïve way, and set up one process:<type> script for each of the four file types we want to deliver into the dist folder.

Along the way, though, we'll need to figure out which tools to use. We'll gloss over most of the usage for each, since this isn't meant to be a tutorial. Check out the links if you'd like to learn more about any of them.

HTML Files

I don't want to do anything special with HTML files — just copy them over as-is into the dist folder. In a perfect world, this wouldn't call for any additional tools, and we'd just use the cp shell command to copy the files. But this is the npm-only challenge, so we need to find a package to do that for us.

I'm going to use the copyfiles package, which does exactly what you'd think. The main draw here is the CLI's support for globs, which will allow us to select multiple HTML files at once. Let's install the package, and then set up the script:

# Install copyfiles
npm install --save-dev copyfiles

# Setup "process:html" script
# Note: The '-f' option flattens the output, so that the `src`
# directory isn't copied over.
npm set-script process:html "copyfiles -f \"src/*.html\" dist"

Notice how we escaped the inner double-quotation marks when defining the source glob — we'll need to do this a few more times throughout the challenge, since the scripts we're providing are going to live inside of package.json.

More importantly, this was our first use of set-script. In general, the syntax goes npm set-script <name> <command>, where <command> is more often than not wrapped in quotes. If we check package.json now, we should see that process:html has been added to the scripts mapping.

Javascript Files

For Javascript, I'm going to use Babel. Why? Because it's the only package I can find that can operate on multiple JS files at once. We'll use it with the preset-env preset to transpile all of our code into something ES5 compatible. (Not that it isn't already, but that's beside the point.)

# Install Babel and related packages
npm install --save-dev @babel/core @babel/cli @babel/preset-env

# Setup "process:js" script
npm set-script process:js "babel src --outDir dist --minified --presets=@babel/env"

I should note that this isn't the best way to use preset-env. Ideally, we'd have a browserslist field in our package.json file to specify which browsers the preset should target. But since this would require us to edit said file manually, I'm going to skip that step. As I mentioned earlier, the json package could be used to achieve this from the command line.

SCSS Files

This one is easy. We install the sass package, and use that to process everything in src at once.

# Install Sass
npm install --save-dev sass

# Setup "process:sass" script
npm set-script process:sass "sass src:dist"

Image Files

Once again, we'll keep this simple and use the imagemin library to process all jpg files in src. Mercifully, the CLI accepts a glob.

# Install imagemin CLI
npm install --save-dev imagemin-cli

# Setup "process:images" script
npm set-script process:images "imagemin src/*.jpg --out-dir dist"

Watching Files

Four libraries and four scripts later, we've cleared the first hurdle of processing all the file types in our project. Next, we'll need to set up watcher scripts, so that if any file changes, it is immediately processed.

I struggled with this one at first. Only the Sass CLI comes with any option to automatically watch files, so we need to bring in some other tool for everything else. Normally, npm-watch would be a solid option for this kind of project, but it requires us to add a watch field to package.json. Eventually I came across the sane package, whose CLI will watch files (which can be globs) and run a command every time a watched file is changed.

We could watch all of the files in src at once, but since we've got some potentially-slow scripts like process:image, we'll create one watcher for each file type.

# Install sane package
npm install --save-dev sane

# Watch HTML files and trigger process:html
npm set-script watch:html "sane \"npm run process:html \" --glob=\"src/*.html\"

# Watch JS files and trigger process:js
npm set-script watch:js "sane \"npm run process:js\" --glob=\"src/*.js\""

# Watch JPG files and trigger process:image
npm set-script watch:images "sane \"npm run process:images\" --glob=\"src/*.jpg""

# Watch SCSS files and trigger process:sass
# (Instead of using sane, we can just tack on the '-w' flag
# at the end of process:sass.)
npm set-script watch:sass "npm run process:sass -- -w"

Once again, note the use of escaped inner quotation marks — they're necessary for this format. With this, we're able to watch each file type individually. Whenever a file changes, all files of that type will be processed. It's not exactly a production-ready solution, but it meets our requirements. Moving on.

Development Server

Now that we can process and watch all of our files, we can move onto the final requirement: we need to provision a development server that will serve files from dist, as well as automatically reload the browser whenever a file in dist changes. There are a few easy options here, but my go-to tool in this situation is BrowserSync. It has a lot of different features, but it will do everything we need it to out of the box.

Let's install it, and set up a server script to launch it.

# Install browser-sync
npm install --save-dev browser-sync

# Setup the 'server' script
# Note: the -s flag sets the directory to serve from.
# -f tells BrowserSync to watch the 'dist' directory for changes.
npm set-script server "broswer-sync start -s dist -f dist"

...and that's all there is to our development server.

Putting It All Together

The last thing left to do is make our watchers run at the same time as BrowserSync — there is no point to watching a directory whose files never update. This means we need to run several scripts at once: server, and each of the watch:* scripts.

Fortunately, the convenient npm-run-all package allows us to do just that. It even supports a glob-like syntax that will save us a few keystrokes on the watch:* scripts. Since this will be our final, top-level script, we'll call it start.

# Install npm-run-all
npm install --save-dev npm-run-all

# Setup "start" script
# Note: -p will cause all of the given scripts to run in parallel,
# which is what we need in this case.
npm set-script start "npm-run-all -p watch:* server"

Now, we can run everything with npm start, and check that everything works as it should. Challenge complete!

Right?

...technically, we've succeeded. But the end result isn't especially elegant. If we edit a file, it will be processed and updated in the browser automatically. But as it stands, the call to sane will probably trigger multiple reloads in BrowserSync as each of the file types is processed. Also, there's the matter of the dist directory itself. We're kind of taking it for granted, but what happens if we start the environment when it doesn't exist? I don't know, and I'd rather not have to worry about it.

Keeping to the rules of the challenge, can we fix these issues?

(Extra Credit) Cleaning Up

Let's give fixing these issues a shot.

First, let's see about implementing a 'clean' script, to ensure we aren't dealing with old assets at any point. As with copyfiles, we'll bring in a package to emulate the rm command. In this case, rimraf is the obvious pick.

# Install rimraf
npm install --save-dev rimraf

# Setup 'clean' script.
npm set-script clean "rimraf dist"

Nothing to it. After we've removed the dist folder, we'll want to do two things. First, re-create the dist folder, which means installing a mkdir equivalent: mkdirp.

# Install mkdirp
npm install --save-dev mkdirp

# Setup 'mkdist script'
# This simply re-creates the 'dist' directory.
npm set-script mkdist "mkdirp dist"

Second, we'll want to run all of the process:* commands at once to get new versions of our files. We can use npm-run-all to orchestrate all of this.

# Setup 'build' script
# Do three things:
# 1 - Remove 'dist' directory
# 2 - Create new 'dist' directory
# 3 - Run all 'process:*' commands in parallel.
npm set-script build "npm-run-all clean mkdist -p process*"

Notice that we're taking advantage of a neat grouping feature of npm-run-all, here. Our script will run clean and mkdist sequentially, one after the other. But then, since they follow the -p option, all of the process:* scripts will run in parallel. npm-run-all has a number of neat tricks up its sleeve, so I'd suggest reading the documentation if you ever want to use it for a real project.

Almost done! When we run the start script, we want to immediately call build, wait until it's done, and then call the server and watch:* scripts. That fits neatly into npm-run-all's domain, so we can accomplish this by updating the start script:

# Update 'start' to run 'build' before starting
# the server.
npm set-script start "npm-run-all build -p server watch:*"

After all that, we can launch the development server with npm start...to find that we still haven't solved the problem of multiple reloads. This is because, by default, the sane CLI will automatically run its attached command once at startup, even before we change any files.

Fortunately, we can tell sane to ignore the startup command by passing it the -o (changes only) option. With this set, sane will only re-process files that have changed. Therefore, once we've passed -o to every instance of sane, our environment should work perfectly.

Yeah, I know — I don't want to type out all of that again either. But if we make the final push of updating every sane script...

# Update 'watch' scripts for HTML, JS, Images
# (Hint: Use your command history to save keystrokes)
npm set-script watch:html "sane \"npm run process:html\" --glob=\"src/*.html\" -o"
npm set-script watch:js "sane \"npm run process:js\" --glob=\"src/*.js\" -o"
npm set-script watch:images "sane \"npm run process:images\" --glob=\"src/*.images\" -o"

...and our work is finally done. We've eliminated the unnecessary reloads at the start of the server while still ensuring that everything else still works. We're being really generous with our definition of "works", but still. We did this using nothing but the npm CLI, so let's give ourselves some credit.

Lessons Learned

Let's be clear. By any professional standard, this is a terrible solution. We're missing a ton of useful features, and the features we do have are pretty inefficient at doing their jobs. Support for more than one type of image might have been nice, too. If the goal of this experiment was to determine whether a good development environment can be achieved using only the npm CLI, I'd say the answer is no.

I had hoped there would be a punchline to this article — I spent some time testing out various bundlers and static-site-generators in hopes of reducing this whole experiment to a one-liner. Parcel came the closest, and would have been a one-liner had its live-preview feature not broken down during my tests. Addressing the issue would have taken me out of the command line, which disqualifies it as a solution.

I think there's a lesson to take from that, though: it illustrates the power of thinking things through. Our solution might be terrible, but it isn't wrong. In fact, it works miles better than any framework or boilerplate code (under the restrictions of the challenge), simply because we chose the right packages for each task. Furthermore, selecting tools in this way allowed us a complete understanding of every part of the toolchain, which is a luxury we usually don't get with something like Webpack.

I doubt I'll be using the npm-only approach in my day-to-day development work, for so many reasons. But if the net effect of a dumb approach is that it forces you to work smarter, maybe it's not so dumb after all.