How to Solve the Global npm Module Dependency Problem

September 4, 2015

The Node Package Manager (a.k.a. npm) has given web developers easy access to a lot of awesome JavaScript modules and has made our lives considerably easier when trying to find and manage dependencies for our applications. It also makes it easy for developers to create and publish their own modules, meaning that other developers can grab them with a simple npm install -g your-tool and start using them any time they want to. It’s utopia! Right?

Err, actually …

We’ve Got a Bit of a Problem

I will never say never use the -g option when installing an npm module, but I do have to say that we are causing problems by using it too much. There are a couple reasons that I think we should cut down on our use of global module installation, especially in the case of build, test, or linting tools such as Gulp, Karma, JSHint, and countless others. I’ll be referring primarily to Gulp throughout this article because it’s quite popular and it’s fun to say, but if you don’t like Gulp, just mentally replace it with whatever you prefer.

First of all, global modules are not listed as dependencies in your projects, even though your project depends on them, which causes extra steps for others using your application. You know that you need to use Gulp in order to get your project ready for production, so you install it globally and use it. When someone else wants to start working on, or using your wonderful open source project, they can’t just type npm install and get going. You end up having to throw directions into your README file saying something along the lines of

To use this project, follow these steps:

git clone the repo
Run npm install
Run npm install -g gulp
Run gulp to build

I see two issues with this: firstly, you are adding the extra step of installing Gulp globally and secondly, you are running gulp directly. I see an extra step that could have been avoided (globally installing Gulp) and I see that the user is required to know that your app uses Gulp in order to build the project. This first issue is the main one I’m going to address in this article, and although the second one isn’t as big of an issue, you’ll need to update the instructions if you end up switching tools. The solution I discuss later should fix both of these issues.

The second big issue relating to installing modules globally is that you can run into conflicts due to having the wrong version of the module installed. This is illustrated by the following two examples:

You created your project six months ago and you used the latest version of Gulp at that time. Today, someone has cloned your project’s repo and tried to run gulp to build it, but runs into errors. This is because the person who cloned your project is either running an older version or a newer version of Gulp that has some breaking differences.
You created a project six months ago that used Gulp. Since then you’ve moved on to other projects and updated Gulp on your machine. Now you go back to this old project and try to run gulp and you experience errors because you’ve updated Gulp since the last time you touched the project. Now you are forced to update your build process to work with the new version of Gulp before you can make any more progress on the project, instead of putting it off until a more convenient time.

These are potentially very crippling issues. Like I said earlier though, I wouldn’t make a blanket statement telling you never to install something globally. There are exceptions.

A Brief Note on Security

By default, on some systems, installing a npm module globally requires elevated privileges. If you find yourself running commands like sudo npm install -g a-package, you should change this. Our beginners guide to npm shows you how.

Exceptions to the Rule

So what can you install globally? To put it simply: anything your project doesn’t depend on. For example, I have a global module installed called local-web-server. Whenever I just have some HTML files I want to view in the browser, I’ll just run ws (that’s the command for local-web-server) and it’ll set the current folder as the root for localhost:8000 and I can pop open any documents under there in my browser and test them out.

I also run into situations where I want to minify JavaScript files that aren’t part of a project, or at least aren’t part of a project where I’m allowed to set up a formal build process (for silly “corporate” reasons). For this, I have uglify-js installed and I can easily minify any script from my command line in seconds.

The Solution

Now that we know where issues can arise, how do we prevent them? The first thing you need to do is remove that -g when you install modules. You should replace that with --save-dev so you can save the module as a development dependency and it will always be installed when someone runs npm install. That only solves one of the minor issues that I mentioned, but it’s a start.

What you need to know is that when you install a dependency locally, if it has any scripts that are meant to be run from the command line, they will be placed in ./node_modules/.bin/. So, right now, if you just install Gulp locally, you could run it by typing ./node_modules/.bin/gulp in your command line. Of course, no one wants to type that whole thing in. You can fix this with npm scripts.

Inside your package.json file, you can add a scripts property that looks something like this:

{
    ...
    "scripts": {
        "gulp": "gulp"
    }
}

Now you can run npm run gulp any time you want to run the local version of Gulp. npm scripts will look for a local copy of an executable command in the ./node_modules/.bin/ directory before checking your PATH for it. If you want, you can even pass other arguments to Gulp by adding -- before those arguments, e.g. npm run gulp -- build-dev is equivalent to gulp build-dev.

It sucks that you still need to type in more than you would if you used Gulp globally, but there are two ways around that. The first way, which also solves one of the issues I brought up earlier, is to use npm scripts to create aliases. For example, you shouldn’t necessarily tie your app to Gulp, so you could create scripts that run Gulp, but do not mention Gulp:

{
    ...
    "scripts": {
        "build": "gulp build-prod",
        "develop": "gulp build-dev"
    }
}

This way, you can keep your calls to Gulp shorter and you keep your scripts generic. By keeping them generic, you can transparently remove Gulp any time and replace it with something else and no one needs to know (unless they work on the build process, in which case, they should know about it already and probably should have been part of the conversation to move away from Gulp). Optionally, you can even throw a postinstall script in there to automatically run the build process immediately after someone runs npm install. This would clean up your README quite a bit. Also, by using npm scripts, anyone who clones your project should have simple and immediate documentation regarding all the processes you run on your project right in the package.json file.

In addition to using npm scripts, there is another trick that will let you use your local installations of command line tools: a relative PATH. I added ./node_modules/.bin/ to my path, so that as long as I am in a project’s root directory, I have access to the command tools simply by typing in the name of the command. I learned this trick from a comment on a different post I wrote (thanks Gabriel Falkenberg).

These tricks cannot necessarily replace every situation where you’d want to use something like Gulp, and they do take a bit of work to set up, but I do believe it should be a best practice to include those tools listed in your dependencies. This will prevent version clashing (which is one of the main reasons behind dependency managers in the first place) and will help simplify the steps necessary for someone to pick up your project.

Going above and Beyond

This may be a bit excessive, but I also believe that Node and npm are dependencies for your project which have several different versions that can clash. If you want to be sure your application will work for EVERYONE, then you need some way to ensure that the user has the correct versions of Node and npm installed as well.

You can install local copies of Node and npm into your project! This doesn’t make everything fine and dandy, though. First of all, Node isn’t the same on every operating system, so each individual would still need to make sure they download the one that works with their operating system. Secondly, even if there was a way to have a universal Node installed, you would need to make sure that each person has a simple way to access Node and npm from their command line, such as ensuring that everyone adds the path to the local copy of Node and npm to their PATH. There’s no simple way to guarantee this.

So, as much as I’d love to be able to enforce specific versions of Node and npm per project, I can’t think of a good way to do so. If you think it’s a good idea and come up with a good solution, let us all know about it in the comments. I’d love to see a simple enough solution that this could become a standard practice!

The Final Word

I hope you can now see the importance of keeping your tools listed as versioned dependencies for your projects. I also hope that you’re willing to do the work required to implement these practices in your own projects so we can push these practices forward as a standard. Unless of course, you’ve got a better idea, in which case speak up and let the world know about it!