JavaScript

How to Write Shell Scripts in Node with Google’s zx Library

    Simon Plenderleith
    Share

    In this article, we’ll learn what Google’s zx library provides, and how we can use it to write shell scripts with Node.js. We’ll then learn how to use the features of zx by building a command-line tool that helps us bootstrap configuration for new Node.js projects.

    Writing Shell Scripts: the Problem

    Creating a shell script — a script that’s executed by a shell such as Bash or zsh — can be a great way of automating repetitive tasks. Node.js seems like an ideal choice for writing a shell script, as it provides us with a number of core modules, and allows us to import any library we choose. It also gives us access to the language features and built-in functions provided by JavaScript.

    But if you’ve tried writing a shell script to run under Node.js, you’ve probably found it’s not quite as smooth as you’d like. You need to write special handling for child processes, take care of escaping command line arguments, and then end up messing around with stdout (standard output) and stderr (standard error). It’s not especially intuitive, and can make shell scripting quite awkward.

    The Bash shell scripting language is a popular choice for writing shell scripts. There’s no need to write code to handle child processes, and it has built-in language features for working with stdout and stderr. But it isn’t so easy to write shell scripts with Bash either. The syntax can be quite confusing, making it difficult to implement logic, or to handle things like prompting for user input.

    The Google’s zx library helps make shell scripting with Node.js efficient and enjoyable.

    Requirements for following along

    There are a few requirements for following along with this article:

    • Ideally, you should be familiar with the basics of JavaScript and Node.js.
    • You’ll need to be comfortable running commands in a terminal.
    • You’ll need to have Node.js >= v14.13.1 installed.

    All of the code in this article is available on GitHub.

    How Does Google’s zx Work?

    Google’s zx provides functions that wrap up the creation of child processes, and the handling of stdout and stderr from those processes. The primary function we’ll be working with is the $ function. Here’s an example of it in action:

    import { $ } from "zx";

await $`ls`;

    And here’s the output from executing that code:

    $ ls
bootstrap-tool
hello-world
node_modules
package.json
README.md
typescript

    The JavaScript syntax in the example above might look a little funky. It’s using a language feature called tagged template literals. It’s functionally the same as writing await $("ls").

    Google’s zx provides several other utility functions to make shell scripting easier, such as:

    • cd(). This allows us to change our current working directory.
    • question(). This is a wrapper around the Node.js readline module. It makes it straightforward to prompt for user input.

    As well as the utility functions that zx provides, it also makes several popular libraries available to us, such as:

    • chalk. This library allows us to add color to the output from our scripts.
    • minimist. A library that parses command-line arguments. They’re then exposed under an argv object.
    • fetch. A popular Node.js implementation of the Fetch API. We can use it to make HTTP requests.
    • fs-extra. A library that exposes the Node.js core fs module, as well as a number of additional methods to make it easier to work with a file system.

    Now that we know what zx gives us, let’s create our first shell script with it.

    Hello World with Google’s zx

    First, let’s create a new project:

    mkdir zx-shell-scripts
cd zx-shell-scripts

npm init --yes

    Then we can install the zx library:

    npm install --save-dev zx

    Note: the zx documentation suggests installing the library globally with npm. By installing it as a local dependency of our project instead, we can ensure that zx is always installed, as well as control the version that our shell scripts use.

    Top-level await

    In order to use top-level await in Node.js — await outside of an async function — we need to write our code in ECMAScript (ES) Modules, which support top-level await. We can indicate that all modules in a project are ES modules by adding "type": "module" in our package.json, or we can set the file extension of individual scripts to .mjs. We’ll be using the .mjs file extension for the examples in this article.

    Running a command and capturing its output

    Let’s create a new script named hello-world.mjs. We’ll add a shebang line, which tells the operating system (OS) kernel to run the script with the node program:

    #! /usr/bin/env node

    Now we’ll add some code that uses zx to run a command.

    In the following code, we’re running a command to execute the ls program. The ls program will list the files in the current working directory (the directory which the script is in). We’ll capture the standard output from the command’s process, store it in a variable and then log it out to the terminal:

    // hello-world.mjs

import { $ } from "zx";

const output = (await $`ls`).stdout;

console.log(output);

    Note: the zx documentation suggests putting /usr/bin/env zx in the shebang line of our scripts, but we’re using /usr/bin/env node instead. This is because we’ve installed zx as a local dependency of our project. We’re then explicitly importing the functions and objects that we want to use from the zx package. This helps make it clear where the dependencies used in our script are coming from.

    We’ll then use chmod to make the script executable:

    chmod u+x hello-world.mjs

    Let’s run our script:

    ./hello-world.mjs

    We should now see the following output:

    $ ls
hello-world.mjs
node_modules
package.json
package-lock.json
README.md
hello-world.mjs
node_modules
package.json
package-lock.json
README.md

    You’ll notice a few things in the output from our shell script:

    • The command we ran (ls) is included in the output.
    • The output from the command is displayed twice.
    • There’s an extra new line at the end of the output.

    zx operates in verbose mode by default. It will output the command you pass to the $ function and also output the standard output from that command. We can change this behavior by adding in the following line of code before we run the ls command:

    $.verbose = false;

    Most command line programs, such as ls, will output a new line character at the end of their output to make the output more readable in the terminal. This is good for readability, but as we’re storing the output in a variable, we don’t want this extra new line. We can get rid of it with the JavaScript String#trim() function:

    - const output = (await $`ls`).stdout;
+ const output = (await $`ls`).stdout.trim();

    If we run our script again, we’ll see things look much better:

    hello-world.mjs
node_modules
package.json
package-lock.json

    Using Google’s zx with TypeScript

    If we want to write shell scripts that use zx in TypeScript, there are a couple of minor differences we need to account for.

    Note: the TypeScript compiler provides a number of configuration options that allow us to adjust how it compiles our TypeScript code. With that in mind, the following TypeScript configuration and code are designed to work under most versions of TypeScript.

    First, let’s install the dependencies we’ll need to run our TypeScript code:

    npm install --save-dev typescript ts-node

    The ts-node package provides a TypeScript execution engine, allowing us to transpile and run TypeScript code.

    We need to create a tsconfig.json file containing the following configuration:

    {
  "compilerOptions": {
    "target": "es2017",
    "module": "commonjs"
  }
}

    Let’s now create a new script named hello-world-typescript.ts. First, we’ll add a shebang line that tells our OS kernel to run the script with the ts-node program:

    #! ./node_modules/.bin/ts-node

    In order to use the await keyword in our TypeScript code, we need to wrap it in an immediately invoked function expression (IIFE), as recommended in the zx documentation:

    // hello-world-typescript.ts

import { $ } from "zx";

void (async function () {
  await $`ls`;
})();

    We then need to make the script executable so that we can execute it directly:

    chmod u+x hello-world-typescript.ts

    When we run the script:

    ./hello-world-typescript.ts

    … we should see the following output:

    $ ls
hello-world-typescript.ts
node_modules
package.json
package-lock.json
README.md
tsconfig.json

    Writing scripts with zx in TypeScript is similar to using JavaScript, but requires a little extra configuration and wrapping of our code.

    Building a Project Bootstrapping Tool

    Now that we’ve learned the basics of writing a shell script with Google’s zx, we’re going to build a tool with it. This tool will automate the creation of a process that’s often time consuming: bootstrapping the configuration for a new Node.js project.

    We’re going to create an interactive shell script that prompts for user input. It will also use the chalk library which zx bundles to highlight output in different colors and provide a friendly user experience. Our shell script will also install the npm packages that our new project needs, so it’s ready for us start development on right away.

    Getting started

    Let’s create a new file named bootstrap-tool.mjs and add a shebang line. We’ll also import the functions and modules we’ll be using from the zx package, as well as the Node.js core path module:

    #! /usr/bin/env node

// bootstrap-tool.mjs

import { $, argv, cd, chalk, fs, question } from "zx";

import path from "path";

    As with the scripts we’ve created previously, we want to make our new script executable:

    chmod u+x bootstrap-tool.mjs

    We’re also going to define a helper function that outputs an error message in red text and exits the Node.js process with an error exit code of 1: 

    function exitWithError(errorMessage) {
  console.error(chalk.red(errorMessage));
  process.exit(1);
}

    We’ll use this helper function in various places through our shell script when we need to handle an error.

    Check dependencies

    The tool we’re creating will need to run commands that use three different programs: git, node and npx. We can use the library which to help us check whether these programs are installed and available to use.

    First, we need to install the which package:

    npm install --save-dev which

    Then we can import it:

    import which from "which";

    Then we’ll create a checkRequiredProgramsExist function that uses it:

    async function checkRequiredProgramsExist(programs) {
  try {
    for (let program of programs) {
      await which(program);
    }
  } catch (error) {
    exitWithError(`Error: Required command ${error.message}`);
  }
}

    The function above accepts an array of program names. It loops through the array, and for each program it calls the which function. If which finds the path to the program, it will return it. Otherwise, if the program is missing, it will throw an error. If any of the programs are missing, we call our exitWithError helper to display an error message and stop running the script.

    We can now add a call to checkRequiredProgramsExist to check that the programs our tool depends on are available:

    await checkRequiredProgramsExist(["git", "node", "npx"]);

    Add a target directory option

    As the tool we’re building is going to help us bootstrap new Node.js projects, we’ll want to run any commands we add in the project’s directory. We’re now going to add a --directory command line argument to our script.

    zx bundles the minimist package, which parses any command-line arguments that are passed to our script. These parsed command-line arguments are made available as argv by the zx package.

    Let’s add a check for a command-line argument named directory:

    let targetDirectory = argv.directory;
if (!targetDirectory) {
  exitWithError("Error: You must specify the --directory argument");
}

    If the directory argument has been passed to our script, we want to check that it’s the path to a directory that exists. We’ll use the fs.pathExists method provided by fs-extra:

    targetDirectory = path.resolve(targetDirectory);

if (!(await fs.pathExists(targetDirectory))) {
  exitWithError(`Error: Target directory '${targetDirectory}' does not exist`);
}

    If the target directory exists, we’ll then use the cd function provided by zx to change our current working directory:

    cd(targetDirectory);

    If we now run our script without a --directory argument, we should receive an error:

    $ ./bootstrap-tool.mjs

Error: You must specify the --directory argument

    Check global Git settings

    In a moment, we’re going to initialize a new Git repository in our project directory, but first we want to check that Git has the configuration it needs. We want to ensure our commits will be attributed correctly by code hosting services like GitHub.

    To do this, let’s create a getGlobalGitSettingValue function. It will run the command git config to retrieve the value of a Git configuration setting:

    async function getGlobalGitSettingValue(settingName) {
  $.verbose = false;

  let settingValue = "";
  try {
    settingValue = (
      await $`git config --global --get ${settingName}`
    ).stdout.trim();
  } catch (error) {
    // Ignore process output
  }

  $.verbose = true;

  return settingValue;
}

    You’ll notice that we’re switching off the verbose mode that zx has set by default. This means that, when we run the git config commands, the command and anything it sends to standard output won’t be displayed. We switch verbose mode back on at the end of the function so we don’t affect any other commands that we add later in our script.

    Now we’ll create a checkGlobalGitSettings that accepts an array of Git setting names. It will loop through each setting name and pass it to the getGlobalGitSettingValue function to retrieve its value. If the setting doesn’t have a value, we’ll display a warning message:

    async function checkGlobalGitSettings(settingsToCheck) {
  for (let settingName of settingsToCheck) {
    const settingValue = await getGlobalGitSettingValue(settingName);
    if (!settingValue) {
      console.warn(
        chalk.yellow(`Warning: Global git setting '${settingName}' is not set.`)
      );
    }
  }
}

    Let’s call add a call to checkGlobalGitSettings and check that the user.name and user.email Git settings have been set:

    await checkGlobalGitSettings(["user.name", "user.email"]);

    Initialize a new Git repository

    We can initialize a new Git repository in the project directory by adding the following command:

    await $`git init`;

    Generate a package.json file

    Every Node.js project needs a package.json file. It’s where we define metadata about the project, specify the packages the project depends upon, and add small utility scripts.

    Before we generate a package.json file for our project, we’re going to create a couple of helper functions. The first is a readPackageJson function, which will read a package.json file from the project directory:

    async function readPackageJson(directory) {
  const packageJsonFilepath = `${directory}/package.json`;

  return await fs.readJSON(packageJsonFilepath);
}

    We’ll then create a writePackageJson function, which we can use to write changes to the project’s package.json file:

    async function writePackageJson(directory, contents) {
  const packageJsonFilepath = `${directory}/package.json`;

  await fs.writeJSON(packageJsonFilepath, contents, { spaces: 2 });
}

    The fs.readJSON and fs.writeJSON methods that we’ve used in the functions above are provided by the fs-extra library.

    With our package.json helper functions defined, we can start to think about the contents of our package.json file.

    Node.js supports two module types:

    • CommonJS Modules (CJS). Uses module.exports to export functions and objects, and require() to load them in another module.
    • ECMAScript Modules (ESM). Uses export to export functions and objects and import to load them in another module.

    The Node.js ecosystem is gradually adopting ES modules, which are common in client-side JavaScript. While things are in this transitional phase, we need to decide whether our Node.js projects will use CJS or ESM modules by default. Let’s create a promptForModuleSystem function that asks which module type this new project should use:

    async function promptForModuleSystem(moduleSystems) {
  const moduleSystem = await question(
    `Which Node.js module system do you want to use? (${moduleSystems.join(
      " or "
    )}) `,
    {
      choices: moduleSystems,
    }
  );

  return moduleSystem;
}

    The function above uses the question function that’s provided by zx.

    We’ll now create a getNodeModuleSystem function to call our promptForModuleSystem function. It will check that the value which is entered is valid. If it isn’t, it will ask the question again:s

    async function getNodeModuleSystem() {
  const moduleSystems = ["module", "commonjs"];
  const selectedModuleSystem = await promptForModuleSystem(moduleSystems);

  const isValidModuleSystem = moduleSystems.includes(selectedModuleSystem);
  if (!isValidModuleSystem) {
    console.error(
      chalk.red(
        `Error: Module system must be either '${moduleSystems.join(
          "' or '"
        )}'\n`
      )
    );

    return await getNodeModuleSystem();
  }

  return selectedModuleSystem;
}

    We can now generate our project’s package.json file by running the npm init command:

    await $`npm init --yes`;

    Then we’ll use our readPackageJson helper function to read the newly created package.json file. We’ll ask which module system the project should use, set it as the value of the type property in the packageJson object, and then write it back to the project’s package.json file:

    const packageJson = await readPackageJson(targetDirectory);
const selectedModuleSystem = await getNodeModuleSystem();

packageJson.type = selectedModuleSystem;

await writePackageJson(targetDirectory, packageJson);

    Tip: to get sensible default values in your package.json when you run npm init with the --yes flag, make sure you set the npm init-* configuration settings.

    Install required project dependencies

    To make it easy to get started on project development after running our bootstrapping tool, we’ll create a promptForPackages function that asks what npm packages to install:

    async function promptForPackages() {
  let packagesToInstall = await question(
    "Which npm packages do you want to install for this project? "
  );

  packagesToInstall = packagesToInstall
    .trim()
    .split(" ")
    .filter((pkg) => pkg);

  return packagesToInstall;
}

    Just in case we end up with a typo when entering a package name, we’ll create an identifyInvalidNpmPackages function. This function will accept an array of npm package names, then run the npm view command to check if they exist:

    async function identifyInvalidNpmPackages(packages) {
  $.verbose = false;

  let invalidPackages = [];
  for (const pkg of packages) {
    try {
      await $`npm view ${pkg}`;
    } catch (error) {
      invalidPackages.push(pkg);
    }
  }

  $.verbose = true;

  return invalidPackages;
}

    Let’s create a getPackagesToInstall function that uses the two functions we’ve just created:

    async function getPackagesToInstall() {
  const packagesToInstall = await promptForPackages();
  const invalidPackages = await identifyInvalidNpmPackages(packagesToInstall);

  const allPackagesExist = invalidPackages.length === 0;
  if (!allPackagesExist) {
    console.error(
      chalk.red(
        `Error: The following packages do not exist on npm: ${invalidPackages.join(
          ", "
        )}\n`
      )
    );

    return await getPackagesToInstall();
  }

  return packagesToInstall;
}

    The function above will display an error if any package names are incorrect, and then ask again for the packages to install.

    Once we’ve got a list of valid packages to install, let’s install them with the npm install command:

    const packagesToInstall = await getPackagesToInstall();
const havePackagesToInstall = packagesToInstall.length > 0;
if (havePackagesToInstall) {
  await $`npm install ${packagesToInstall}`;
}

    Generate configuration for tooling

    Creating project configuration is the perfect thing for us to automate with our project bootstrapping tool. First, let’s add a command to generate a .gitignore file so we don’t accidentally commit files that we don’t want in our Git repository:

    await $`npx gitignore node`;

    The command above uses the gitignore package to pull in the Node.js .gitignore file from GitHub’s gitignore templates.

    To generate our EditorConfig, Prettier and ESLint configuration files, we’ll use a command-line tool called Mrm.

    Let’s globally install the mrm dependencies we’ll need:

    npm install --global mrm mrm-task-editorconfig mrm-task-prettier mrm-task-eslint

    And then add the mrm commands to generate the configuration files:

    await $`npx mrm editorconfig`;
await $`npx mrm prettier`;
await $`npx mrm eslint`;

    Mrm takes care of generating the configuration files, as well as installing the required npm packages. It also provides plenty of configuration options, allowing us to tune the generated configuration files to match our personal preferences.

    Generate a basic README

    We can use our readPackageJson helper function to read the project name from the project’s package.json file. Then we can generate a basic Markdown formatted README and write it to a README.md file:

    const { name: projectName } = await readPackageJson(targetDirectory);
const readmeContents = `# ${projectName}

...
`;

await fs.writeFile(`${targetDirectory}/README.md`, readmeContents);

    In the function above, we’re using the promise variant of fs.writeFile that’s exposed by fs-extra.

    Commit the project skeleton to Git

    Lastly, it’s time to commit the project skeleton that we’ve created with git:

    await $`git add .`;
await $`git commit -m "Add project skeleton"`;

    Then we’ll display a message confirming that our new project has been successfully bootstrapped:

    console.log(
  chalk.green(
    `\n✔️ The project ${projectName} has been successfully bootstrapped!\n`
  )
);
console.log(chalk.green(`Add a git remote and push your changes.`));

    Bootstrap a new project

    Now we can use the tool we’ve created to bootstrap a new project:

    mkdir new-project

./bootstrap-tool.mjs --directory new-project

    And watch everything we’ve put together in action!

    Conclusion

    In this article, we’ve learned how we can create powerful shell scripts in Node.js with the help of Google’s zx library. We’ve used the utility functions and libraries it provides to create a flexible command-line tool.

    The tool that we’ve built so far is just the beginning. Here are a few feature ideas you might like to try adding yourself:

    • Automatically create the target directory. If the target directory doesn’t already exist, prompt the user and ask if they would like it to be created for them.
    • Open-source hygiene. Ask the user if they’re creating a project that will be open-source. If they are, run commands to generate license and Contributor Convenant files.
    • Automate the creation of a repository on GitHub. Add commands that use the GitHub CLI to create a remote repository on GitHub. The new project can then be pushed to this repository once the initial skeleton has been committed with Git.

    All of the code in this article is available on GitHub.