Node can't find certain modules after synchron

Question

I've got a script that synchronously installs non-built-in modules at startup that looks like this

const cp = require('child_process')

function requireOrInstall (module) {
  try {
    require.resolve(module)
  } catch (e) {
    console.log(`Could not resolve "${module}"\nInstalling`)
    cp.execSync(`npm install ${module}`)
    console.log(`"${module}" has been installed`)
  }
  console.log(`Requiring "${module}"`)
  try {
    return require(module)
  } catch (e) {
    console.log(require.cache)
    console.log(e)
  }
}

const http    = require('http')
const path    = require('path')
const fs      = require('fs')
const ffp     = requireOrInstall('find-free-port')
const express = requireOrInstall('express')
const socket  = requireOrInstall('socket.io')
// List goes on...

When I uninstall modules, they get installed successfully when I start the server again, which is what I want. However, the script starts throwing Cannot find module errors when I uninstall the first or first two modules of the list that use the function requireOrInstall. That's right, the errors only occur when the script has to install either the first or the first two modules, not when only the second module needs installing.

In this example, the error will be thrown when I uninstall find-free-port, unless I move its require at least one spot down ¯\_(• _ •)_/¯

I've also tried adding a delay directly after the synchronous install to give it a little more breathing time with the following two lines:

var until = new Date().getTime() + 1000
while (new Date().getTime() < until) {}

The pause was there. It didn't fix anything.

@velocityzen came with the idea to check the cache, which I've now added to the script. It doesn't show anything out of the ordinary.

@vaughan's comment on another question noted that this exact error occurs when requiring a module twice. I've changed the script to use require.resolve(), but the error still remains.

Does anybody know what could be causing this?

Edit

Since the question has been answered, I'm posting the one-liner (139 characters!). It doesn't globally define child_modules, has no last try-catch and doesn't log anything in the console:

const req=async m=>{let r=require;try{r.resolve(m)}catch(e){r('child_process').execSync('npm i '+m);await setImmediate(()=>{})}return r(m)}

The name of the function is req() and can be used like in @alex-rokabilis' answer.

Answer 1

cp.execSync is an async call so try check if the module is installed in it's call back function. I have tried it, installation is clean now:

const cp = require('child_process')

function requireOrInstall (module) {
    try {
        require.resolve(module)
    } catch (e) {
        console.log(`Could not resolve "${module}"\nInstalling`)
        cp.execSync(`npm install ${module}`, () => {
            console.log(`"${module}" has been installed`)
            try {
                return require(module)
            } catch (e) {
                console.log(require.cache)
                console.log(e)
            }
        })

    }
    console.log(`Requiring "${module}"`)

}

const http    = require('http')
const path    = require('path')
const fs      = require('fs')
const ffp     = requireOrInstall('find-free-port')
const express = requireOrInstall('express')
const socket  = requireOrInstall('socket.io')

When node_modules not available yet :

When node_modules available already:

Answer 2

I think your best option is either:

• (ugly) to install package globally, instead of locally
• (best solution ?) to define YOUR new 'package repository installation', when installing, AND when requiring

First, you may consider using the npm-programmatic package.

Then, you may define your repository path with something like:

const PATH='/tmp/myNodeModuleRepository';

Then, replace your installation instruction with something like:

const npm = require('npm-programmatic');
npm.install(`${module}`, {
        cwd: PATH,
        save:true
}

Eventually, replace your failback require instruction, with something like:

return require(module, { paths: [ PATH ] });

If it is still not working, you may update the require.cache variable, for instance to invalide a module, you can do something like:

delete require.cache[process.cwd() + 'node_modules/bluebird/js/release/bluebird.js'];

You may need to update it manually, to add information about your new module, before loading it.

Answer 3

It seems that the require operation after an npm install needs a certain delay. Also the problem is worse in windows, it will always fail if the module needs to be npm installed. It's like at a specific event snapshot is already known what modules can be required and what cannot. Probably that's why require.cache was mentioned in the comments. Nevertheless I suggest you to check the 2 following solutions.

1) Use a delay

const cp = require("child_process");

const requireOrInstall = async module => {
  try {
    require.resolve(module);
  } catch (e) {
    console.log(`Could not resolve "${module}"\nInstalling`);
    cp.execSync(`npm install ${module}`);
    // Use one of the two awaits below
    // The first one waits 1000 milliseconds
    // The other waits until the next event cycle
    // Both work
    await new Promise(resolve => setTimeout(() => resolve(), 1000));
    await new Promise(resolve => setImmediate(() => resolve()));
    console.log(`"${module}" has been installed`);
  }
  console.log(`Requiring "${module}"`);
  try {
    return require(module);
  } catch (e) {
    console.log(require.cache);
    console.log(e);
  }
}

const main = async() => {
  const http = require("http");
  const path = require("path");
  const fs = require("fs");
  const ffp = await requireOrInstall("find-free-port");
  const express = await requireOrInstall("express");
  const socket = await requireOrInstall("socket.io");
}

main();

await always needs a promise to work with, but it's not needed to explicitly create one as await will wrap whatever it is waiting for in a promise if it isn't handed one.

2) Use a cluster

const cp = require("child_process");

function requireOrInstall(module) {
  try {
    require.resolve(module);
  } catch (e) {
    console.log(`Could not resolve "${module}"\nInstalling`);
    cp.execSync(`npm install ${module}`);
    console.log(`"${module}" has been installed`);
  }
  console.log(`Requiring "${module}"`);
  try {
    return require(module);
  } catch (e) {
    console.log(require.cache);
    console.log(e);
    process.exit(1007);
  }
}

const cluster = require("cluster");

if (cluster.isMaster) {
  cluster.fork();
  cluster.on("exit", (worker, code, signal) => {
    if (code === 1007) {
      cluster.fork();
    }
  });
} else if (cluster.isWorker) {
  // The real work here for the worker

  const http = require("http");
  const path = require("path");
  const fs = require("fs");
  const ffp = requireOrInstall("find-free-port");
  const express = requireOrInstall("express");
  const socket = requireOrInstall("socket.io");

  process.exit(0);
}

The idea here is to re-run the process in case of a missing module. This way we fully reproduce a manual npm install so as you guess it works! Also it seems more synchronous rather the first option, but a bit more complex.