wirejs and dojo using the dojo build system (2

Question

(This is the same question as wirejs and dojo using the dojo build system, but with more detail about the problem and tried solutions. The duplicate question is created because this was suggested in a comment).

When building a dojo application that uses wire, the dojo loader throws an undefinedModule Error for "./lib/context" that we can't get rid of.

I added wire to a large, working dojo project using a git submodule. It turned out that also cujojs when and cujojs meld are required. I added them as a git submodule too.

In this project, libraries are not next to the application folder (src/app), but one level deep, in src/lib. So I have src/lib/wire, src/lib/when, src/lib/meld, next to src/lib/dgrid, etc. dojo libraries are 2 levels deep: src/lib/dojo/dojo, src/lib/dojo/dijit, src/lib/dojo/dojox and src/lib/dojo/util.

Versions:

• dojo 1.10.4
• wire 0.10.9
• when 3.7.2
• meld 1.3.1

Things work in development (unbuild) once I add the following package definitions to dojoConfig:

var dojoConfig = (function() {

  [...]

  return {

    [...]

    packages: [

      [...]

      {name: "wire", location: "../../wire", main: "wire"},
      {name: "when", location: "../../when", main: "when" },
      {name: "meld", location: "../../meld", main: "meld" },

      [...]

    ],

    [...]
  };

})();

Note that it was necessary to add the main entries. This makes it possible, e.g., to refer to when/when.js as just "when" in a dependency list, instead of "when/when", which the cujojs code does internally.

So, this works in development mode.

Next, I tried to get the build working.

In the build.profile.js, I again added the package references. The build fails that way in a number of ways.

First, for packages, the dojo builder expects a dojoBuild property in the package.json files, that refers to a <mypackage>.profile.js or package.js file, that essentially defines the build configuration for the packages. This build configuration is most often (it is in all our packages) just a resourceTags object, with functions that define what files are AMD resources in the package, what files are tests, and which should be copied as is.

The cujojs packages do not have such build definitions usable for the dojo builder.

The builder complains first that there is no build configuration for these packages, and secondly that resources are not flagged as AMD resources (which is the main reason of being for this build configuration files).

Since I do not want to change files of an external library I use, I am not inclined to add a dojoBuild property and build configuration file to the cujojs packages, so I sought a workaround.

After stepping through the dojo builder code, I found that the builder complains about a package.json file not having a dojoBuild property, but finally just looks at the internal package definition structure to get the resourceTags object. The internal package definition structure starts out with the package definition from the main build profile. So, I ended up piggy-backing a resourceTags object there:

[...]

/*
 wire, when and meld don't have a good build profile set up.
 Here, we doctor a resourceTags (i.e., the core of a build profile). We set it on the package definition.
 This is an unsupported hack. The builder will still complain about there not being a build profile, but
 it will use these definitions when building those packages.

 Note: // see https://github.com/cujojs/when/wiki/Using-with-Dojo does not work!
*/
var generalResourceTags = (function () {

  function isTest(filename, mid) {
    return filename.indexOf("test/") >= 0;
  }

  function isCopyOnly(filename, mid) {
    return filename.indexOf(".html") >= 0;
  }

  function isAmd(filename, mid) {
    return filename.indexOf(".json") < 0 && filename.indexOf(".js") >= 0 && filename.indexOf(".profile.js") < 0;
  }

  function isGivingTroubleButUnused(filename, mid) {
    return /^wire\/jquery\//.test(mid) ||
           mid === "wire/sizzle" ||
           /^when\/build\//.test(mid) ||
           /^when\/es6-shim\//.test(mid) ||
           mid === "when/generator"; // see https://github.com/cujojs/when/issues/429
  }

  return {
    test: function (filename, mid) {
      return isTest(filename, mid);
    },

    copyOnly: function (filename, mid) {
      return isCopyOnly(filename, mid) || isGivingTroubleButUnused(filename, mid);
    },

    amd: function (filename, mid) {
      return !isTest(filename, mid) && !isCopyOnly(filename, mid) && isAmd(filename, mid) && !isGivingTroubleButUnused(filename, mid);
    }
  }
})();

var profile = {
  releaseName: releaseName,
  releaseDir: "../../../../release",
  action: 'release',
  cssOptimize: 'comments',
  mini: true,
  optimize: 'closure',
  layerOptimize: 'closure',
  stripConsole: 'normal',
  selectorEngine: 'acme',
  useSourceMaps: false,

  [...]  

  packages: [

    [...]

    {name: "wire", location: "../../wire", main: "wire", destLocation: "./lib/wire", resourceTags: generalResourceTags},
    {name: "when", location: "../../when", main: "when", destLocation: "./lib/when", resourceTags: generalResourceTags},
    {name: "meld", location: "../../meld", main: "meld", destLocation: "./lib/meld", resourceTags: generalResourceTags},

    [...]

  ],
  layers: {

    [...]

  },
  staticHasFeatures: {
    "config-publishRequireResult": false,
    "dijit-legacy-requires": false,
    "dojo-debug-messages": false,
    "dojo-firebug": false,
    "dojo-log-api": 0,
    "dojo-mobile-parser": false,
    "dojo-moduleUrl": false,
    "dojo-parser": true,
    "dojo-publish-privates": 0,
    "dojo-test-sniff": 0,
    "dojo-trace-api": 0,
    "dom-addeventlistener": true,
    "extend-dojo": true,
    "host-browser": true,
    "mvc-bindings-log-api": false,

    [...]

  }
};

Note the extra resourceTags property in the package definitions for wire, when and meld. I kept things "simple", and just used one general object to handle all 3 cases, generalResourceTags. This is a pretty standard resourceTags setup, apart from isGivingTroubleButUnused, which I'll get to in a moment.

Note that I tried the solution mentioned explicity at https://github.com/cujojs/when/wiki/Using-with-Dojo for when, and that it did not work.

This way, we get a pretty decent report from the dojo builder, but it mentions some missing resources. E.g., some modules require jquery. But, those modules aren't used in this project.

Note that the solution mentioned in wirejs and dojo using the dojo build system, piggy-backing packageJson might be better. I haven't tested it.

I ended up introducing the isGivingTroubleButUnused to copy these modules without handling them, as we don't use them anyway. Note that 1 of the issues is in when/generator. The Closure compiler sees a syntax error there (see https://github.com/cujojs/when/issues/429).

The net result of this setup is a build without errors.

And then, we want to test this ... and it fails. The first error we get is indeed the dojo loader throwing an undefinedModule Error for "./lib/context". The only place this occurs in indeed on line 23 of wire/wire.js:

createContext = require('./lib/context');

This is intended to resolve to wire/lib/context.js, and obviously this works in development (unbuild) mode.

In any case, require is intended to be a context-sensitive require (see dojotoolkit.org /documentation/tutorials/1.10/modules_advanced/, "Conditionally requiring modules"), as the module reference is relative. But that should be no reason why it works unbuild, and doesn't work build.

Next, I tried to just copy all the wire, when and meld resources. If a resource is not included in the dojo build of a layer, it just falls back to async loading. So, this could work:

var generalResourceTags = (function () {

  function isTest(filename, mid) {
    return filename.indexOf("test/") >= 0;
  }

  [...]

  return {
    test: function (filename, mid) {
      return isTest(filename, mid);
    },

    copyOnly: function (filename, mid) {
      return !isTest(filename, mid);
    },

    amd: function (filename, mid) {
      return false;
    }
  }
})();

The build now passes, and copies what is needed. There are more errors, of course, where the app code reference wire, but that is to be expected (error(311) Missing dependency. module: MY_MODULE; dependency: wire!SOME_PACKAGE/_wire-serverRequests).

Yet, the browser gives the same error, in the same place: an undefinedModule Error for "./lib/context".

The working hypothesis at this point that the function used by wire under the name require is not a context-sensitive require in the build version, but is in the unbuild version.

This is not true. First, I changed the wire code or a test to read

createContext = require('wire/lib/context');

making the reference absolute. Same problem.

Then, I tried to debug using sourceMaps (useSourceMaps = true). This is a nightmare, but I believe I see that the require used is the same, the context-sensitive require.

Maybe "preloading" works? So, in the top HTML page, I surrounde all code with

require(["wire/lib/context"], function() {

  [...]

});

This makes sure context is loaded before we do anything else. Same error.

Next, I added logging to the code in wire/lib/context

console.error("Defining context");

var when, mixin, loaderAdapter, relativeLoader, Container, specUtils;

when = require('when');
console.error("Loaded when");
mixin = require('./object').mixin;
console.error("Loaded ./object");
loaderAdapter = require('./loader/adapter');
console.error("Loaded ./loader/adapter");
relativeLoader = require('./loader/relative');
console.error("Loaded ./loader/relative");
Container = require('./Container');
console.error("Loaded ./Container");
specUtils = require('./specUtils');

console.error("Defined context. Returning.");

(using error, because other messages are stripped in the build).

In the unbuild version, I see all messages. In the build version, I see only "Defining context" before the error occurs. So the problem is not with "loading" wire/lib/context, but in defining it, and probably in loading or defining when!

So, I applied to same trick to when

console.error("Defining when");

var timed = require('./lib/decorators/timed');
console.error("Loaded lib/decorators/timed");
var array = require('./lib/decorators/array');
console.error("Loaded lib/decorators/array");
var flow = require('./lib/decorators/flow');
console.error("Loaded lib/decorators/flow");
var fold = require('./lib/decorators/fold');
console.error("Loaded lib/decorators/fold");
var inspect = require('./lib/decorators/inspect');
console.error("Loaded lib/decorators/inspect");
var generate = require('./lib/decorators/iterate');
console.error("Loaded lib/decorators/iterate");
var progress = require('./lib/decorators/progress');
console.error("Loaded lib/decorators/progress");
var withThis = require('./lib/decorators/with');
console.error("Loaded lib/decorators/with");
var unhandledRejection = require('./lib/decorators/unhandledRejection');
console.error("Loaded lib/decorators/unhandledRejection");
var TimeoutError = require('./lib/TimeoutError');
console.error("Loaded lib/TimeoutError");
[...]

Now there is a surprise in the unbuild version. The output is:

Defining when
Loaded lib/decorators/timed
Loaded lib/decorators/array
Loaded lib/decorators/flow
Loaded lib/decorators/fold
Loaded lib/decorators/inspect
Loaded lib/decorators/iterate
Loaded lib/decorators/progress
Loaded lib/decorators/with
Loaded lib/decorators/unhandledRejection
Loaded lib/TimeoutError
Loaded lib/Promise
Loaded lib/apply
Defined when
Defining context
Loaded when
Loaded ./object
Loaded ./loader/adapter
Loaded ./loader/relative
Loaded ./Container
Defined context. Returning.

Which means when definition started before context definition. This is weird, because I see no code that can tell the loader that when is required in context, before the line

when = require('when');

is executed, which comes after the logging of "Defining context".

In the build version, we still only get

Defining context

before the error occurs! This behaviour is clearly different!

Next, I removed the preloading code. Unbuild code gives the same output, build code now has no messages before the error occurs.

So in any case, this wire code is doing things I don't expect and understand during loading.

I do hope this extra detail will trigger someone to get us on the right track ...