JavaScript dependency management

Question

I am currently maintaining a large number of JS files and the dependency issue is growing over my head. Right now I have each function in a separate file and I manually maintain a database to work out the dependencies between functions.

This I would like to automate. For instance if I have the function f

Array.prototype.f = function() {};

which is referenced in another function g

MyObject.g = function() {
    var a = new Array();
    a.f();
};

I want to be able to detect that g is referencing f.

How do I go about this? Where do I start? Do I need to actually write a compiler or can I tweak Spidermonkey for instance? Did anyone else already do this?

Any pointers to get me started is very much appreciated

Thanks Dok

Answer 1

As @bobince already suggested, doing static analysis on a JavaScript program is a close to impossible problem to crack. Google Closure compiler does it to some extent but then it also relies on external help from JSDoc comments.

I had a similar problem of finding the order in which JS files should be concatenated in a previous project, and since there were loads of JS files, manually updating the inclusion order seemed too tedious. Instead, I stuck with certain conventions of what constitutes a dependency for my purposes, and based upon that and using simple regexp :) I was able to generated the correct inclusion order.

The solution used a topological sort algorithm to generate a dependency graph which then listed the files in the order in which they should be included to satisfy all dependencies. Since each file was basically a pseudo-class using MooTools syntax, there were only 3 ways dependencies could be created for my situation.

1. When a class Extended some other class.
2. When a class Implemented some other class.
3. When a class instantiated an object of some other class using the new keyword.

It was a simple, and definitely a broken solution for general purpose usage but it served me well. If you're interested in the solution, you can see the code here - it's in Ruby.

If your dependencies are more complex, then perhaps you could manually list the dependencies in each JS file itself using comments and some homegrown syntax such as:

// requires: Array
// requires: view/TabPanel
// requires: view/TabBar

Then read each JS file, parse out the requires comments, and construct a dependency graph which will give you the inclusion order you need.

Answer 2

JSAnalyse uses static code analysis to detect dependencies between javascript files: http://jsanalyse.codeplex.com/

It also allows you to define the allowed dependencies and to ensure it during the build, for instance. Of course, it cannot detect all dependencies because javascript is dynamic interpretet language which is not type-safe, like already mentioned. But it at least makes you aware of your javascript dependency graph and helps you to keep it under control.

Answer 3

Whilst you could theoretically write a static analysis tool that detected use of globals defined in other files, such as use of MyObject, you couldn't realistically track usage of prototype extension methods.

JavaScript is a dynamically-typed language so there's no practical way for any tool to know that a, if passed out of the g function, is an Array, and so if f() is called on it there's a dependency. It only gets determined what variables hold what types at run-time, so to find out you'd need an interpreter and you've made yourself a Turing-complete problem.

Not to mention the other dynamic aspects of JavaScript that completely defy static analysis, such as fetching properties by square bracket notation, the dreaded eval, or strings in timeouts or event handler attributes.

I think it's a bit of a non-starter really. You're probably better of tracking dependencies manually, but simplifying it by grouping related functions into modules which will be your basic unit of dependency tracking. OK, you'll pull in a few more functions that you technically need, but hopefully not too much.

It's also a good idea to namespace each module, so it's very clear where each call is going, making it easy to keep the dependencies in control manually (eg. by a // uses: ThisModule, ThatModule comment at the top).

Since extensions of the built-in prototypes are trickier to keep track of, keep them down to a bare minimum. Extending eg. Array to include the ECMAScript Fifth Edition methods (like indexOf) on browsers that don't already have them is a good thing to do as a basic fixup that all scripts will use. Adding completely new arbitrary functionality to existing prototypes is questionable.

Answer 4

Have you tried using a dependency manager like RequireJS or LabJS? I noticed no one's mentioned them in this thread.

From http://requirejs.org/docs/start.html:

Inside of main.js, you can use require() to load any other scripts you need to run:

require(["helper/util"], function(util) {
    //This function is called when scripts/helper/util.js is loaded.
    //If util.js calls define(), then this function is not fired until
    //util's dependencies have loaded, and the util argument will hold
    //the module value for "helper/util".
});

You can nest those dependencies as well, so helper/util can require some other files within itself.

Answer 5

It would be nice to have a tool that can automatically detect those dependencies for you and choose how they are loaded. The best solutions today are a bit cruder though. I created a dependency manager for my particular needs that I want to add to the list (Pyramid Dependency Manager). It has some key features which solve some unique use cases.

1. Handles other files (including inserting html for views...yes, you can separate your views during development)
2. Combines the files for you in javascript when you are ready for release (no need to install external tools)
3. Has a generic include for all html pages. You only have to update one file when a dependency gets added, removed, renamed, etc

Some sample code to show how it works during development.

File: dependencyLoader.js

//Set up file dependencies
Pyramid.newDependency({
    name: 'standard',
    files: [
    'standardResources/jquery.1.6.1.min.js'
    ]
});

Pyramid.newDependency({
name:'lookAndFeel',
files: [
    'styles.css',
    'customStyles.css',
    'applyStyles.js'
    ]
});

Pyramid.newDependency({
name:'main',
files: [
    'createNamespace.js',
    'views/buttonView.view', //contains just html code for a jquery.tmpl template
    'models/person.js',
    'init.js'
    ],
    dependencies: ['standard','lookAndFeel']
});

Html Files

<head>
    <script src="standardResources/pyramid-1.0.1.js"></script>
    <script src="dependencyLoader.js"></script>
    <script type="text/javascript">
        Pyramid.load('main');
    </script>
</head>

It does require you to maintain a single file to manage dependencies. I am thinking about creating a program that can automatically generate the loader file for you based on includes in the header but since it handles many different types of dependencies, maintaining them in one file might actually be better.

Answer 6

I have written a tool to do something like this: http://github.com/damonsmith/js-class-loader

It's most useful if you have a java webapp and you structure your JS code in the java style. If you do that, it can detect all of your code dependencies and bundle them up, with support for both runtime and parse-time dependencies.