combine javascript files at deployment in python

Question

I'm trying to reduce the number of scripts included in our website and we use buildout to handle deployments. Has anybody successfully implemented a method of combining and compressing scripts with buildout?

Answer 1

Here's a Python script I made that I use with all my heavy JavaScript projects. I'm using YUICompressor, but you can change the code to use another compressor.

import os, os.path, shutil

YUI_COMPRESSOR = 'yuicompressor-2.4.2.jar'

def compress(in_files, out_file, in_type='js', verbose=False,
             temp_file='.temp'):
    temp = open(temp_file, 'w')
    for f in in_files:
        fh = open(f)
        data = fh.read() + '\n'
        fh.close()

        temp.write(data)

        print ' + %s' % f
    temp.close()

    options = ['-o "%s"' % out_file,
               '--type %s' % in_type]

    if verbose:
        options.append('-v')

    os.system('java -jar "%s" %s "%s"' % (YUI_COMPRESSOR,
                                          ' '.join(options),
                                          temp_file))

    org_size = os.path.getsize(temp_file)
    new_size = os.path.getsize(out_file)

    print '=> %s' % out_file
    print 'Original: %.2f kB' % (org_size / 1024.0)
    print 'Compressed: %.2f kB' % (new_size / 1024.0)
    print 'Reduction: %.1f%%' % (float(org_size - new_size) / org_size * 100)
    print ''

    #os.remove(temp_file)

I use it like this (the below is just a code snippet, and assumes that the compress function exists in the current namespace):

SCRIPTS = [
    'app/js/libs/EventSource.js',
    'app/js/libs/Hash.js',
    'app/js/libs/JSON.js',
    'app/js/libs/ServiceClient.js',
    'app/js/libs/jquery.hash.js',
    'app/js/libs/Application.js',
    'app/js/intro.js',
    'app/js/jquery-extras.js',
    'app/js/settings.js',
    'app/js/api.js',
    'app/js/game.js',
    'app/js/user.js',
    'app/js/pages.intro.js',
    'app/js/pages.home.js',
    'app/js/pages.log-in.js',
    'app/js/pages.log-out.js',
    'app/js/pages.new-command.js',
    'app/js/pages.new-frame.js',
    'app/js/pages.not-found.js',
    'app/js/pages.register.js',
    'app/js/pages.outro.js',
    'app/js/outro.js',
    ]
SCRIPTS_OUT_DEBUG = 'app/js/multifarce.js'
SCRIPTS_OUT = 'app/js/multifarce.min.js'

STYLESHEETS = [
    'app/media/style.css',
    ]
STYLESHEETS_OUT = 'app/media/style.min.css'

def main():
    print 'Compressing JavaScript...'
    compress(SCRIPTS, SCRIPTS_OUT, 'js', False, SCRIPTS_OUT_DEBUG)

    print 'Compressing CSS...'
    compress(STYLESHEETS, STYLESHEETS_OUT, 'css')

if __name__ == '__main__':
    main()

Answer 2

A slightly different take on the solution proposed by Rushabh. Rather than a file based compress function, this is string based and somewhat simpler:

def jsmerge(file_names, debug=False):
"""combines several js files together, with optional minification"""
js = ""
for file_name in file_names:
    js += open(file_name).read()

# if debug is enabled, we skip the minification
if debug:
    return js
else:
    return jsmin(js)

Answer 3

If you're using WSGI middleware you could also use Fanstatic. It's probably some more work to integrate it into your stack than "simply" changing something in Buildout. The things you get with Fanstatic on the other hand are pretty good. It allows you to only send exactly the scripts you need for every page. It also does concatting (bundling) and minification of "resources" (JavaScript and CSS).

Answer 4

I created Minifpy : a tool to merge and minify JS and CSS file by using Python.

This tool use a very easy JSON configuration file to define if files must be merge , minify or not :

{
    "js": {
        "minify_files": [
            {"from": "static/file.js", "to":"static/file.min.js"},
        ],
        "merge_files": [
            {"from" : ["static/file1.js", "static/file2.js"], "to":"static/public.js", "to_min": "static/public.min.js"}
        ]
    },
    "css" : {
        "minify_files": [
            {"from": "static/file.css", "to":"static/file.min.css"},
        ],
        "merge_files": [
            {"from" : ["static/file1.css", "static/file2.css"], "to":"static/public.css", "to_min": "static/public.min.css"}
        ]
    }
}

Minifpy detect any modifications on JS/CSS files and merge/minify them automatically (useful for development).

Answer 5

The qooxdoo project comes with a Javascript compressor written in Python. Although it's tightly integrated with the framework, you should be able to utilize the compressor component. If you get the latest SDK there is a tool/bin/compile.py command line tool you can use to compress JS files, with various options (use the -h command line switch for help). I'm sure builtout can call this through a shell.

If you want to roll your own you can draw the compressor into your own Python code, by using the Python modules that come with the qooxdoo SDK (under tool/pylib/). It's not documented but you can look at the compile.py script how to achieve that.

Answer 6

Combining Blixt's solution with JS Min. Here is the code:

Just call the compress(in_files, out_file) method

import os, os.path, shutil

# This code is original from jsmin by Douglas Crockford, it was translated to
# Python by Baruch Even. The original code had the following copyright and
# license.
#
# /* jsmin.c
#    2007-05-22
#
# Copyright (c) 2002 Douglas Crockford  (www.crockford.com)
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal in
# the Software without restriction, including without limitation the rights to
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
# of the Software, and to permit persons to whom the Software is furnished to do
# so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# The Software shall be used for Good, not Evil.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
# */

from StringIO import StringIO

def jsmin(js):
    ins = StringIO(js)
    outs = StringIO()
    JavascriptMinify().minify(ins, outs)
    str = outs.getvalue()
    if len(str) > 0 and str[0] == '\n':
        str = str[1:]
    return str

def isAlphanum(c):
    """return true if the character is a letter, digit, underscore,
           dollar sign, or non-ASCII character.
    """
    return ((c >= 'a' and c <= 'z') or (c >= '0' and c <= '9') or
            (c >= 'A' and c <= 'Z') or c == '_' or c == '$' or c == '\\' or (c is not None and ord(c) > 126));

class UnterminatedComment(Exception):
    pass

class UnterminatedStringLiteral(Exception):
    pass

class UnterminatedRegularExpression(Exception):
    pass

class JavascriptMinify(object):

    def _outA(self):
        self.outstream.write(self.theA)
    def _outB(self):
        self.outstream.write(self.theB)

    def _get(self):
        """return the next character from stdin. Watch out for lookahead. If
           the character is a control character, translate it to a space or
           linefeed.
        """
        c = self.theLookahead
        self.theLookahead = None
        if c == None:
            c = self.instream.read(1)
        if c >= ' ' or c == '\n':
            return c
        if c == '': # EOF
            return '\000'
        if c == '\r':
            return '\n'
        return ' '

    def _peek(self):
        self.theLookahead = self._get()
        return self.theLookahead

    def _next(self):
        """get the next character, excluding comments. peek() is used to see
           if an unescaped '/' is followed by a '/' or '*'.
        """
        c = self._get()
        if c == '/' and self.theA != '\\':
            p = self._peek()
            if p == '/':
                c = self._get()
                while c > '\n':
                    c = self._get()
                return c
            if p == '*':
                c = self._get()
                while 1:
                    c = self._get()
                    if c == '*':
                        if self._peek() == '/':
                            self._get()
                            return ' '
                    if c == '\000':
                        raise UnterminatedComment()

        return c

    def _action(self, action):
        """do something! What you do is determined by the argument:
           1   Output A. Copy B to A. Get the next B.
           2   Copy B to A. Get the next B. (Delete A).
           3   Get the next B. (Delete B).
           action treats a string as a single character. Wow!
           action recognizes a regular expression if it is preceded by ( or , or =.
        """
        if action <= 1:
            self._outA()

        if action <= 2:
            self.theA = self.theB
            if self.theA == "'" or self.theA == '"':
                while 1:
                    self._outA()
                    self.theA = self._get()
                    if self.theA == self.theB:
                        break
                    if self.theA <= '\n':
                        raise UnterminatedStringLiteral()
                    if self.theA == '\\':
                        self._outA()
                        self.theA = self._get()


        if action <= 3:
            self.theB = self._next()
            if self.theB == '/' and (self.theA == '(' or self.theA == ',' or
                                     self.theA == '=' or self.theA == ':' or
                                     self.theA == '[' or self.theA == '?' or
                                     self.theA == '!' or self.theA == '&' or
                                     self.theA == '|' or self.theA == ';' or
                                     self.theA == '{' or self.theA == '}' or
                                     self.theA == '\n'):
                self._outA()
                self._outB()
                while 1:
                    self.theA = self._get()
                    if self.theA == '/':
                        break
                    elif self.theA == '\\':
                        self._outA()
                        self.theA = self._get()
                    elif self.theA <= '\n':
                        raise UnterminatedRegularExpression()
                    self._outA()
                self.theB = self._next()


    def _jsmin(self):
        """Copy the input to the output, deleting the characters which are
           insignificant to JavaScript. Comments will be removed. Tabs will be
           replaced with spaces. Carriage returns will be replaced with linefeeds.
           Most spaces and linefeeds will be removed.
        """
        self.theA = '\n'
        self._action(3)

        while self.theA != '\000':
            if self.theA == ' ':
                if isAlphanum(self.theB):
                    self._action(1)
                else:
                    self._action(2)
            elif self.theA == '\n':
                if self.theB in ['{', '[', '(', '+', '-']:
                    self._action(1)
                elif self.theB == ' ':
                    self._action(3)
                else:
                    if isAlphanum(self.theB):
                        self._action(1)
                    else:
                        self._action(2)
            else:
                if self.theB == ' ':
                    if isAlphanum(self.theA):
                        self._action(1)
                    else:
                        self._action(3)
                elif self.theB == '\n':
                    if self.theA in ['}', ']', ')', '+', '-', '"', '\'']:
                        self._action(1)
                    else:
                        if isAlphanum(self.theA):
                            self._action(1)
                        else:
                            self._action(3)
                else:
                    self._action(1)

    def minify(self, instream, outstream):
        self.instream = instream
        self.outstream = outstream
        self.theA = '\n'
        self.theB = None
        self.theLookahead = None

        self._jsmin()
        self.instream.close()

def compress(in_files, out_file, in_type='js', verbose=False,
             temp_file='.temp'):
    temp = open(temp_file, 'w')
    for f in in_files:
        fh = open(f)
        data = fh.read() + '\n'
        fh.close()

        temp.write(data)

        print ' + %s' % f
    temp.close()

    out = open(out_file, 'w')

    jsm = JavascriptMinify()
    jsm.minify(open(temp_file,'r'), out)

    out.close()

    org_size = os.path.getsize(temp_file)
    new_size = os.path.getsize(out_file)

    print '=> %s' % out_file
    print 'Original: %.2f kB' % (org_size / 1024.0)
    print 'Compressed: %.2f kB' % (new_size / 1024.0)
    print 'Reduction: %.1f%%' % (float(org_size - new_size) / org_size * 100)
    print ''

    os.remove(temp_file)