Apache Ivy: Dependency Ivy FIles

Threequick questions regarding Apache Ivy:

(1) Throughout our projects, we use over 100 "common" JARs (log4j, junit, commons-cli, etc.). Do we have to write the ivy.xml ("module descriptor") files for all of them, or are there generic ones I can find in the ibiblio (or other) repo? To force your users to write their own ivy files for each dependency sounds pretty cruel and unusual to me.

(2) Are ivy files even required for a particular JAR, or does Ivy have defaults for when it looks in a repo for a dependency that doesn't have a corresponding ivy file?

(3) Is it possible to have all my dependencies in one folder (repo) and define 1 ivy.xml file that configures all of them inside it?

Thanks for any help here!

(1) Ivy files do not have to list every single jar. Some jars are dependencies of others and automatically pulled by ivy from the repository. No defaults are available. For a new project I normally generate my first ivy file (see the attached ant2ivy script)

(2) An ivy file is only required in an ivy repository when the jar has dependencies. Having said that it's good practice to have one. Personally I cheat and use a Maven repository manager like Nexus to store my jars.

(3) You can create a local repository using the filesystem resolver as follows in your settings file:

<ivysettings>
  <settings defaultResolver='maven-repos' />
  <resolvers>
    <chain name='maven-repos'>
      <ibiblio name='central' m2compatible='true' />
      <ibiblio name='spring-external' m2compatible='true' root='http://repository.springsource.com/maven/bundles/external' />
    </chain>
    <filesystem name='local'>
      <artifact pattern='/home/mark/tmp/petclinic/build/jars/[artifact]' />
    </filesystem>
  </resolvers>
  <modules>
    <module organisation='NA' name='mylibrary1.jar' resolver='local' />
    <module organisation='NA' name='mylibrary2.jar' resolver='local' />
    ..
  </modules>
</ivysettings>

When combined with the module declarations this makes the following possible in your ivy.xml file:

   <dependency org='NA' name='mylibrary1.jar' rev='NA' />
   <dependency org='NA' name='mylibrary2.jar' rev='NA' />

All other dependencies are retrieved from Maven repositories.

If you following the logic of my attached ant2ivy script you'll see I use this strategy with the jars I cannot identify using Sonatype's repository REST API

ant2ivy script

This is a rough and ready groovy script that performs a lookup of Sonatypes repository to identify jars in a specified directory

//
// Dependencies
// ============

import groovy.xml.MarkupBuilder

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Grapes([
    @Grab(group='org.slf4j', module='slf4j-simple', version='1.6.2') 
])

//
// Classes
// =======

class Ant2Ivy {

    Logger log = LoggerFactory.getLogger(this.class.name);
    String groupId
    String artifactId
    String repoUrl

    Ant2Ivy(groupId, artifactId) {
        this(groupId, artifactId, "http://repository.sonatype.org")
    }

    Ant2Ivy(groupId, artifactId, repoUrl) {
        this.groupId = groupId
        this.artifactId = artifactId
        this.repoUrl = repoUrl

        log.debug "groupId: {}, artifactId: {}", groupId, artifactId
    }

    //
    // Given a directory, find all jar and search Nexus
    // based on the file's checksum
    //
    // Return a data structure containing the GAV coordinates of each jar
    //
    def search(File inputDir) {
        def results = [:]
        results["found"] = []
        results["missing"] = []

        log.info "Searching: {} ...", repoUrl

        def ant = new AntBuilder()
        ant.fileset(id:"jars", dir:inputDir.absolutePath, includes:"**/*.jar")

        ant.project.references.jars.each {
            def jar = new File(inputDir, it.name)

            // Checksum URL
            ant.checksum(file:jar.absolutePath, algorithm:"SHA1", property:jar.name)

            def searchUrl = "${repoUrl}/service/local/data_index?sha1=${ant.project.properties[jar.name]}"
            log.debug "SearchUrl: {}, File: {}", searchUrl, jar.name

            // Search for the first result
            def searchResults = new XmlParser().parseText(searchUrl.toURL().text)
            def artifact = searchResults.data.artifact[0]

            if (artifact) {
                log.debug "Found: {}", jar.name
                results["found"].add([file:jar.name, groupId:artifact.groupId.text(), artifactId:artifact.artifactId.text(), version:artifact.version.text()])
            }
            else {
                log.warn "Not Found: {}", jar.name
                results["missing"].add([file:jar.name, fileObj:jar])
            }
        }

        return results
    }

    //
    // Given an input direcory, search for the GAV coordinates 
    // and use this information to write two XML files:
    //
    // ivy.xml          Contains the ivy dependency declarations
    // ivysettings.xml  Resolver configuration
    //
    def generate(File inputDir, File outputDir) {
        outputDir.mkdir()

        def antFile = new File(outputDir, "build.xml")
        def ivyFile = new File(outputDir, "ivy.xml")
        def ivySettingsFile = new File(outputDir, "ivysettings.xml")
        def localRepo = new File(outputDir, "jars")
        def results = search(inputDir)

        //
        // Generate the ant build file
        //
        log.info "Generating ant file: {} ...", antFile.absolutePath
        def antContent = new MarkupBuilder(antFile.newPrintWriter())

        antContent.project(name: "Sample ivy builde", default:"resolve", "xmlns:ivy":"antlib:org.apache.ivy.ant" ) {
            target(name:"resolve") {
                "ivy:resolve"()
            }
            target(name:"clean") {
                "ivy:cleancache"()
            }
        }

        // 
        // Generate the ivy file
        //
        log.info "Generating ivy file: {} ...", ivyFile.absolutePath
        def ivyConfig = new MarkupBuilder(ivyFile.newPrintWriter())

        ivyConfig."ivy-module"(version:"2.0") {
            info(organisation:this.groupId, module:this.artifactId) 
            configurations(defaultconfmapping:"default")
            dependencies() {
                results.found.each {
                    dependency(org:it.groupId, name:it.artifactId, rev:it.version, conf:"default->master")
                }
                results.missing.each {
                    dependency(org:"NA", name:it.file, rev:"NA")
                }
            }
        }

        // 
        // Generate the ivy settings file
        //
        log.info "Generating ivy settings file: {} ...", ivySettingsFile.absolutePath
        def ivySettings = new MarkupBuilder(ivySettingsFile.newPrintWriter())
        def ant = new AntBuilder()

        ivySettings.ivysettings() {
            settings(defaultResolver:"maven-repos") 
            resolvers() {
                chain(name:"maven-repos") {
                    // TODO: Make this list of Maven repos configurable
                    ibiblio(name:"central", m2compatible:"true")
                    ibiblio(name:"spring-external", m2compatible:"true", root:"http://repository.springsource.com/maven/bundles/external")
                }
                if (results.missing.size() > 0) {
                    filesystem(name:"local") {
                        artifact(pattern:"${localRepo.absolutePath}/[artifact]")
                    }
                }
            }
            if (results.missing.size() > 0) {
                modules() {
                    results.missing.each {
                        module(organisation:"NA", name:it.file, resolver:"local")
                        ant.copy(file:it.fileObj.absolutePath, tofile:"${localRepo.absolutePath}/${it.file}")
                    }
                }
            }
        }
    }
}

// 
// Main program
// ============
def cli = new CliBuilder(usage: 'ant2ivy')
cli.with {
    h longOpt: 'help', 'Show usage information'
    g longOpt: 'groupid',    args: 1, 'Module groupid', required: true
    a longOpt: 'artifactid', args: 1, 'Module artifactid', required: true
    s longOpt: 'sourcedir',  args: 1, 'Source directory containing jars', required: true
    t longOpt: 'targetdir',  args: 1, 'Target directory where write ivy build files', required: true
}

def options = cli.parse(args)
if (!options) {
    return
}

if (options.help) {
    cli.usage()
}

// 
// Generate ivy configuration
//
def ant2ivy = new Ant2Ivy(options.groupid, options.artifactid)
ant2ivy.generate(new File(options.sourcedir), new File(options.targetdir))

Script is run as follows:

groovy ant2ivy.groovy -g com.hello -a test -s targetdir/WEB-INF/lib -t build

When run against the petclinic sample it produced the following files

build/build.xml
build/ivy.xml
build/ivysettings.xml
build/jars/..
..

The jars directory contains those libraries which could not be found in a Maven repo.