filesystem resolver and transitive dependencies /

Question

I have setup Ivy with a filesystem resolver as explained in this answer. I.e. placing my jars in a 'lib' directory and configuring a filesystem resolver to pick them up from there (instead of using the default ibiblio resolver).

The file ivysettings.xml is as follows:

<ivysettings>
  <caches defaultCacheDir="${ivy.settings.dir}/cache"/>
  <settings defaultResolver="local"/>
  <resolvers>
     <filesystem name="local">
        <artifact pattern="${ivy.settings.dir}/lib/[artifact]-[revision].[ext]"/>
     </filesystem>
  </resolvers>
</ivysettings>

My ivy.xml is also very simple and for test purposes I only define a single configuration:

<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0" 
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
    <info organisation="foo" module="foo" status="foo"/>
    <configurations>
        <conf name="theConf"  description="just one configuration"/>
    </configurations>
    <dependencies>
        <dependency org="org.apache.commons" name="commons-lang3" rev="3.1" conf="theConf->default"/>
    </dependencies>
</ivy-module>

The above works and the dependency is properly retrieved from the lib directory. However I've noticed that if I change the dependency element in the ivy.xml file to fetch the "master" as opposed to the "default" configuration:

<dependency org="org.apache.commons" name="commons-lang3" rev="3.1" conf="theConf->master"/>

… then it no longer works and I get the following:

resolve: [ivy:resolve] :: Apache Ivy 2.4.0-local-20161112135640 -
20161112135640 :: http://ant.apache.org/ivy/ :: [ivy:resolve] ::
loading settings :: file =
/home/mperdikeas/play/ivy-with-local-filesystem/ivysettings.xml
[ivy:resolve] :: resolving dependencies :: foo#foo;working@mp-t420
[ivy:resolve]   confs: [theConf] [ivy:resolve]  found
org.apache.commons#commons-lang3;3.1 in local [ivy:resolve] ::
resolution report :: resolve 66ms :: artifacts dl 0ms
---------------------------------------------------------------------
|                  |            modules            ||   artifacts   |
|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
|      theConf     |   1   |   0   |   0   |   0   ||   0   |   0   |
---------------------------------------------------------------------
[ivy:resolve]  [ivy:resolve] :: problems summary :: [ivy:resolve] ::::
WARNINGS [ivy:resolve]
    :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve]        ::   
UNRESOLVED DEPENDENCIES         :: [ivy:resolve]
    :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve]        ::
org.apache.commons#commons-lang3;3.1: configuration not found in
org.apache.commons#commons-lang3;3.1: 'master'. It was required from
foo#foo;working@mp-t420 theConf [ivy:resolve]
    :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] 
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

So even though the module is found it's "master" configuration is not found.

To make this work I either need to change theConf->master to theConf->default or simply remove the conf mapping altogether.

Further reflecting upon this I realized that I also don't understand how a filesystem resolver that's picking up jars laid out flat insider a directory is supposed to distinguish between different configurations and fetch, e.g. only compile time versus transitive dependencies of a module.

At any rate having to always use the "default" configuration causes me problems as I have to modify my ivy.xml file. When I resolve using the ibiblio resolver I typically have in my ivy.xml either:

conf="compile->master"

… or:

conf="runtime->default"

… depending on whether I want the transitive dependencies or not. I would like to keep the same ivy.xml file regardless of the resolver I have configured to use in ivysettings.xml.

My questions are:

1. is it possible to keep the same ivy.xml file for both the ibiblio and the filesystem resolver and what to do with "master" configurations in such a case?
2. how are configurations understood by a filesystem resolver that's picking up jars that are laid flat inside a directory?
3. is it possible to configure a filesystem resolver such that transitive dependencies can also be fetched with a dependency and what should be the file system structure in such a case?
4. I am simply copying jars inside the directory that the filesystem resolver is looking at. Should I be using some importation mechanism instead (e.g. to create the proper directory structure that might perhaps allow the resolver to also fetch transitive dependencies).

Answer 1

I gave up using a flat directory structure with just a bunch of jars. It is clear to me now that in addition to the jars, declarative files need to be present to properly identify the dependencies of each module.

What I have discovered works is to use the ivy:install task to import the dependencies I need from the ibiblio resolver into my local filesystem resolver. More or less as described in this post.

I've created a build-import-Maven-dependencies-to-local.xml script to automate this process. With some added "cleverness" (which is not crucial and you can dispense with) for aligning all the dependencies visually line after line it looks like the following (the gist is the last target, "install"):

<project name="local repository importation" default="install-deps-locally"
         xmlns:contrib="http://net.sf.antcontrib"
         xmlns:ivy="antlib:org.apache.ivy.ant">

    <taskdef uri="http://net.sf.antcontrib" resource="net/sf/antcontrib/antlib.xml" classpath="${basedir}/tools/ant-contrib.jar" />

    <target name="install-deps-locally" description="local at non default location">
        <!-- "iw" starts for "Import Wrapper" and "d" stands for "dependency" -->
        <antcall target="iw"><param name="d" value="javax.servlet              javax.servlet-api   3.1.0"   /></antcall>
        <antcall target="iw"><param name="d" value="com.google.code.gson       gson                2.8.0"   /></antcall>
        <antcall target="iw"><param name="d" value="junit                      junit               4.12"    /></antcall>
    </target>

    <target name="iw"> <!-- "iw" stands for "Import Wrapper" and "d" stands for "dependency" -->
        <property name="REGULAR_EXPRESSION" value="(\S*)(\s*)(\S*)(\s*)(\S*)"/>
        <contrib:propertyregex property="organisation"
                       input="${d}"
                       regexp="${REGULAR_EXPRESSION}"
                       select="\1"
                       casesensitive="false"/>
        <contrib:propertyregex property="module"
                       input="${d}"
                       regexp="${REGULAR_EXPRESSION}"
                       select="\3"
                       casesensitive="false"/>
        <contrib:propertyregex property="revision"
                       input="${d}"
                       regexp="${REGULAR_EXPRESSION}"
                       select="\5"
                       casesensitive="false"/>
        <antcall target="install">
            <param name="organisation" value="${organisation}"/>
            <param name="module"       value="${module}"/>
            <param name="revision"     value="${revision}"/>
        </antcall>
    </target>

    <target name="install" description="import module from public Maven repository into local repository">
        <ivy:settings id="ivysettings-ibiblio-to-local" file="ivysettings-ibiblio-to-local.xml"/>
        <ivy:install settingsRef="ivysettings-ibiblio-to-local"
                     organisation="${organisation}"
                     module="${module}"
                     revision="${revision}"
                     from="public"
                     to="local"
                     transitive="true"
                     overwrite="true"/>
    </target>    
</project>

Resolvers "public" and "local" are defined in the ivysettings.file and correspond to ibiblio and filesystem respectively.

This creates the correct structure and places ivy-x.y.z.xml files that provide the necessary information. This is a partial example of the directory structure that got created in my system and the files that are present.

$ tree repo/ | head -20
repo/
├── avalon-framework
│   └── avalon-framework
│       ├── ivys
│       │   ├── ivy-4.1.5.xml
│       │   ├── ivy-4.1.5.xml.md5
│       │   └── ivy-4.1.5.xml.sha1
│       └── jars
│           ├── avalon-framework-4.1.5.jar
│           ├── avalon-framework-4.1.5.jar.md5
│           └── avalon-framework-4.1.5.jar.sha1
├── com.google.code.findbugs
│   └── jsr305
│       ├── ivys
│       │   ├── ivy-1.3.9.xml
│       │   ├── ivy-1.3.9.xml.md5
│       │   └── ivy-1.3.9.xml.sha1
│       └── jars
│           ├── jsr305-1.3.9.jar
│           ├── jsr305-1.3.9.jar.md5

Once this is in place the filesystem resolver works like a charm for both transitive (default) and compile-time (master) dependencies.