Duplicate files copied in APK reference.conf

Question

I want to use my Android App as a “Producing client” for Kafka. After adding following dependecies:

// https://mvnrepository.com/artifact/org.apache.flink/flink-java
compile group: 'org.apache.flink', name: 'flink-java', version: '1.1.3'

// https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java_2.10
compile group: 'org.apache.flink', name: 'flink-streaming-java_2.10', version: '1.1.3'

// https://mvnrepository.com/artifact/org.apache.flink/flink-clients_2.10
compile group: 'org.apache.flink', name: 'flink-clients_2.10', version: '1.1.3'

// https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka-0.8_2.10
compile group: 'org.apache.flink', name: 'flink-connector-kafka-0.8_2.10', version: '1.1.3'

I get this error message:

What went wrong: Execution failed for task ':app:transformResourcesWithMergeJavaResForDebug'.

com.android.build.api.transform.TransformException: com.android.builder.packaging.DuplicateFileException: Duplicate files copied in APK reference.conf File1: /home/lidox/.gradle/caches/modules-2/files-2.1/com.typesafe.akka/akka-remote_2.10/2.3.7/d5603ae49eefde410f5e75f41b4f60bee61b7bfd/akka-remote_2.10-2.3.7.jar File2: /home/lidox/.gradle/caches/modules-2/files-2.1/com.typesafe.akka/akka-actor_2.10/2.3.7/b20f18109689918f05238deea29e681946d30ef7/akka-actor_2.10-2.3.7.jar

Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

I try to exclude one of the JARs, but no success:

packagingOptions {
    exclude 'META-INF/**'
    exclude 'META-INF/license.txt'
    exclude 'META-INF/LICENSE'
    exclude 'META-INF/notice.txt'
    exclude 'META-INF/NOTICE'
    exclude('**/akka-remote_2.10-*.jar')
    exclude 'reference.conf/**'
}

Update 1:

I need something what is doing this job in gradle:

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.1</version>
        <executions>
            <!-- Run shade goal on package phase -->
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
                <configuration>
                    <artifactSet>
                        <excludes>
                            <!-- This list contains all dependencies of flink-dist
                            Everything else will be packaged into the fat-jar
                            -->
                            <exclude>org.apache.flink:flink-annotations</exclude>
                            <exclude>org.apache.flink:flink-shaded-hadoop1</exclude>
                            <exclude>org.apache.flink:flink-shaded-hadoop2</exclude>
                            <exclude>org.apache.flink:flink-shaded-curator-recipes</exclude>
                            <exclude>org.apache.flink:flink-core</exclude>
                            <exclude>org.apache.flink:flink-java</exclude>
                            <exclude>org.apache.flink:flink-scala_2.10</exclude>
                            <exclude>org.apache.flink:flink-runtime_2.10</exclude>
                            <exclude>org.apache.flink:flink-optimizer_2.10</exclude>
                            <exclude>org.apache.flink:flink-clients_2.10</exclude>
                            <exclude>org.apache.flink:flink-avro_2.10</exclude>
                            <exclude>org.apache.flink:flink-examples-batch_2.10</exclude>
                            <exclude>org.apache.flink:flink-examples-streaming_2.10</exclude>
                            <exclude>org.apache.flink:flink-streaming-java_2.10</exclude>

                            <!-- Also exclude very big transitive dependencies of Flink

                            WARNING: You have to remove these excludes if your code relies on other
                            versions of these dependencies.

                            -->
                            <exclude>org.scala-lang:scala-library</exclude>
                            <exclude>org.scala-lang:scala-compiler</exclude>
                            <exclude>org.scala-lang:scala-reflect</exclude>
                            <exclude>com.amazonaws:aws-java-sdk</exclude>
                            <exclude>com.typesafe.akka:akka-actor_*</exclude>
                            <exclude>com.typesafe.akka:akka-remote_*</exclude>
                            <exclude>com.typesafe.akka:akka-slf4j_*</exclude>
                            <exclude>io.netty:netty-all</exclude>
                            <exclude>io.netty:netty</exclude>
                            <exclude>commons-fileupload:commons-fileupload</exclude>
                            <exclude>org.apache.avro:avro</exclude>
                            <exclude>commons-collections:commons-collections</exclude>
                            <exclude>org.codehaus.jackson:jackson-core-asl</exclude>
                            <exclude>org.codehaus.jackson:jackson-mapper-asl</exclude>
                            <exclude>com.thoughtworks.paranamer:paranamer</exclude>
                            <exclude>org.xerial.snappy:snappy-java</exclude>
                            <exclude>org.apache.commons:commons-compress</exclude>
                            <exclude>org.tukaani:xz</exclude>
                            <exclude>com.esotericsoftware.kryo:kryo</exclude>
                            <exclude>com.esotericsoftware.minlog:minlog</exclude>
                            <exclude>org.objenesis:objenesis</exclude>
                            <exclude>com.twitter:chill_*</exclude>
                            <exclude>com.twitter:chill-java</exclude>
                            <exclude>com.twitter:chill-avro_*</exclude>
                            <exclude>com.twitter:chill-bijection_*</exclude>
                            <exclude>com.twitter:bijection-core_*</exclude>
                            <exclude>com.twitter:bijection-avro_*</exclude>
                            <exclude>commons-lang:commons-lang</exclude>
                            <exclude>junit:junit</exclude>
                            <exclude>de.javakaffee:kryo-serializers</exclude>
                            <exclude>joda-time:joda-time</exclude>
                            <exclude>org.apache.commons:commons-lang3</exclude>
                            <exclude>org.slf4j:slf4j-api</exclude>
                            <exclude>org.slf4j:slf4j-log4j12</exclude>
                            <exclude>log4j:log4j</exclude>
                            <exclude>org.apache.commons:commons-math</exclude>
                            <exclude>org.apache.sling:org.apache.sling.commons.json</exclude>
                            <exclude>commons-logging:commons-logging</exclude>
                            <exclude>commons-codec:commons-codec</exclude>
                            <exclude>com.fasterxml.jackson.core:jackson-core</exclude>
                            <exclude>com.fasterxml.jackson.core:jackson-databind</exclude>
                            <exclude>com.fasterxml.jackson.core:jackson-annotations</exclude>
                            <exclude>stax:stax-api</exclude>
                            <exclude>com.typesafe:config</exclude>
                            <exclude>org.uncommons.maths:uncommons-maths</exclude>
                            <exclude>com.github.scopt:scopt_*</exclude>
                            <exclude>commons-io:commons-io</exclude>
                            <exclude>commons-cli:commons-cli</exclude>
                        </excludes>
                    </artifactSet>
                    <filters>
                        <filter>
                            <artifact>org.apache.flink:*</artifact>
                            <excludes>
                                <!-- exclude shaded google but include shaded curator -->
                                <exclude>org/apache/flink/shaded/com/**</exclude>
                                <exclude>web-docs/**</exclude>
                            </excludes>
                        </filter>
                        <filter>
                            <!-- Do not copy the signatures in the META-INF folder.
                            Otherwise, this might cause SecurityExceptions when using the JAR. -->
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                    <transformers>
                        <!-- add Main-Class to manifest file -->
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            <mainClass>reactiontest.Job</mainClass>
                        </transformer>
                    </transformers>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                </configuration>
            </execution>
        </executions>
    </plugin>

Alternativly one could try to build an APK using maven, maybe it will work. The pom could be a combination of this and the pom building Android Applications with Maven

Answer 1

First if all, look at this question: Kafka Producer on Android. It seems it's not yet possible to use kafka on mobile devices yet out of the box.

Regarding you question, did you already try to exclude the full name of the file?

exclude 'akka-actor_2.10-2.3.7.jar'

Answer 2

I changed it to exclude 'reference.conf' and the build worked.

You will need to concatenate all the reference.conf into application.conf but that should not be too hard.