ClassNotFoundException on custom AndroidJUnitRunne

When I run connected Android tests I get the following issue:

java.lang.RuntimeException: Unable to instantiate instrumentation
ComponentInfo{my.package.app.debug.test/my.package.app.ui.AppTestRunner}:
java.lang.ClassNotFoundException: Didn't find class "my.package.app.ui.AppTestRunner" on 
path: DexPathList[[zip file "/system/framework/android.test.runner.jar", zip file "/data/app
my.package.app.debug.test-1/base.apk", zip file "/data/app/my.package.app.debug-1
base.apk"],nativeLibraryDirectories=[/data/app/my.package.app.debug.test-1/lib/arm, /data
app/my.package.app.debug-1/lib/arm, /data/app/my.package.app.debug.test-1/base.apk!/lib
armeabi-v7a, /data/app/my.package.app.debug-1/base.apk!/lib/armeabi-v7a, /vendor/lib, 
system/lib]]

Same issue no matter if I run the tests via command line or via Android Studio.

Part of my build.gradle

android {
    defaultConfig {
        testInstrumentationRunner "my.package.app.ui.AppTestRunner"
        multiDexEnabled true

// rest is missing

    }
}
dependencies {
    // Android Connected Tests
    androidTestCompile 'com.android.support.test.espresso:espresso-core:2.2.2'
    androidTestCompile 'com.android.support.test:runner:0.5'
    androidTestCompile 'com.android.support:multidex-instrumentation:1.0.1'
    androidTestCompile "com.android.support:support-annotations:${supportLibVersion}"
    androidTestCompile 'org.mockito:mockito-android:2.6.2'

// rest is missing
}

My custom test runner

package my.package.app.ui;

import android.app.Application;
import android.content.Context;
import android.support.test.runner.AndroidJUnitRunner;

public class AppTestRunner extends AndroidJUnitRunner {
    @Override
    public Application newApplication(ClassLoader cl, String className, Context context)
            throws InstantiationException, IllegalAccessException, ClassNotFoundException {
        return super.newApplication(cl, TestApplication.class.getName(), context);
    }
}

This is the generated test manifest in the build folder

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="my.package.app.debug.test" >

    <uses-sdk
        android:minSdkVersion="23"
        android:targetSdkVersion="23" />

    <instrumentation
        android:name="my.package.app.ui.AppTestRunner"
        android:functionalTest="false"
        android:handleProfiling="false"
        android:label="Tests for my.package.app.debug"
        android:targetPackage="my.package.app.debug" />

    <application>
        <uses-library android:name="android.test.runner" />
    </application>

</manifest>

And my app AndroidManifest.xml (all about activities removed)

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
          xmlns:tools="http://schemas.android.com/tools"
          package=“my.package.app"
          android:installLocation="auto">

    <application
        android:name=".App"
        android:allowBackup="false"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">

        <activity
            android:name=".ui.main.MainActivity"

       //... more stuff

    </application>

</manifest>

Same issue happens when I switch to testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner" or testInstrumentationRunner "android.support.multidex.MultiDexTestRunner"

Any hints on how to fix the issue would be appreciated!

I could fix the issue at the end by removing one test class I was creating. It's a shame that the project was compiling and the error I got was completely misleading.