Running tests from jar with “sbt testOnly” in SBT?

Question

As part of a CI setup the very first step is to create a package/jar using SBT dist. The following steps are to run unit, integration and functional tests with the dist-created jar.

Is it possible to do that with SBT?

Normally I use sbt testOnly "Unit.*", but that works in the context of a project. I can't find any documentation showing how to do this when there is already a jar.

I'm using ScalaTest and I know there is a runner for it I could use http://www.scalatest.org/user_guide/using_the_runner. But using SBT would be simpler, if that is possible.

As an example, something like this is what I am looking for:

sbt testOnly "Unit.* -jar myjar.jar"

Will my tests even be included in the jar when I use the following:

sbt dist

?

EDIT

1. I created a new folder

2. I added build.sbt with the following content:

   name := "abc"
   
   version := "1.0-SNAPSHOT"
   
   scalaVersion := "2.10.0"
   

3. I added the lib folder and copied my tests jar into it

4. I ran sbt testOnly Unit.*

5. It could not find any tests

EDIT 2

I tried with the following "proper" SBT file:

name := "ihs2tests"

version := "1.0-SNAPSHOT"

scalaVersion := "2.10.0"

unmanagedBase in Test := new java.io.File(".")

and moved test.jar into the root of the project. Again, no tests found.

Answer 1

Normally I use sbt testOnly "Unit.*", but that works in the context of a project. I can't find any documentation showing how to do this when there is already a jar.

The test-family tasks in SBT (with testOnly as an example) work with compile task that returns a list of compiled files through an sbt.inc.Analysis instance. I couldn't figure out how to amend it and inject the changed Analysis instance to testOnly so it knows the test I'm going to run exists.

I'm proposing another solution - a trick.

Package the tests classes to a jar with test:package task as follows:

[test-lib]> test:package
[info] Updating {file:/Users/jacek/sandbox/so/test-lib/}test-lib...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/jacek/sandbox/so/test-lib/target/scala-2.10/test-classes...
[info] Packaging /Users/jacek/sandbox/so/test-lib/target/scala-2.10/test-lib_2.10-0.1-SNAPSHOT-tests.jar ...
[info] Done packaging.
[success] Total time: 9 s, completed Mar 4, 2014 11:34:13 PM

When you have the test jar, you can execute the test framework on the command line without SBT (I assume you use ScalaTest given the scalatest tag, but I'll use Specs2). Read the documentation of the test framework on how to do it and for Specs2 it's specs2.run as described in Console output.

Executing tests from the test jar requires defining a proper CLASSPATH that may or may not be easy to get right. That's where SBT can be of great help - to manage dependencies and hence the CLASSPATH.

Create another SBT project and save the test jar in lib subdirectory to have it on CLASSPATH (as described in Unmanaged dependencies).

Dependencies in lib go on all the classpaths (for compile, test, run, and console).

Add a sample build.sbt where you define your test framework as a dependency of the project. For Specs2 it's as follows (I used the default configuration as described in the Specs2 home page):

libraryDependencies += "org.specs2" %% "specs2" % "2.3.8" % "test"

scalacOptions in Test ++= Seq("-Yrangepos")

The trick is to execute the main class of the test framework, e.g. specs2.run for Specs2, as if the class were executed on the command line. SBT helps with test:runMain.

[my-another-project]> test:runMain specs2.run HelloWorldSpec
[info] Running specs2.run HelloWorldSpec
HelloWorldSpec

The 'Hello world' string should
+ contain 11 characters
+ start with 'Hello'
+ end with 'world'

Total for specification HelloWorldSpec
Finished in 62 ms
3 examples, 0 failure, 0 error

Exception: sbt.TrapExitSecurityException thrown from the UncaughtExceptionHandler in thread "run-main-0"
[success] Total time: 5 s, completed Mar 4, 2014 11:15:14 PM

Don't worry about this Exception as it comes from SBT that catches exit from Specs2 (after test execution) so SBT remains up.

Answer 2

It seems that SBT can't read jar file without any additional/manual setup - I might be wrong but I didn't find anything in the docs. So I tried something like this to make the task simpler:

unzip some-test.jar

java -jar sbt-launch.jar \
  'set scalaSource in Test := new java.io.File(".")' \
  'set fullClasspath in Test += Attributed.blank(file("."))' \
  'test'

This runs without errors but does not find tests.

If I add 'set includeFilter in (Test, unmanagedSources) := "*Suite*.class"' to force it to find tests it obviously fails because it expects *.scala files, not the compiled *.class files.

I'm not an SBT expert, but I think this must be close to a solution. There must be a way to read all files from a jar path programmatically and then tell test framework to use *.class files.

At this point is seems more reasonable to run tests using a Scalatest test runner or from sbt using the project.

If you wish to dig deeper take a look at SBT source code and the default sbt shell script that does lots of setup before it runs the sbt launcher jar.