How to cut a long ScalaTest spec to pieces

问题:

I'm testing a REST API, and the code goes like this:

1. Setting up stuff, populating a database using PUSH calls
2. Testing API a
3. Testing API b ...

The code is currently in one rather huge FlatSpec:

class RestAPITest extends FlatSpec
  with Matchers
  with ScalatestRouteTest
  with SprayJsonSupport

I would like to chop the "Testing API a/b/..." parts out, to have the code more manageable. Trying to do that seems like a no-no: what's the type of it - how to pass that on, etc. etc.

So, what's the recommended way to go about such stuff.

The a/b/... tests could be run in parallel, once the basic setup has succeeded.

I'm currently using assume within the a/b/... tests to make them cancel if the initialization failed.

Should I look at "fixtures" or what for this? Have tried BeforeAndAfterAll earlier, but didn't really get it working for me.

Thanks for the pointers / opinions. How do you keep your test suites short?

回答1:

I'd say mixing in BeforeAndAfter or BeforeAndAfterAll are among the most intuitive ways to reduce duplication in a scenario where you want to do: "Setup" -> "run test1" -> "Setup" -> "run test2", "Setup" being (mostly) the same.

Suppose we have a nasty, hard to test Database:

object Database {
  private var content: List[Int] = Nil

  def add(value: Int) = content = value :: content

  def remove(): Unit = content = if (content.nonEmpty) content.tail else Nil

  def delete(): Unit = content = Nil

  def get: Option[Int] = content.headOption

  override def toString = content.toString()
}

It's a singleton (so we can't just instantiate a new Database for each test) and its mutable (so if the first test changes something, it would affect the second test).

Obviously, it would be more desireable to not have such a structure (e.g., it would be much nicer to work with the List that implements the Database in this example), but suppose we cannot simply change this structure.

Edit: Note that, in such a case, it is impossible (at least I can't think of a way) to run tests mutating the same singleton instance in parallel.

To still be able to test it, we need to have a clean state before running each test. Assuming we want to populate the database with the same values for each test, we could let our base testsuite class extend BeforeAndAfter. Note: There exists two traits: BeforeAndAfter, which defines before and after that are run before and after the execution of each test case, and BeforeAndAfterAll, which is different in that it defines methods that are run before and after each test suite.

class RestAPITest extends FlatSpec with ShouldMatchers with BeforeAndAfter {
  before {
    Database.delete()
    Database.add(4)
    Database.add(2)
  }
}

Now we can have a test suite ATest extend this base class:

class ATest extends RestAPITest {
  "The database" should "not be empty" in {
    Database.get shouldBe defined
  }
  it should "contain at least two entries" in {
    Database.remove()

    Database.get shouldBe defined
  }
  it should "contain at most two entries" in {
    Database.remove()
    Database.remove()

    Database.get should not be defined
  }
}

At the beginning of each test, the database contains the two values 4 and 2. We can now have other testsuits extend this base class:

class BTest extends RestAPITest {
  "The contents of the database" should "add up to 6" in {
    getAll.sum shouldBe 6
  }

  "After adding seven, the contents of the database" should "add up to 13" in {
    Database.add(7)

    getAll.sum shouldBe 13
  }

  def getAll: List[Int] = {
    var result: List[Int] = Nil
    var next = Database.get
    while(next.isDefined){
      result = next.get :: result
      Database.remove()
      next = Database.get
    }
    result
  }
}

Of course, we can also factor out common functionality in regular methods, as done in getAll which is used by both test cases.

Addendum:

Quote from the question:

How do you keep your test suites short?

Test code isn't much different from production code in my opinion. Factor out common functionality using methods and put them into separate traits if they don't belong to a specific class you already have.

However, if your production code requires the tests to execute always the same piece of code, then maybe there are too many dependencies in your production code. Say you have a function (in your production code)

def plus: Int = {
  val x = Database.get.get
  Database.remove()
  x + Database.get.get
}

then you cannot test this function unless you populate your database with the two values that you want to add. The best way to make your tests shorter and more readable in such a case would be to refactor your production code.

"plus 3 2" should "be 5" in {
  Database.add(3)
  Database.add(2)

  plus shouldBe 5
}

may become

"plus 3 2" should "be 5" in {
  plus(3,2) shouldBe 5
}

In some cases it's not easy to get rid of dependencies. But you may want your objects in a test scenario to depend on a special test environment. The database is a great example for that, as is the file system, or logging. Those things tend to be more costly in execution (I/O access) and may have themselves further dependencies that you must first establish.

In these cases, your tests will most likely profit from using mock objects. For example, you may like to implement an in-memory database that implements your database's interface.

回答2:

The way I got things to work is below.

I cause tests B and C to execute A before them, by mixing in a TestAFirst trait. That trait also makes sure TestA will only get executed once.

There are a couple of variations possible. I chose to disallow automatic launching of TestA itself by DoNotDiscover annotation. Ideally, I'd like to keep TestA look as much a normal test as possible, pushing all dependency handling into TestAFirst.

import java.util.concurrent.atomic.{AtomicBoolean}
import org.scalatest.{DoNotDiscover, FlatSpec}

/*
* Mix this trait into any specs that need 'TestA' to have been run first.
*/
trait TestAFirst extends FlatSpec {
  import TestAFirst._

  if (!doneTestA.getAndSet(true)) {
    // tbd. Can we detect here if 'execute' failed? Would be a better place to set 'testASuccess' than within the
    //      'TestA' itself (= limit all dependency things to 'TestAFirst').
    //
    (new TestA).execute
  }
}

object TestAFirst {
  val doneTestA= new AtomicBoolean
  @volatile var testASuccess= false   // remains 'false' if 'TestA' failed, causing B and C to cancel
}

/*
* 'TestA' is a test *almost* like any other.
*/
@DoNotDiscover
class TestA extends FlatSpec {
  import TestAFirst._

  behavior of "Root class"; {
    it should "run prior to any of the B,C classes" in {

      assert(true)    // ... A tests

      testASuccess = true
    }
  }
}

class TestB extends TestAFirst {
  import TestAFirst._

  behavior of "class B"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... B tests
    }
  }
}

class TestC extends TestAFirst {
  import TestAFirst._

  behavior of "class C"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... C tests
    }
  }
}

Better solutions are still welcome, but since this works I wanted to post it out. There are also other threads in SO (Doing something before or after all Scalatest tests and org.scalatest: Global setup (like beforeAllSuites?) ) that deal with similar issues, but not with a clear answer.

Naturally, the idea here is to place TestB, TestC etc. in different source files, getting to the modularity that I aimed for. This is just a snippet.

回答3:

Adding as a new answer, so the differences are clear and the discussion above need not be removed. If I didn't do any typos, this should work (I did test it, and adopted in my project).

import org.scalatest._

/*
* Mix this trait into any specs that need 'TestA' to have been run first.
*/
trait TestAFirst {

  // Reading a 'TestA' object's field causes it to be instantiated and 'TestA' to be executed (but just once).
  //
  val testASuccess = TestA.success
}

/*
* 'TestA' gets instantiated via the companion object explicitly (thus @DoNotDiscover)
* and creates a success value field. Otherwise, it's a test just like any other.
*/
@DoNotDiscover
class TestA private extends FlatSpec {
  private var success = false   // read once, by the companion object

  behavior of "Root class"; {
    it should "run prior to any of the B,C classes" in {

      assert(true)    // ... A tests

      success = true
    }
  }
}

object TestA {
  val success = {
    val o= new TestA
    o.execute
    o.success   // getting a value from the executed test ('.execute()' itself doesn't provide a status)
  }
}

class TestB extends FlatSpec with TestAFirst {

  behavior of "class B"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... B tests
    }
  }
}

class TestC extends FlatSpec with TestAFirst {

  behavior of "class C"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... C tests
    }
  }
}

回答4:

You use Spray framework? You can try spray.testkit.Specs2RouteTest

   class RestAPISpec extends Specification with Specs2RouteTest {
     "RestAPITest" should {
        "Test A" in {
          ... some code
        }
        "Test B" in {
          ... some code
        }
     }
   }