Fitnesse vs any other subsystem testing tool [clos

Question

We are currently using Fitness for subsystem testing. we are having lot of issues using the tool, few to mention

1. Development time for writing Fixture is more then writing the actual code
2. Issues around check in of the dlls so that Qa can test them
3. Issues in running Fitnesse for project which uses NHibernate
4. limited help online

We are planning to use some other tool to do the testing Few options which we know are

1. SOAP UI
2. Story teller

I am not sure whether we will have similar problems with these tools It would be great to know if someone has experience using these tool and could guide us

In our project we have adopted TDD so we have Nuits for unit testing. It would be great if anyone is aware of tools/ideas which could extend nunits for subsystem testing as well.

Answer 1

Component testing tools are all about calling functions. Your tests cause functions to be called in "fixtures" that then call into the SUT. Any tool based on this premise will encounter the problems you reference above.

However, most of those problem are manageable. For example you should not be writing lots of fixtures. If you are, something is wrong. Secondly, your fixtures ought to be little more than wiring code to call the APIs in your application. If your fixtures are doing significant work, then something is wrong.

In most FitNesse environments the number of fixtures is rather small. For example, there are over two hundred acceptance tests for fitnesse itself, but the number of fixtures in on the order of a dozen, and they are all relatively simple.

Get help on the [e-mail] site. The folks there are usually very responsive to questions.

Answer 2

If you can communicate with your software using text, then I have had success on past projects rolling my own framework using expect.

The framework I cooked up stored tests as XML files, using a simple xUnit style markup. The xml files were then transformed into executable tests using a stylesheet. I ended up transforming the tests into Tcl/Expect, but you could transform them into anything. In fact, if you wanted, you could transform them into multiple languages, depending on your needs.

Several people have kindly reminded me (in the same way you remind you poor dottering grandfather about the drool on his chin) that we are in the 21^st century when they inquire why I would choose Tcl over some more modern language. As it turns out, for the purposes of this kind of testing, I haven't yet found a better choice. The Tcl language still kicks butt in this area. Trust me, I didn't wake up one day and say to myself "self, what I need a test framework implemented in a scripting language everyone will hate!"

Believe it or not, I really was looking for a tool, any tool, that had the following characteristics:

• Cross platform. This was non-negotiable. We do a lot of cross platform development and we already use WAY too many tools that don't support cross platform development.
• Simple syntax. Say what you want about Tcl, but the syntax is very regular. I knew that some native code would probably creep even into the XML files (and originally it was Tcl only, no XML) and I wanted the syntax to be comprehensible to a non-programmer. This simplicity is a core strength of Tcl. As it turns out, it also made transforming the XML easier too.
• Free. My favorite price ;-)
• Writing tests as simple xml files allowed non-programmers to write customer acceptance level tests - no programming required.
• Easily extended.

I did not set out to home grow this to the extent I have. Initially, I looked at established test frameworks like DejaGnu and android. Mostly they had way too many features. They were so feature laden that I didn't think they would be easy for a project to start using without a lot of up front training. Looking at DejaGnu, got me interested in Tcl in general, and after a brief look at tcltest, I almost gave up. Both DejaGnu and tcltest assume you are an advanced Tcl scripter, which I didn't think anyone at my company ever would be. In addition, I wanted the test framework (if possible) to support an xUnit type of test framework and neither of these tools did.

Eventually I found TclTkUnit, a Tcl based testing framework that is designed along xUnit lines. It was only a short leap of logic to realize I could run TclTkUnit in Expect instead of tclsh and get everything I needed.

As it ended up getting used more, I added another stylesheet to render the xml files nicely in a web browser. The test framework generated it's own documentation.

On another project we needs a very basic sim / stim environment to emulate a person throwing switches and pushing buttons on a piece of hardware we didn't have. It only took a few hours to hack the test framework to function as a simulator. Creating the framework took some work, but we felt that it did pay benefits in the long run. I really believe that these types of unforseen consequences of creating your own tools is why people in the agile community & XP in particular have always been such strong advocates.

Answer 3

We use unit-testing frameworks like NUnit to drive our subsystem tests as well - the tests don't care how they are run. It doesn't have fitnesse's document-based approach, though.

Answer 4

We have adopted a Fitnesse-based but practically-code-free approach using GenericFixture (google for Anubhava to find his wordpress site) for Fitnesse. What this allows us to do is to create "executable test narratives" using a language that is friendly to the business-side (as opposed to the technical-side). This language, which is very easily defined, practically without coding, in Generic Fixture, is called a DSL (domain specific language). So we can write our test narratives using e.g. medical terms or even in a language other than English. Basically what we get is transforming our Use Cases into executable narratives. We are starting to use it in a large project (15 ppl for 2 years) and it seems (so far) to have a good future. It easily allows Test Driven Development or test-creation after development (traditional approach). It is wiki-based (Fitnesse) and its versioning and refactoring funcitonality has proven so far sufficient. I can give more info if anyone is interested.

best regards,

Aristotelis.