Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
I have an application/product which is created using .net technologies. This product has a GUI, which connects to a DB using a Web API (SOAP on an application server). Majority of the tests are executed for the values in the DB, while the others may fall into Usability, Performance, etc. As such, 60-70% of the tests to be executed are to verify if the values in the DB are correctly shown on the GUI, while the others test if the GUI is working as desired in the requirement specifications.
The other aspect to the testing will also be testing the backend Application server interface with both the DB and the GUI. As this will enable us to determine, if the App Server is sending bad values to the GUI or is it the DB which has bad values!
The product UI does not change much, except to add new features in the form of additional dropdown menu items and additional pages for these menu items.
Considering the above, which framework would be more suitable? QTP or Selenium or some other Commercial/Open Source tool?
(Cost is not a major concern, and neither is browser compatibility or OS. The system installs on a Windows Server, and DB is compatible.)
We have an acceptance testing 'framework' built around Selenium to test our app which has a flex UI that talks to a java + db back-end.
We could easily integrate Selenium into our existing testing and continuous integration infrastructure because we can write our scripts in java and use junit to drive the testing. These tests are also written and maintained by developers. We also use dbunit to set up the database before each test.
Our test department however decided to go with QTP. They were shown our infrastructure built around Selenium but they found it hard to understand. I know they have a full-time dedicated person to write the tests and maintain them.
Since I do not know your exact situation I can only suggest you to consider the following:
- Who is writing and maintaining the tests?
- Are the suites going to be part of a larger infrastructure?
Selenium is an excellent choice if...
- you have developers responsible for writing and maintaining the tests
- you need to have these tests as part of a larger infrastructure
- you are pretty sure you do not need to do much testing outside of the browser
- you think you might want to test other browsers and other platforms besides Windows
- you want something free
QTP might be a good choice if...
- you have less code-savy people writing and maintaining tests (though I am not sure if Selenium IDE is hard to learn for non-coders)
- if you need significant testing outside of the browser
This article might also help you.
I think it is obvious which one I prefer, but you need to decide what works best for your situation.
I think your best choice will be HP new solution for GUI and non GUI testing.
The new solution is a new integration with 2 products - QTP & ServiceTest.
- QTP is a known GUI automation solution with enhanced capabilities that help organizations to automate their application in the GUI layer.
- ServiceTest is quite new solution that can automate web services & other non GUI interfaces.
The integration between those 2 give the users one solution to automate cross layers applications as well as integration testing between different applications.
You can find more information at HP site.
.Net you say?
If you are using VS2010, Coded UI is really good, as a long time QTP user (on the more technical side of that of course) Coded UI was a dream, even in its early stage of existence it is so powerful and integrates tightly into the Visual Studio eco system that if you can use it i recomend it highly. You can read about here
Another tool that my experience is more limited with but i have heard good things about is (specflow) which is can work nicely with coded UI to create and organise tests in a sensible way.
Might give you something to think about anyway!
With access to a vast array (no pun intended) of programming libraries (Java, etc.) my vote would be to use Selenium-RC. But like the first poster attributed, this does require an additional learning and maintenance curve. But once setup, the applications will only be bound by your imagination (and programming prowess :)).
DB integration using Selenium-RC (in java) was a cinch. We were also able to extensively use Selenium in measuring server responses. I'm sure there are hacks in QTP to do the same, but with HP support, being what it is lately, your bets should lie with the open source community (and stackoverflow :))...
My take on this -
Selenium is an excellent choice if...
- If you don't need script to be in Quality Center. Continuous integration is way easier on other hand.
- Selenium is a lower level tool with less functionality i.e. checkpoints. Not a big deal but it also requires more time to develop and maintain scripts. Selenium 3 is coming, every time new Selenium is out, API changes, so be ready to say good bye to old scripts (unless separate machines with different version of Selenium).
- Selenium is a free tool, but you may find yourself not being able to
accomplish certain action on given platform or browser simply because the
functionality is not developed yet.
- No, developers are not usually great with selenium, especially web tier (browser automation). Knowledge of Java doesn't come with full understanding of html dom. Do not confuse with JUnit, which they do use sometimes.
- Large Selenium projects look like any other project in Eclipse. QTP way better in this case, especially with QC.
- Java is great for data structures and is a full OO language, unlike VBScript.
QTP might be a good choice if...
- You need integration with QC. No straightforward support Continiuos Integration.
- Large infrustructure projects look cleaner with QTP. QTP has web tier
descriptions separated from code by design (GUI files containing html
attributes for elements in use by script). All library files are kept in
QC and test data stored in excel file (by best practice).
- VBScript is great on Windows: along with QTP API you do get pretty strong access to Win API and direct access to HTML DOM.
- QTP supports old code since 5 years ago, and yes they have Customer support.
- time to market is less than with Selenium. You do not need to search forums, when you need to click on link by xpath or by text ...you just click. Also extra functionality in form of chekpoints and other
Think twice before you start developing automation test scripts against different browsers. Automation does come up short when it comes to GUI defects. And almost all browser- specific defects are styling, layout (GUI). Functional defects incorrect values displayed, or non working controls are functional defects that can be identified on 1 platform.
