Companies that develop and maintain software can dramatically improve the quality of their software releases by creating regression test cases that ensure that existing features are not broken with new releases.

This white paper discusses:

• How to create regression test cases.
• When to automate regression test cases.
• Best practices for automation analysis.

Creating Regression Test Cases

Once a software product has been released to product, each new release of the software could cause existing features to fail. To prevent this, it is wise to create a set of regression test cases that are run with each new release. Below are some best practices when developing a regression suite:

• Categorize by Functional Area - Your software product most likely has different sets of functional areas (e.g. Invoicing, Billing, etc). When creating regression test cases, categorize them by functional area so that you can ensure you have good test coverage for each functional area of your software.
• Regression Test Case Design - Regression test cases do not normally need to test bounds, invalid data entry, etc - normally they will be designed to test the software the way it is designed to work. The reason for this is that when the feature was originally designed, it should have been thoroughly tested for bounds, invalid data, etc. An exception to this is if you find that new releases tend to break existing features from a validation perspective. If this is the case, keep some specialized regression test cases to ensure that the validations are still in place.
• Revisit the Regression Set with each New Release - Upon implementing a new release of your software, it is wise to recognize new features shipped with the new release and to create a new set of regression test cases that test the new features. If you do not revisit your regression test cases with each new release of your software, the regression test cases will become stale and out dated.

When to Automate Regression Test Cases

Many companies run their regression test cases manually, so when does it make sense to begin automating your regression test cases? It makes sense to automate your test cases when you can no longer run the regression test cases on each build created. For example, if you are doing daily or weekly builds of your code to quality assurance and you cannot quickly run your regression test cases with each build, it is time to consider automating them.

To automate test cases, you must purchase an automated testing tool. There are many great tools on the market, including Automated QA Test Complete, HP Quick Test Pro, HP Win Runner, Rational Robot and Rational Functional Test, just to name a few. We normally recommend AutomatedQA TestComplete, as it is competitively priced and has similar features as the others.

Once you have purchased an automated tool, you can use the tool to create your regression test cases. Each test case becomes a script. Many tools have record and playback features where you can turn the recorder on, open your software and perform the actions for a test scenario, then save the recording. This is a great way to learn the scripting engine, but it is not usually adequate to create well designed automated test scripts.

Normally, you will want to have a technically minded software quality engineer in your organization that creates and maintains the automated scripts, as using these tools require knowledge of the tool, programming skills and great trouble shooting skills.

If you are initially creating your automation strategy, it is wise to consult with an automation expert to ensure best practices for your automation design. There are many companies that specialize in this; we have worked extensively with Star-QA (http://www.star-qa.com) with great results. Contracting with an automation expert can save effort and costs in the long term, as they will normally work with you to provide an automation framework that will be reusable and can provide training to your software quality engineer(s), allowing them to make great strides with their automation skill set in very little time.

If you would like to see how Software Planner integrates with the major automated testing tools, watch a narrated movie at Pragmatic Software.

Best Practices for Automation Analysis

Once your regression test cases are automated, they should be automatically run upon new builds of your software. If you can do daily builds of your software into your quality assurance environment, this is ideal. Once the automation is running daily, you will need a way to quickly determine how many automation test cases were run, how many passed and how many failed. For failed tests, you will want to drill into the detailed logs to determine what caused the failure.

Software Planner is an ALM tool that can manage this process. Software Planner integrates with all the major automated testing tools including AutomatedQA TestComplete, HP Quick Test Pro, HP Win Runner, Rational Robot, and Rational Functional Test. By integrating automated testing into Software Planner, you can launch the tests from within Software Planner, create test sets, analyze the results (which tests passed or failed), and automatically send emails upon test completion. You can also trend these results using graphical dashboards.

Automatic Test Case Detection

Software Planner can automatically scan a directory containing your automation test scripts and create test cases within Software Planner without any intervention from a human. Below is a screen that shows test cases that were automatically found and created by Software Planner. Notice it can scan for any of the major automated tools.

Test Sets

Once automated test cases have been detected, you can create test sets. Test sets are simply a way to group a set of automated test cases together to test specific areas of your software. Below is an example of test sets in Software Planner:

Kicking off the test set is a simple as clicking the Execute link. Below is an example of one of the test sets above with the associated test cases:

Keyword Driven Testing

As an alternative to record and playback and writing scripts using a programming language, Software Planner also offers keyword driven testing. Using keyword driven testing allows non-programmers to easily create scripts that can be re-used -- empowering subject matter experts to create automated scripts without detailed knowledge of the scripting tool itself. How is it done? The subject matter expert uses Software Planner's Keyword Driven Testing application to keep track of their automated test cases and the automated test cases can be run from within Software Planner. We also have a movie that illustrates this feature.

Analyzing the Test Runs

Software Planner provides reports and dashboards that show the results of each automation run. Below is a trending report that shows the daily runs:

Return on Investment

You should see a return on investment within one release of your software after implementing an automation test strategy. This will be achieved by:

• Quicker Releases - By having your regression test cases run automatically, your software quality team can concentrate on testing new features of your software and less time regressing existing features.
• Higher quality releases - Your software releases will have fewer bugs and require less customer support because they will be of higher quality.
• Happier Customers - Your customers will be happier and more willing to serve as testimonials for future prospects.

Test Case Automation White Paper Download

• Test Case Automation White Paper - PDF

Download Software Planner

If you wish to learn more about Software Planner or receive a personalized demo of the product, contact AutomatedQA at +1 303-768-7480. If you haven't tried Software Planner, be sure to download and try it today. You can also learn more at SoftwarePlanner.com. You can learn more about the automation integration by checking out our Services.