The ultimate test – testing year 2000 conversion code – Industry Trend or Event
Testing tools abound, but as in any bet-your-business acquisition — caveat emptorAs the new millennium approaches, many IT organizations are mired in the process of making their software Year 2000 compliant. In many companies, this process requires programmers to fix tens of thousands of two-digit date fields embedded in millions of lines of code. Unfortunately, that’s the easy part. The real Y2K crisis begins with testing the code once it’s fixed.
One company that has entered the test phase of its Year 2000 project is Sundstrand Aerospace in Rockford, Ill. Nearly 85% of Sundstrand’s applications –10 million lines of code worth — must be converted and tested. Testing began in February 1996, says Bill Schuyler, project manager. The company plans to spend nearly 50% of its Year 2000 budget on this phase.
Such budget allocation is commonplace. London-based Commercial Union Insurance, for example, is spending nearly 65% of its Y2K budget on testing.
Both Sundstrand and Commercial Union found that creating their own test environments ate up significant resources. Besides establishing the hard- ware environments, setting up the initial test cases to establish the baseline tests for post-conversion regression testing proved to be a major expense. In an effort to keep their projects on schedule, both companies decided to test only for conversion-related failures.
The only real way to minimize testing costs and ensure timely results is to employ automated tools that can consistently execute tests the same way under the same conditions — both pre- and post-Y2K conversion. Firms must then be able to identify, capture, store, and compare test results for each application.
Before and After
A major question in testing converted Y2K applications is whether to test the entire application or only those areas associated with the date changes. As with any software development or maintenance effort, Year 2000 date changes can introduce new defects — also known as regression failures — into the code. While testing only the affected code simplifies the effort, firms should also consider the regression failure issue. Regression defects can be significant time and cost wasters and are a major reason for delays in updates to new applications.
An effective test methodology requires firms to test the entire application completely prior to the start of the code conversion effort. They test again after code conversion with old and new date fields to ensure the application will continue to work this century and into the next. Finally, firms can compare the test results to ensure that no regression failures have been incorporated into the code. This methodology requires that the testing process begin immediately after the inventory and assessment phase of any Y2K conversion effort.
Most consultants, as well as test tool vendors such as Compuware, Mercury Interactive, and Intersolv, recommend this multistage test process to ensure that non-conversion-related defects are eliminated early in the process. According to Sundstrand’s Schuyler, the best way is to test the entire application before and after the conversion of each application component. The com- pany has employed this three-step process in its Y2K testing efforts, setting up three distinct conversion test beds.
Both Sundstrand and Commercial Union first create and establish the baseline tests with the existing code prior to conversion. Once the baseline is established, the converted code is checked, modified, and tested in a second test environment using the same tests and dates used to establish the baseline. This retesting ensures that the application functions in the 20th century date formats and that no regression failures have been incorporated into the code. The third step is the actual testing of the converted code with converted date fields to verify that the applications operate with 21st century dates.
Testing applications before and after conversion requires multiple sets of tests to check for old and new date fields. Modifying existing tests is a major programming effort, as is writing new ones. Test tools such as Mercury Interactive’s WinRunner 2000, Viasoft’s VIA/AutoTest, and Compuware’s QA Hiperstation GUI automate the process of ad- ding incremental dates in existing tests.
Resetting the system clock is another important task that’s not only time-consuming but can disrupt the entire development and test environment. Without changing the system clock, firms can’t be sure that the application will work in the next century. While some companies are touting expensive hardware-based time-shifting capabilities, test tool vendors have developed inexpensive software-based simulation environments. Vector Software’s Virtual Date Utility, for instance, enables date variable testing of code and simulates 21st century system clocks.
Once all the required tests and test environments are established, managing them becomes the major issue. For each stage, companies must perform the various tests, and gather and compare the results. Here again, tools can help.
Test management programs are a new type of tool that supports automated test environments. They range from simple tools that store and compare test scripts and results to more complex systems that provide centralized test frameworks and incorporate test planning, test execution, result collection and comparison, and reporting.
Test management tools are available for all of the major platforms, covering mainframe-, Unix- and PC-based application testing. Mainframe tool vendors such as AutoTester, Viasoft, and Compuware provide test management products that combine centralized test planning, storage, and management with test result analysis capabilities. These products enable centralized test planning, a central test repository, and change control. These products typically allow tests to be executed across a network of systems during off-hours.
Some of the PC-based test management tools have additional integrated capabilities to further simplify the process of test management, execution, and defect tracking. Mercury Interactive’s TestDirector and Rational’s SQA Manager provide internal defect-tracking capabilities.
In addition to providing significant planning, management, and reporting capabilities, test management tools perform another major function that won’t be important until after the turn of the century. Lawyers are already preparing for the deluge of litigation expected from problems associated with Year 2000 compliance. To protect themselves against lawsuits, companies will need to be able to provide the data they gathered when testing specific applications. Test management tools store this information in their repositories.
Test coverage analysis tools will also be effective in protecting against litigation. While test management tools store and report test results, they, like individual test tools, cannot assure that all parts of the application are being tested completely. Without some way to measure code testing, especially date-converted code, firms can’t be sure how effective their tests are. Test coverage tools provide detailed reports of untested applications and those not tested effectively.
One large database vendor purchased test coverage tools to verify the level of testing performed by its test suite. The vendor applied the coverage analysis to two versions of the test suite. The older version took four hours to run and provided what the vendor considered to be adequate test coverage. When the code was analyzed using a test coverage tool, however, it found that only around 35% of the code was actually being tested. The newer version of the test suite, which took 12 hours to execute, added a significant number of tests to provide what the vendor felt was very good test coverage. When they analyzed their work with the test coverage tool, however, they found that coverage had increased to only slightly over 40%. For firms testing against a hard and fast deadline, test coverage tools are a significant asset in keeping test time and costs to a minimum.
COPYRIGHT 1997 Wiesner Publications, Inc.
COPYRIGHT 2004 Gale Group