Videoscope: a nonintrusive test tool for personal computers – includes related article on video signature analyzer operation – technical
Myron R. Tuttle
Videoscope: A Nonintrusive Test Tool for Personal Computers
INTERACTIVE TESTING OF APPLICATION SOFTWARE requires the tester to sit at the test system and enter test data using a keyboard and/or some other input device such as a mouse, and observe the results on the screen to determine if the software being tested produces the correct results for each set of test data. This process is time-consuming and error-prone if it is done manually each time the tester wants to repeat the same set of tests. This process must be automated to ensure adequate test coverage and improve the productivity of testing.
Videoscope is a test tool developed and used by HP’s Personal Computer Group (PCG) for automated performance, compatibility, and regression testing of interactive applications running on HP Vectra Personal Computers. It is independent of the operating system and nonintrusive. Nonintrusive means that it does not interfere with or affect the performance and behavior of the application being tested or the operating system. Videoscope is for internal use and is not available as a product.
An overview of the operation of Videoscope is illustrated in Fig. 1. During test creation the tester manually enters test data to the application being tested and waits for the correct result to show on the display. As the tester enters the test data, Videoscope records the data into a file called a test script. At the command of the tester Videoscope also records the screen result in the test script. The tester continues this process for each set of test data and at the end of testing the test script contains a sequence of test data interspersed with correct screen results. For retesting the same application, Videoscope automates the process by replacing the tester and playing back the test script to the application (Fig. 1b). The test data is sent to the application as it was entered during test recording. Whenever a screen result is encountered in the test script, Videoscope waits for it to occur on the display and then automatically does a comparison between the current screen and the correct screen results in the test script to determine if the test passes or fails.
The concepts and motivation for developing Videoscope evolved from experiences with trying provide the best test coverage of applications running on the HP Vectra PC. When the HP Vectra PC was developed, a major goal of the product was that it be compatible with the industry standards established by the IBM PC/AT. Compatibility was determined by running various applications written for the IBM PC/AT and evaluating how well they ran on the Vectra. The first iteration of testing was done by hand using most of the engineers in the lab. This was clearly an inefficient and expensive way to run these tests. The tests were then automated using two utility programs, Superkey and Sidekick from Borland International Incorporated. Superkey was used to capture and play back keystrokes, and Sidekick was used to capture screen displays and save them to disc files where they were compared with files containing known-correct screen displays. These tools and certain standards for creating tests were called the regression test system (RTS).
While RTS initially proved adequate, long-term use revealed weaknesses in the system. First, mouse movements could not be captured or played back, so any program that used a mouse had to be tested by hand. Second, the system was intrusive. This meant that certain programs did not act the same when RTS was loaded. The deviations ranged from running differently to not running at all. For example, Microsoft Windows could not run at all with RTS because of conflicts over control of the interrupt vectors. Other applications could not run because RTS used up so much memory that there was not enough left for the application. Finally, RTS could not be used to do performance testing since it used system resources and affected the performance of the system.
Videoscope was developed to replace RTS and to compensate for its weaknesses. This resulted in the following design objectives for the Videoscope system:
* It had to have the same capabilities as RTS.
* It had to be nonintrusive.
* It had to be able to do controlled-time and real-time performance testing. Controlled-time means that fixed time delays are inserted in the test scripts to control when certain events take place on the system under test. Real-time performance testing means the ability to determine the actual response time of events taking place on the system under test.
* It had to be able to handle a mouse and any other pointing device that HP sells for the Vectra.
* It had to support HP extensions to the PC standard.
* Test scripts had to be portable. The intent of this objective is to be able to port test scripts to other PC operating systems such as Xenix, OS/2, or even HP-UX. It was also considered necessary to be able to use a multitasking computer system such as the HP 3000 Computer System as a host to test multiple systems on playback.
* It had to be able to handle a list of programs (e.g., Microsoft Windows and HP AdvanceWrite) that we needed to test but were unable to test with RTS.
The Videoscope system consists of two major parts: a program called vscope that resides in a system known as the host system, and a board called the Videoscope board that occupies one slot in the PC running the application being tested (see Fig. 2). This system is called the system under test (SUT). The vscope program is used by the tester to create and perform the actual tests. The videoscope board provides the links to the keyboard and pointing device (i.e., mouse, tablet, etc.) ports on the SUT. These connections enable the host keyboard and pointing device to be used in place of the SUT keyboard and pointing device during test recording. the Videoscope board is also connected to the video adapter of the SUT, which enables it to capture the video signal of the screen contents of the SUT. The video signal is used to compute a digital representation of the screen. This representation is called a signature, and it is the signature that is stored in the test script.
Although two complete PCs are required for test development, the SUT does not need to have a keyboard or pointing device. For playback, a monitor is optional in the SUT since normally no human will need to look at it. Also for playback, it is possible to use any computer system with appropriate software as the host–not just a PC. This satisfies the portability objective.
The vscope program provides the interface between the tester and the recording and playback features of the Videoscope system. For recording the tester uses the keyboard and pointing device on the host and runs the application being tested on the SUT (see Fig. 3). The front routine captures the test data, which is composed of keystrokes from the keyboard and pointing device movements from the HP-HIL (HP Human Interface Link), converts it to symbolic names, and passed it on to the parser. Tha parser performs two functions with symbolic names: it saves them in the test script, and it translates them into commands for the videoscope board. These commands are transmitted over the RS-232-D line to the videoscope board on the SUT. Periodically, the tester must tell vscope to capture a screen display and save it for comparison during playback. Each time a screen is captured a signature is computed by the code on the videoscope board and passed back to vscope to be saved in the test script. This whole process results in a test script composed of two files. One file (test data) contains keystrokes, pointer device movements and picks, and markers for signatures, and the other file (signature file) contains the screen signatures. Because of blinking fields and cursors on the display, vscope takes several signatures for one screen result. A histogram of the signatures is built and only the most frequent ones are included in a signature list, which goes into the file.
The syntax of the symbolic names and the algorithm to interpret them is based on those commonly used by PC keyboard macro programs. The reason for this design decision was to maintain the look and feel of the RTS, and to reuse existing data stuctures and algorithms.
Not all keys are recognized by the front routine. Keys that do not send data to the keyboard buffer (e.g., CTRL, Alt, Shift) are not recognized. These keys, known as “hot keys,” are commonly used by terminate and stay resident (TSR) programs such as Superkey or Sidekick to activate themselves. TSRs are so commonly used in PCs that it was decided that vscope had to accommodate their presence on the host and the SUT. This created the problem of how to send such nondata keys to the SUT. Fortunately, the solution to this problem was a natural consequence of the method used to encode and decode the symbolic names created by vscope. For example, the keystrokes Enter cls Enter dir /w [cmd]getcrc[cmd] results in a clear screen, a listing of filenames in wide format on the SUT, and the capture of the resulting screen in the signature file. These keystrokes result in the following stream of symbolic names being generated by the front routine:
This stream is interpreted as follows:
* cls – send press C, L, and S key commands
* dir /w – send press D,I,R,Space,/,and W key commands
* [cmd]getcrc[cmd] – Execute getcrc command
Presssing a key on the host means sending a press key command to the videoscope board over the RS-232-D line. Under this scheme, an Enter key can be inserted in the test script in two ways. The first way is to press the Enter key on the host keyboard. The second way is to press the < key, E key, N key, T key, E key, R key and > key in that order. The pattern
The pattern [cmd] is used to mark the beginning and end of a vscope command. Some vscope commands translate directly into commands used by the Videoscope board firmware, and other commands are used only by vscope. For example, the vscope set command translates directly into a Videoscope firmware command to set switches on the board. On the other hand, the vscope log command, which writes information to a file on the host system, has no association with the Videoscope board. Other commands translate into a complex series of operations. The getcrc command is such an example. During test script recording this command retrieves the screen signature and stores it in the signature file. During test script playback, it reads the signatures from the test script file, compares them with the current screen signature and reports the results.
Keystrokes and mouse movements are sent as quickly as possible to the application. A method is provided for slowing them down to a fixed maximum (time command). Associated with each HP-HIL transaction is a delay which can be set from 0 to 32,767 milliseconds. The effect of this delay is to limit the maximum speed at which mouse movements are sent to the application. Many mouse-orinted applications can lose mouse movements if they come too fast. Normally this is of no concern when the user is part of a closed feedback loop and can reposition the mouse. Videoscope is not tolerant of any differences on the display that would be caused by missing a mouse movement. By
experimentally changing the delay value, the test can be run at the maximum speed that gives consistent results. Keystrokes can be programmed with separate press and release delays. Each of these delays can be specified in increments of 32 milliseconds over the range of 32 to 8,160 milliseconds. This gives a maximum typing speed of about 180 words per minute. Allowing these fixed and varying wait times to be inserted between keystrokes provides a method for modeling user think times for performance measurements.
For playback mode the tester runs the vscope program, selects the playback option, and specifies the test script files to use. In vscope the play routin shown in Fig. 4 reads the symbolic names from the test script and sends them to the parser. This time the parser does not create another test data file but just translates the data stream and sends it to the SUT. Whenever a signature marker is encountered in the test data file, the associated signature list is retrieved from the signature file and passed to the Videoscope board. The Videoscope board will compare the current screen signatures with signatures passed from vscope and send back a pass or fail indication depending on the outcome of the comparison. If a test fails, vscope will either log the result and continue testing or halt further testing. This decision is based on the options specified by the tester when vscope is set up for playback.
In addition to test recording and playback, vscope provides another operating mode called the replay or regeneration mode. Screen signatures are highly dependent on the video system in use. Even though the display may look exactly the same, signatures from an HP multimode card and a monochrome card are different. If a test developed using a multimode card needs to be played back on a monochrome card (e.g., to test whether the software properly supports the monochrome card), a new set of signatures for the monochrome card needs to be captured. The replay mode automates this process by playing back the test data file and replacing the old signatures with new signatures instead of comparing them as it would in a normal playback. A single test data file can access various signature files, allowing it to be used with several video hardware configurations.
The Videoscope board is partitioned into two major sections: the Videoscope processor and the system under test interface (see Fig. 5). The two sections operate independently and are connected by an 8-bit bidirectional port. The processor contains the video signature analyzer (VSA) and the keyboard/HP-HIL emulator. The Videoscope board is a full-length PC/AT-style accessory board (it can be used in a PC/XT-size machine if the cover is left off). During normal operation the board derives power from the + 5Vdc, – 12 Vdc, and + 12Vdc lines of the SUT backplane. The PC/AT extended backplane connector is used only to access the additional interrupt lines.
Videosope Processor. The Videoscope processor is based on an Intel 80188 microprocessor. This microprocessor was chosen because of its low cost and high level of integration, and the fact that it used the same language development tools as the Intel 80286. The 80188 contains built-in timers, DMA controllers, an interrupt controller, and peripheral select logic. It eliminates the need for at least two 40-pin packages and several smaller-scale chips.
The processor system is equipped with 32K bytes of ROM and 8K bytes of RAM. Connected as peripheral devices are a UART for datacom, two HP-HIL slave link controllers (SLC) for implementing two HP-HIL interfaces, the video signature analyzer, several switches, an LED indicator register, a port to emulate the DIN keyboard, and the SUT interface. The slave link controllers are HP proprietary chips for implementing the HP-HIL protocol.
The Videoscope processor firmware is written entirely in Intel 80188 assembly language. It is modular and all main routines are reached through a jump table in RAM. A loader function is provided so that a user can write a custom module and download it into the processor RAM. The jump table can be overwritten so that the downloaded module is executed instead of the ROM-resident code. There is a command (normally a null operation) that can be used as the entry to a user module. The firmware is structured as a real-time interrupt-driven system. The code normally sits in an idle loop until a command needs to be processed or an interrupt serviced. Some of the command processing routines themselves introduce new interrupt service routines for their operation.
Communication with the host system is through the RS-232-D interface shown in Fig 5. The firmware supports up to 9600 baud using a straightforward command-response protocol with a simple DC1 handshake. The ACK/ENQ protocol used on the HP 3000 is also supported. All data transfers between the host and Videoscope are in ASCII hexadecimal characters. This was chosen to allow complete independence from the host datacom. It also enables test scripts be stored in readable and editable form. No compilation or decompilation of the scripts is necessary. All commands and data from the host parser routine to the Videoscope processor are of the form: *
* Set Attribute (*A). Allows the host software to change default settings.
* Load Siglist (*C). Implements downloading of signature lists for screen matching.
* HP-HIL (*H). Sends an HP-HIL device X, Y, and button data frame over the HP-HIL interface.
* Include (*I). Sets the start and stop limits of the area of the screen to include in a signature. Used to avoid time variant areas of the screen.
* Keystroke (.suP.*.J .sup.*.K). Sends a keystroke keycode and shift modifier. The *J form uses default timing while the *K form allows explicit press and release times.
* Load (*L). Implements downloading of code routines.
* Resend (*R). Resends the last data record in case of datacom error.
* Signature (*S). Takes a signature of the screen. Used for building the signature file for later playback.
* Test and Report (*T). Provides dumps of various sets of variables.
* Wait for Match (*W). Compares screen signatures until either the downloaded list is matched or a time-out occurs.
Responses to the host are of the form: <+ ->
Video Signature Analyzer. The VSA is the key component of the Videoscope concept. By using the technique pioneered by the HP 5004A Digital Signature Analyzer, it is possible to monitor the video signals generated by the SUT’s display adapter in real time in a totally nonintrusive manner. The main component of the signature analyzer is a 24-bit linear feedback shift register. The linear feedback shift register is used to accumulate a signature (similar to a cyclic redundancy check) of the video data stream. The signature is a 6-digit hexadecimal number that describes the state of the screen. The linear feedback shift register is a pseudorandom number generator driven by the video signal. This means that even a one-pixel difference will change the signature. A state machine using the display’s horizontal and vertical sync signals controls when the signature is taken. Since some applications put time variant data on the screen such as dates, a clock, or file names and paths, a method is provided to allow the signature to be started and stopped by a count of scan lines after the start of the display. In this way only the nonvariant portion of the screen will be included in the signature.
To accommodate the various display adapters used in a PC, the video signature analyzer has an eight-input multiplexer which can select from eight separate video streams. This allows exhaustive testing of all image planes in a multiplane adapter (e.g., EGA or VGA) and minimizes testing when the different planes contain redundant information. The tester can use the vscope set command to control how many planes are signatured. A separate signature is computed for each plane, and when doing a multiplane match to a signature list, all enabled planes must match or a failure is reported.
To reduce the part count while maintaining reasonable precision and speed, the linear feedback shift register is wired as 22-bit maximum length with an additional two bits following. This provides a maximum pattern length of 4 X (2.sup.22 – 1) states. Although a typical display has many more states than this (a 640 X 350-pixel display has over 2.sup.224000 distinct states), the probability of having an incorrect display with a valid signature is extremely low. In instances where a match of an invalid screen (an alias signature) does occur, the next screen signatured will almost certainly fail. Very few alias signatures have been detected in actual use.
The VSA is controlled from the Videsoscope processor by a high-speed state machine implemented partially in hardware and partially in firmware. The firmware uses a state machine consisting of an idle loop and several interrupt service routines. The present hardware portion of the state machine will operate with video dot rates in excess of 50 MHz and scan rates in excess of 32 kHz. See the box on page 62 for more details about the VSA state machine architecture.
At the completion of the signaturing process the firmware can either pass the signature back to the host or compare it against a list of valid signatures downloaded from the host. The first method is generally used during script creation when the vscope program is building the signature list file, and the second method is used during test script playback. Datacom traffic is kept to a minimum by downloading lists of signatures and doing the comparisons locally on the VSA board.
Keyboard/HP-HIL Emulator. Videoscope has the capability of emulating the keyboard used on the earlier Vectra models as well as the industry standard DIN keyboard used in both the PC/XT and Pc/AT protocols. The new Vectra models use the DIN keyboard and are compatible with the PC/XT and PC/AT protocols. For the HP-HIL interface, there are two slave controller chips on the board, which are capable of emulating any two of the following devices: a mouse, a tablet, a touchscreen, or a keyboard. The controllers are directly driven by routines in the Videoscope processor firmware and no additioanl processors are needed. Additional devices can be emulated by changing the firmware.
SUT Interface. The SUT interface port provides a communication path between the Videoscope processor and the SUT processor. This is an 8-bit bidirectional port with provision for generating an interrupt wwhen written to. In the SUT this can neither be a nonmaskable interrupt (NMI) or any one of the interrupt (INTRn) lines on the backplane. The desired line is selected by a switch. The SUT must have an interrupt handler installed for this feature to be used. Also included in the SUT interface are 8K bytes of ROM and 8K bytes of RAM. This 16K-byte address space can be configured within the C0000-DFFFF address range of the SUT. The ROM is seen by the SUT power-up routines as an option ROM and normally includes the necessary interrupt handler. The I/O ports of the SUT interface can be located at any of several addresses normally reserved for the IBM prototype card. Both the memory and the I/O ports can be relocated to avoid conflicts with hardware installed in the SUT. The current implementation of the Videoscope system does not make use of the SUT interface. However, the hooks are available for users to create routines for special-purpose testing requirements.
Videoscope has met or exceeded the original objectives established for the system. One minor disappointment is that the the signature generated from the video signal is not unique. However, the probability of two screens having the same signature is very small, so this is a very minor problem and simple workarounds have been found.
The design and implementation of Videoscope was highly leveraged. The video signature analysis hardware and firmware are based on the HP 5004A Signature Analyzer. The data structures and algorithms for interpreting data in the script file are based on those commonly used by keyboard macro programs, and the data communication software used by vscope to communicate with the Videoscope processor firmware is a package called Greenleaf Data Comm Library from Greenleaf Software, Inc.
Videoscope provides a major productivity tool to improve the quality of software. The PC intrinsically is an interactive computer system. This means that batch-style tests cannot adequately test the capability of any software written for a PC. Videoscope provides an automatic alternative to slow, error-prone, and expensive manual testing. Currently there are over 100 Videoscope systems in use at 18 HP divisions.
We would like to acknowledge the help of our manager, Jean-Claude Roy, who made a major contribution to the design of Videoscope, and the entire Videoscope design review team, in particular John Schimandle who provided much appreciated user feedback. The development of Videoscope was a corporatewide effort involving may people, and so we thank all of those who contributed to the successful development of Videoscope.
COPYRIGHT 1989 Hewlett Packard Company
COPYRIGHT 2004 Gale Group