TestProNews Q1 - 2006
Understanding Test Performance Analysis

Companies spend a lot of time and money developing their tests stations. They willingly incur this expense in order to collect data from their tests to improve their product designs and manufacturing processes. Unfortunately, because many companies do not have the proper tools, they unknowingly spend too much in labor costs to organize and access test data and reports.

Companies most often spend too much attempting to:

Learn which tests are failing the most frequently Learn why products are failing tests Determine if there is a particular pattern to the test data Make the data and reports easily accessible to key people in the company

Figure 1: Multi-dimensional TPA allows test engineers and managers to immediately analyze a test data cube from an unlimited number of perspectives. Enlarge Image

Engineers and managers want to analyze how their tests are performing, which is the role of test performance analysis (TPA) in a test data management (TDM) system.

Defining TPA
TPA is the multi-dimensional data analysis that gives immediate visualization of test data from different perspectives. The information gained from TPA allows engineers and managers to quickly identify problem areas and take immediate action to improve product capabilities and manufacturing efficiency. By representing the test data in a multi-dimensional cube, the test engineering team is able to analyze and view data from an unlimited number of perspectives (Figure 1). Some of the most common perspectives are combinations of test results by:

date test station product product line operator test revision

The technology that enables the immediate, multi-dimensional viewing of TPA is online analytical processing (OLAP). Without TPA, reports take hours and even days. With TPA, these same reports are visible in seconds, even with gigabytes of data. For more information on OLAP, see the An Overview of OLAP Technology article in this newsletter.


Example of TPA
A commonly used application of TPA best illustrates its capabilities and benefits. For an example, consider a system of three test stations whose purpose is to characterize prototype transmitters. The transmitter family in this example consists of three types: a low cost transmitter, a high performance transmitter, and a government, special-ordered transmitter for use in military applications.

Test engineering teams commonly want to first determine which product tests are most frequently failing. For this example, this is accomplished by viewing:

          Failure Total
                    by all transmitter types

Figure 2’s TPA results show in spreadsheet and graphical form that the low cost transmitters have the highest number of failures. With this information, the test engineering team may want to perform additional TPA to pinpoint the cause of the low cost transmitters failures.

Figure 2: To determine which product tests are failing most frequently, the engineering team can use one-dimensional TPA to view failures by product.  Enlarge Image

A second dimension to add is failures by oper-ator. By adding this second dimension and narrowing the analysis to the low cost transmitters, the combined TPA becomes:

          Failure Total
                    by low cost transmitters
                    by operator

Figure 3’s TPA results identify that Cindy and Jeff have a significantly larger number of failures with low cost transmitters than the other testers.

A third dimension to add is failures by test stations. With this third dimension, the combined TPA becomes:

          Failure Total
                    by low cost transmitters
                    by operator
                              by test station

Figure 3: A two-dimensional TPA of failures by product by operator shows operators 'Cindy' and 'Jeff' have a significantly larger number of failures.  Enlarge Image

These TPA results, shown in Figure 4, identify that most failures are occurring on test station 3 when either Cindy or Jeff is the operator.

By adding date as a fourth dimension, the TPA will identify if the failure problem is occurring consistently across all dates. Other measurements that can help visualize the test performance include the percentage of failures and the total number of failures. Furthermore, the TPA can be narrowed to only view failures when Cindy or Jeff are operating the station to reduce the quantity of information presented. The combined TPA then becomes:

          Failure Total &
          Failure Percentage &
          Total Number of Executions
                    by low cost transmitters
                    by operator
                              by test station
                                         by date

Figure 4: A three dimensional TPA of failures by product - by operator - by test station shows a potential problem with test station 3.  Enlarge Image

Figure 5’s four-dimensional TPA of the three measurements identifies that the failures are not limited to any specific date.

Using Test Performance Analysis
All the tables and charts shown in the previous example were copied and pasted into this article from a standard web browser using Arendar Enterprise 2006’s new TPA reports. Simply dragging and dropping the dimensions and measurements from Arendar’s TPA pallet to the workspace created the reports.

These examples used Microsoft SQL Server as the database to store millions of data measurement. Although this example only showed four TPA dimensions and the measurements all related to pass/fail data, the dimensions and measurements that can be added to the reports are unlimited. In addition, Arendar’s TPA is able to narrow the dimensions so that the engineering team only views the data necessary to make informed decisions.

Figure 5: A four-dimensional TPA with additional measurements of failures by product - by operator - by test station - by date shows no correlation to date.  Enlarge Image

Before TPA, creating multi-dimensional test reports typically took days. With the introduction of Arendar 2006, these reports for the first time can be created in seconds. Because access to TPA reports is through a standard web browser, engineering teams can grant secure access to view or create these reports from their test data to anyone in the company at any geographic location with internet access.

Ask the Expert
Email questions concerning this article to:
expert.testpronews@vi-tech.com