TestProNews Q3 - 2006
File vs Database Options for Test Data Management
One of the major challenges that companies face when testing their products is managing the large quantities of test data generated by automated test systems. The role
of a test data management (TDM) solution is to solve this problem. TDM systems capture,
aggregate, and organize test data making it available for analysis and reporting.
A good understanding of the architectures available in TDM systems is essential
because it determines the security, expandability and performance of the TDM system,
as well as the efficiency of your organization when using the system. The architecture
of TDM systems can be grouped into two categories
(Figure 1) :
1. Test data is stored in individual files
2. Test data is stored in a database
|
Figure 1: The architecture of TDM systems can be grouped into two categories: individual file based architecture or database architecture |
With a file-based TDM architecture, you store test data in standard flat file formats,
use common-purpose search tools to find and index the data, and import the disparate
test data into specialized analysis software (Figure 2).
The most common tools used today in this flat-file TDM architecture rely on the
Microsoft Windows file system for storing data, Windows or Google desktop search
to find data, and Microsoft Excel to analyze and report on the data.
The skills required to implement this type of TDM system are the basic computer
software skills held by test engineers and technicians. The cost for this system
is very low - in the $1k or less range.
Figure 2: With a file-based TDM system, you store test data in standard flat file formats, use search tools to find the data, and import disparate test data into specialized analysis software.
The benefit of this type of system is that it is simple to implement and use. Test
software can easily be configured or programmed to create flat files. You then use
either the same or similar searching features for your test data as you would use
to find other files on your PC. Data security is controlled in the same way you
share files on your PC over the network relying on built-in Windows security.
The disadvantage of a file-based system is that its simplicity results in a lack
of capabilities needed to deliver the efficiency of use, security, and performance
of a large or expandable TDM system. Because these systems are built on the Windows
operating system (OS) instead of a secure database and appropriate IT infrastructure,
none of the advantages of enterprise database technologies can be readily leveraged
for test.
One of the biggest disadvantages that users of automated test systems experience
are the inefficiencies associated with retrieving data and reports from the system.
Because the data is not organized in a consistent manner or in a format that facilitates
drilling through multiple files, finding, correlating, and comparing your data can
be a tedious process even with the greatest search capabilities. This becomes even
more of a problem as data is spread out over different test stations.
The file-based TDM system is also inefficient because of the time it takes to aggregate
data from different files. For example, one of VI Technology’s customers in the
semiconductor industry originally opted for a file-based system. However, once the
test department started acquiring large amounts of test data, they realized that
the time required for the computer to retrieve the data from each of the test stations,
aggregate it, and create a report took over twenty four hours per report. In order
to reduce this time and increase the department’s efficiency, VI Technology deployed
a database TDM system that reduced the required time to create the same reports
down to seconds.
Additional deficiencies of file-based data management systems include limitations
in accessing test data and reports, which directly affect usability for the user.
For example, accessing data stored off-site with a file based system is usually
hindered by firewalls, disconnected systems, slow transfer speeds, unreliable networks,
etc. Therefore, data access is frequently restricted to repositories inside the
company intranet or possibly only on the test station itself. In addition, the Windows
operating system and many common applications used for analysis and reporting do
not elegantly handle concurrent data access on the same file. Therefore, you will
find instances in which you are unable to access data, that data is lost, or a file
is corrupted due to a user accessing a file at the same time that a test station
is writing data to the file.
Data security is another significant disadvantage for the file-based system because
the security features are limited to what the built-in Windows OS security system
can provide. Windows security must be granted to an individual at the folder level.
This means that once you give a user a certain level of access privileges to a folder,
those privileges permeate and apply to all subfolders and data in that folder. You
can potentially solve this problem by breaking up test files into multiple folders, but obtaining a useful level of granularity for even a modest number of files and
users usually becomes very complex. This complexity increases as the number of folders
grows.
For example, some folder-based data management layouts rely on first breaking up
test results into folders by day. Then, within each folder you would create subfolders
for each type of product tested. If you have 3 products and 5 users, you would find
yourself wrestling with setting up security privileges approaching 500 entities
in the first month alone. This security maintenance problem is more extreme when
the data files are stored on multiple test stations because you now must go to each
test station and set the different privilege levels for each user.
As an attempt in solving this problem, test engineers often recruit assistance from
the IT department in order to set up folders on a server. However, the test engineer will often run into problems and delays as they now have to involve IT every time
they want to make changes to privileges. This eliminates one of the key reasons
the file-based system was chosen in the first place: to avoid IT involvement.
You may also find that creating dynamic reports based on data stored in folders
is a seemingly viable option. However, problems with a Windows-based file system
often occur because you may set the privileges correctly for a user on the report
folder, but the user of the report may not have the correct privileges on all the
folders holding the data the report needs to access. If you need to reconfigure
the structure of your data because you add a different type of user, you now have
to also re-organize the test data into different folders which breaks the links
of the reports to the test data.
File-based systems also pose significant limitations in terms of establishing company-wide
standards for data file formats. As such, the problem of data chaos continues to
grow over time as each new test developer creates the test data files in his or
her own unique format without any constraints of a vision for organized test data.
Because the file-based system relies on common search tools and specialized analysis
tools, anyone needing to create a report on the test data must have some understanding
of the data locations and how to use the analysis tool. This requirement prevents
engineering managers from easily running needed reports without learning the lower
level engineering file details and tools.
Even with these limitations, the file-based TDM system does have its place for small
test setups or bench-top systems. The simplicity of the file-based system makes it a viable system for a single test station, with a very small number of UUTs,
running only a few tests per UUT. File-based systems are appropriate for labs and small design groups, but not for manufacturing operations, as these require a different
database TDM system.
Database TDM Systems
The main architecture of a database TDM system supports the capture of test data
into a unified, industry-standard database and utilizes ubiquitous, readily-available
Web technologies through enterprise TDM software to access data for analysis and
reporting.
The most common tools used today in this TDM architecture are a Microsoft SQL Server database, Arendar TDM software, and Microsoft Internet Explorer.
The skills required to deploy this type of database TDM system are basic database
installation skills. The only skill required to use this system is the ability to
use a standard Web browser. The typical entry cost of this kind of system is now
under $20k.
A key benefit to the database TDM system is that it leverages all the benefits of
commercial, enterprise-grade databases and technologies into test. This system overcomes
all the limitations previously described that are specific to a file-based system.
The efficiency of using such a system increases dramatically because all data is
organized in an industry-standard relational database. Aggregating and correlating
data to extract information from different tests is an implicit and automatic task
with near-immediate response thanks to database technologies such as Online Analytical
Processing (OLAP.)
Access to data and reports is available to anyone with the appropriate security
and user privileges from any location as long as they have network access and a
Web browser. A file-based system typically requires IT department involvement to
open ports or set correct security permissions on folders residing in corporate
servers. In comparison, a database TDM system does not require IT involvement in
order to configure who can access test data and reports.
Figure 3: The most common tools used today for an enterprise TDM system include Microsoft SQL Server, Arendar TDM software, and Microsoft Internet Explorer. |
Database TDM technologies are designed to handle concurrent access and complex security
requirements. As such, the problems of data loss, corruption, and access privileges
are eliminated.
The learning curve required for a user to access data and reports is much shorter
with a database TDM system. The database TDM system provides an all-in-one environment integrating search and analysis capabilities through a familiar Web browser interface.
However, users still have the option of using familiar scientific or specialized
analysis software with the database TDM system enabling all required analysis capabilities
beyond what the product offers out of the box.
The database TDM system establishes a framework for data organization, such that
the data structure becomes standardized throughout the company. With a database
TDM system, the problem of chaotic data is solved once in a simple yet elegant fashion.
The most common concern about database TDM systems is the upfront product and installation
costs. Before the introduction of enterprise TDM software, such as Arendar, development
time and costs were prohibitive for most companies. But with today’s latest enterprise
TDM software, customers are regularly up and running within days and the costs, which include the industry/standard database, are well within most test department
budgets.
The database TDM system is the most efficient and scalable solution for all but
the smallest test
applications. Once you have a large number of test results, multiple
test stations, multiple users viewing and consuming test data, or multiple people
viewing test reports, the database TDM system becomes the most cost effective and
requires the least amount of time to maintain.
In terms of manufacturing operations, the database TDM system is the only viable
system given the sheer quantities of data involved. For contract manufacturing operations,
only the database system can provide the required security and data accessibility.
The database TDM system is ideal for design and manufacturing, and as such your
company can use the same TDM tool in design and manufacturing, even if you outsource
your manufacturing operations to a contract manufacturer.
Best of Both Worlds
Both the file-based TDM system and the database TDM system suit specific applications
and each have benefits. If you want a simple yet rudimentary system to find and
analyze your data in the design lab, then the file-based system is appropriate.
If you want a professional, scalable, and efficient system to securely manage your
data with universal accessibility, a database TDM system is the most appropriate
(Figure 4).
|
Figure 4: Comparison of File Based and Enterprise Database Options for Test Data Management |
Ask the Expert
Email questions concerning this article to:
expert.testpronews@vi-tech.com
|
|
|
|
Read about the latest information for integrating automated test systems throughout the enterprise. |
|
|
|
 |
Download PDF |
|
|
