TestProNews Q1 - 2007
Build or Buy a Test Data Management Solution?
So you want to make the most of your test data and have realized the benefits of having a test data management (TDM) system. The question now is whether you should build or buy your TDM system. This question is more complex than it seems. Among the many factors affecting this decision is coming to the realization that for multi-user, multi-system, or multi-product test configurations, a database-centric system for storing and managing your test data is better than a file-based solution. Database-centric TDM systems offer many advantages over file-based systems such as concurrent data access, enhanced data integrity and security, centralized data storage and management, standardized data formats, scalability, and data aggregation. This article assumes that you understand these advantages and therefore concentrates on the next step: deciding whether you should build or buy your TDM system.
There are two options for such a solution: designing, programming, and deploying a TDM system from the ground up or buying an off-the-shelf solution. When talking about off-the-shelf we are not referring to shrink-wrapped software that you can simply install out-of-the-box and run. Such shrink-wrapped packages may offer many features, but are hardly customizable or adaptable to your company’s specific needs. True enterprise TDM systems that can efficiently handle many concurrent data producers and consumers are usually complex solutions that require some customization, system integration and deployment to make them work for you (figure 1). This is similar to enterprise business systems such as SAP.
|
Figure 1: Arendar is an out-of-the-box TDM solution that is customizable to meet your needs. |
There are many benefits to a commercial off-the-shelf solution including lower deployment costs, more built-in features, better reliability, greater flexibility, and scalability because such systems are in essence an extensible platform with many built-in capabilities. The platform also includes interfaces and modules which are fully customizable. Improved quality results from a larger number of users identifying problems, extensive testing, and sound architectural design principles built into the product by experts in TDM and enterprise systems.
Considerations for Developing Your Own TDM System
Even a basic TDM system should rely on commercial database technology and products such as Microsoft SQL Server. The development and deployment teams must be well trained in order to successfully configure, tune, operate, and manage the powerful and often complex capabilities of a commercial database product.
The complexity doesn’t stop there if you decide to build it yourself. In order to develop your own TDM system, the team must also possess database design and SQL skills along with the ability to normalize the database, establish data relationships, and replicate and synchronize the data. This ensures that your system maximizes storage capability, maintains data integrity, and efficiently distributes data to multiple servers and test stations. These designs rely on database technologies such as ADO.NET, ODBC, and XML. Adequately selecting the right drivers, database schemas, and interfaces is essential to the successful implementation of your TDM system.
Ensuring that the data is secure and accessible only to the right people requires an in-depth understanding of network and database security paradigms. Transporting test data through the network reliably requires mastering and implementing technologies such as Microsoft Message Queuing (MSMQ), data buffering, importing and exporting data, and Web services. Each technology has its own set of advantages and limitations depending on the peculiarities of the system. Efficient data transfer is also dependent upon factors such as network topologies, firewalls, and even virtual private networking (VPN). Off-the-shelf solutions are designed to maximize the performance of the system while taking all these factors into account.
Your system should be designed to suit not only your immediate needs, but also account for future requirements and additions. Whatever your preferred test system programming language, be it LabVIEW, TestStand, VB.NET, C#, or C++; your TDM system should integrate well and provide standard interfaces so that both legacy and future test systems fit into your TDM strategy.
The most efficient way for sharing data with many users across the enterprise is through a standard Web browser. However, providing sufficient flexibility so that anyone with the right permissions can log into a simple Web page and manipulate test data stored in a database is not a trivial task if you choose to develop the TDM system yourself. Users want to interactively filter and select the data they need, create reports and custom data views, extract and export information to other applications, all of which require expertise in Web technologies such as HTML, XML, ASP.NET, PHP, Apache, Internet Information Server (IIS), AJAX, etc.
There are many skills and technologies that are essential to putting a home-grown TDM system together. Undertaking your own development effort requires a development team with highly-skilled people in software architecture principles (figure 2).
Figure 2: A diverse set of skills and knowledge are required to develop an enterprise TDM system.
|
Development Time
Companies that offer off-the-shelf TDM products have invested hundreds of thousands of hours, using teams of experts with extensive training and experience in computer science, database, communication, Web, and test and measurement technologies to ensure that their products truly present an all-encompassing, full-featured solution. These experts are focused exclusively on TDM solutions and are constantly looking for ways to improve productivity, efficiency, and ease of use for test engineers and managers who rely on their TDM solution to manage their test data.
It is easy to underestimate the time and cost required to develop a TDM system from the ground up. It is a complex, project-management-intensive effort even when the team working on it is well versed in these technologies. Any internal team that is tasked with developing a TDM system, where it is either second to other responsibilities or simply not within their core expertise, can easily waste a lot of time and effort that results in a system that is not maintainable or manageable. To further complicate matters, there are the tens or even hundreds of business owners each contributing a list of must-haves, nice-to-haves, and other requests which inevitably result in feature creep.
You could argue that the cost of using internal resources to put together such a system is negligible compared to submitting a purchase order for a commercial system. However, the internal development cost is often miscalculated when you consider the added impact of employee benefits, overhead, long term maintenance and support, and loss of productivity impacting other duties for which the team member is responsible. This is by far the biggest expense, which occurs when people get moved on and off the project because of other priorities, requiring the project to be put on hold or delayed while new personnel are trained (figure 3).
Figure 3: Companies should not underestimate the time and cost to develop, maintain, and support their own TDM solution.
|
Where off-the-shelf TDM software can be configured and deployed within days, a home-grown system will at a minimum take months before the first trials can be run, and in some cases will take years before the entire system is complete – if it ever makes it to that point. In many companies, the opportunity costs of waiting on this system can be many times the development cost of the system. The money and time invested in the proprietary system is lost and the company ends up purchasing an off-the-shelf product in order to get a useable system for their application.
A few companies have succeeded in deploying their own home-grown TDM system. Most were completed before off-the-shelf products were available, and so they had no choice but to invest large amounts of money and resources to get a workable solution. These companies spent years in development and millions of dollars to achieve their goal.
Maintenance
Given the accelerated release of new software technologies and the ever-growing user demands maintenance and updating of a system is essential. New versions of operating systems, database technologies, networking and security protocols, and test software put significant pressure on a TDM system to incorporate these new technologies over time or risk increased support costs, even obsolescence.
For the TDM system to be kept up to date, it must be designed in a modular fashion and well architected. You must also ensure that the right team members with the right skills are available for long-term maintenance and support of the system when required. The advantage of an off-the-shelf solution is that they evolve over time to take advantage and readily incorporate new technologies, isolating you from the complexities of having to learn these new tools and then update your software accordingly
Product Quality
Off-the-shelf software, whether for TDM or for any other purpose, is the result of state-of-the-art software development methodologies that utilize multiple development iterations, extensive alpha and beta testing, usability feedback, and continuous improvement of the code base resulting in a net increase in product quality. With every release of the software, existing features are improved and new ones are added. As users find new ways to use the product and funnel requests back to the manufacturer, the software is enhanced and fine tuned to increase the return on investment for the user.
Summary
An off-the-shelf TDM software product from a well-established company that specializes in TDM is significantly more cost effective and will be of higher quality than the development of a home-grown TDM system. The costs required to put such a system together using in-house resources include obtaining the right project team with the required skills and knowledge, the cost of the actual project development time, and the long-term cost to maintain and support the home-grown TDM system after its completion. The ease of use and product quality is also higher with off-the-shelf TDM software because of the iterative approach to continuously release new improved versions based on user feedback and modern software development principles.
Gricha Raether
Director - Sales and Marketing
VI Technology
|
|
|
|
Read about the latest information for integrating automated test systems throughout the enterprise. |
|
|
|
 |
Download PDF |
|
|
