Source lines: 18,494
ADL: 1,858 10% TBD
Ada: 16,636 90% complete
D.5.4 The Incremental Test Process
Although the overall test requirements were extremely complex, the CCPDS-R build structure accommodated a manageable and straightforward test program. Substantial informal testing occurred as a natural by-product of the early architecture demonstrations and the requirement that all components be maintained in a compilable format.
Because compilable Ada was used as the primary format throughout the life cycle, most conventional integration issues—such as data type consistency, program unit obsolescence, and program unit dependencies—were caught and resolved in compilation.
The informal testing inherent in the demonstration activities was far from sufficient to verify that requirements were satisfied and reliability expectations were met for this mission-critical, nationally important system. A highly rigorous test sequence was devised with five different test activities: stand-alone test, build integration test, reliability test, engineering string test, and final qualification test.
1. Stand-Alone Test (SAT). The development teams were responsible for standalone testing of components before delivery into a formal, configuration-controlled test baseline used for all other test activities. SAT typically tested a single component (which may comprise several lower level components) in a stand-alone environment. This level of testing corresponds to completeness and boundary condition tests to the extent possible in a standalone context.
2. Build Integration Test (BIT). This was mostly a smoke test to ensure that previously demonstrated capabilities still operated as expected. A BIT sequence is the primary quality assessment vehicle for closing out a turnover review. A given build turnover may take days or weeks, depending on its size or the percentage of new componentry. The purpose of BIT is not to verify requirements but to establish a stable, reliable baseline. It is very informal, dynamic, and focused on exposing errors and inconsistencies. BITs validate the following:
• Previously demonstrated threads can be repeated successfully.
• Previously defined deficiencies have been resolved.
• Interfaces across components are completely tested.
• The baseline is stable enough for efficient requirements verification testing.
3. Reliability Test. One of the outputs of the BIT process and a turnover review was a stable test baseline that was subjected to extensive after-hours stress testing for long periods of time under randomized but realistic test scenarios. This sort of testing was designed to help uncover potentially insipid, transient errors of major design consequence. Reliability testing logged as much test time as possible while resources were otherwise mostly idle (on nights and weekends).
4. Engineering String Test (EST). These tests focused on verifying specific subsets of requirements across multiple CSCIs through demonstration and test of use case realizations (called capability threads).
5. Final Qualification Test (FQT). These tests were equivalent to ESTs except that they represented the set of requirements that could not be verified unless the whole system was present. For example, a 50% reserve capacity requirement could not be verified until FQT, when the whole system was operational.
The overall subsystem build plan was driven by allocating all reliability-critical components (components that could cause type 0 errors) to build 0, 1, or 2. Figure D-7 illustrates the overall flow of test activities and test baselines supporting this build plan. The sequence of baselines allowed maximum time for the early-build, critical-thread components to mature. These components were also subjected to much more extensive testing, increasing trustworthiness in their readiness for operational use. Sufficient test time was logged to derive an empirical software mean time between
Dev/SAT: Development and Stand-Alone Test
Dev/SAT: Development and Stand-Alone Test
failures (MTBF) that was demonstrable and acceptable to the customer. For example, early builds of the Common Subsystem contained all the components for processing thread state management, fault isolation, fault recovery, operating system interfaces, and real-time data distribution. Roughly 90% of the components that could expose the system to critical failures, causing mission degradation, were encapsulated.
The CCPDS-R build sequence and test program are good examples of confronting the most important risks first. A stable architecture was also achieved early in the life cycle so that substantial reliability testing could be performed. This strategy allowed useful maturity metrics, such as those presented in Section 13.3, to be established to demonstrate a realistic software MTBF to the customer.
Was this article helpful?
What you need to know about… Project Management Made Easy! Project management consists of more than just a large building project and can encompass small projects as well. No matter what the size of your project, you need to have some sort of project management. How you manage your project has everything to do with its outcome.