Traditionally in hardware systems, the strategy of increasing the testability of components through built-in testing is a widespread concept. Complex hardware components are designed to be easily tested in a way that can ensure the correct operation before the mass production assembly lines. The necessity to design testable hardware components is related to the high costs of repair then when failures are identified after a mass production stage.
Similarly as shown in hardware components context, we (as professional programmers) must provide the appropriate support for the use of embedded tests in software components. As one of the primary characteristics of testability, built-in tests can be (at least) classified into four secondary groups: 1) test input validation, 2) test assertions, 3) test verification probes and 4) initialization testing.
First, let’s talk a little bit about input validations…
Built-in testing of input validations are used to verify the integrity and origin of the data and providing constraints to the concrete implementation of a software component. The restrictions found in these pre-conditions ensure the minimum conditions for the business rules of the component, without these pre-conditions it would be required to write several test scenarios to evaluate the behavior of the component in front of a large number of possible invalid entries. Once the input validation mechanism is reusable and reliable, the number of test scenarios decreases, reducing the effort required for the approval of the component.
The concept of input validation can also be related to software observability and software controllability. A pre-condition provides a mechanism to observe and control the entry of data into a component. This feature enhances the testability of the component, since the validation scenarios are restricted to the behavior of software in the face of valid entries, reducing the number of test cases.
Besides enhancing the testability, built-in test input validation collaborate for the security of a component by verifying patterns of malicious data. The use of standard security APIs reduces costs associated to penetration testing and increase the reliability of the component.
Secondly, we must consider built-in assertions.
Models, use cases, diagrams and software contracts describe the expected behavior of a component, but these artifacts are related to the component specification rather than the concrete implementation of the component. However it is possible to approximate the rules described in the specification with the implementation of a component by means of tests that validate those rules embedded into specific points of the code. These mechanisms, little specialized tests that check for conditions are known as built-in assertions.
An assertion is a Boolean expression that declares the constraint of a condition that you believe will be true when a single point of code executes. Using this approach an assertive naturally extends the observability of internal state of a component and spreads built-in tests through the code.
In a general way built-in assertions can be enabled/disabled by configuration files or parameters at the initialization of the component, this feature helps quality team during development cycles (with assertions enabled) and application performance in production environment (assertions disabled). As any other architectural attribute, the setting on/off of a built-in assertion depends on the tradeoff between testability and performance of the component.
Assertions can also be used as mechanisms to verify the achievement of non-functional requirements. Performance metrics are an example of using assertive post-conditions on a component. If an assertion is violated (e.g. component processing time must be less than 10 seconds) then a failure message is propagated in order to increase the observability and correctness of the implementation of the component.
In the third place, we can use a verification probe as a built-in test…
Unlike the input validations and using assertive behavior to assess the dynamic exchange of messages between components, verification probe is a passive monitor that reports objects and data in inconsistent states.
Processes scheduled or performed in background trigger verification probes which who work evaluating and comparing data with historical samples. Verification probes can also be used to check the availability of the component at regular intervals.
Mechanisms of pre-conditions and post-conditions directly influence the testability of a component by means of observation and control of the areas of inputs and outputs. However these mechanisms do not check errors and intermediaries inconsistencies at the end of a process execution.
It is possible that a fault occurs and the mechanisms for input validation and assertive post-conditions do not identify it properly. In general, these mechanisms compare the message against an expected format within a predetermined domain. In some scenarios the message formats of input and output can be considered valid, but the state of the message or the object internal data may be incorrect. Verification probes can be used to provide additional control over the internal state of objects and data integrity in order to increase the reliability of the component.
Verification probes extend the testability of a software component; it is also a positive result for operations team to monitor the system behavior. Results of errors found by verification probes can trigger alarms and notifications with the objective of improving communication, reducing system downtime.
As the total observability is unworkable in practical terms, the component should at least provide mechanisms to expose the errors when they occur. There are no advantages of hiding an inconsistent state.
As the probe itself may fail, a double check can be used. For critical components, it is recommended to add a listener that can trigger alarms when a verification probe stops sending data (inoperative probe).
And the last but not the least type of built-in test is the built-in initialization test…
Every software component has a set of preconditions as implicit or explicit rule to allow its proper execution. Introducing code segments to verify the condition of the component and ensures that the call which does not satisfy the condition can be properly observed. It facilitates the identification of faults during the execution of the component.
Failures in integration points represent a major cause of instability in a system. Each remote call, open socket, process, or even the search for records in a database can fail, so we need a centralized service to validate all integration points in the shortest possible time. This validation should be performed immediately after the delivery of new versions (to detect possible integration breaks due to changes) and after operation incidents (as a tool to aid in the detection of problems).
The built-in initialization test mechanism should try to establish connections with each integration point. In languages like Java, C, C#, Ruby etc. there are mechanisms and APIs that make clear when a connection problem happens (such as Java exceptions or returns “-1″ in C). The built- in initialization test component should know how to deal with connection failures and report exactly what is the impact on the affected component.
One of the challenges for initialization tests is linked to the variety types of failures of integration points. These failures can range from network problems to XML syntax errors. The error is not always clear, sometimes the fault may be associated with protocol violations, slow responses etc. Thus the mechanisms of a built-in initialization test may have an abstract core, but the characteristics of each integration point should be considered individually in a concrete implementation.
The output response of the built-in initialization test should present the results of validation to indicate the reason for the failure and facilitate the process of debugging. The initialization test can perform a large number of verifications (database, remote calls, disk access etc), so a facade can be developed to trigger specific test groups (or even all verifications), facilitating the test execution process. The output of the initialization test can display the percentage, the response time and the name of the components that were checked during the validation process.
