In Part 3 of our Testing series, we talked about independent testing, specifically testing performed by DSPs. Perhaps the most critical aspect of an integration project is the coordinated testing (or Certification Testing) between both parties which so often is known for being cumbersome, slow and expensive.
Typical problems we’ve experienced first-hand include:
- DSPs enter the certification test phase without having undergone solid independent testing
- Errors communicated through a STAR ConfirmBOD do not indicate all business rule violations of OEM back-end systems
- Feedback cycle is 24-48 hours or more
- Test case evaluation requires participation of many OEM personnel to prepare data, coordinate communications and investigate results in back-end systems
- Time is lost due to the need to coordinate testing activities between OEM and DSP personnel
Certification Testing involves testing across multiple QA environments and systems, requiring coordination across multiple parties to determine results and resolve issues. Too often, it’s not until this phase of testing that issues are uncovered which, ideally, should have been identified much earlier in the project (see Part 3). Depending on the problems, certification testing may need to be suspended while the DSP modifies its code. This can add considerable time if the DSP needs to repeat the entire development, testing, QA process cycle. This type of iterative, stop / start process is inappropriate during the certification testing phase when so many people from both the DSP and OEM organizations are involved.
Motive has taken an approach that addresses these kinds of issues by:
- Providing DSP testing apps for use during development and independent testing for earlier identification of problems
- Supporting a 2 phase certification testing process where issues can be identified and resolved during a preliminary phase of testing performed entirely within Motive Integrator
Phase 1 tests communications, and payload (field edits and business rule validation) entirely within Motive Integrator. Phase 2 tests communications and business process end–to-end in the OEM QA environment. DSPs execute defined test cases for each phase and only when successfully passed in Phase 1 are they approved to begin Phase 2.
Our approach has the following impacts on projects:
- Time demand on OEM resources is minimized due to the identification and resolution of problems through Motive’s automated testing in Phase 1
- DSP tester productivity is increased due to instantaneous/clear/comprehensive feedback in Phase 1
- All business conditions and communications requirements are tested during Phase 1 without the involvement of OEM personnel
- Phase 2 testing with the OEM systems is focused more on verification than problem solving
- Overall certification process is faster for both DSPs and OEMs
At the end of the day, our goal is to make the Certification Testing phase easier, faster and ultimately less expensive for both automakers and DSPs.