The world is not binary – every time we look closer at a binary split it becomes more fuzzy. Even the computer bit 0/1 is a simplification of low and high voltage. The historic idea of testing frameworks to have either functional or non-functional requirements in focus is failing us similarly. There is much more to testing activities than both functional and non-functional requirements.

Compliance

One of the testing activities that fail to be recognized in the functional/non-functional split, is compliance requirements. US SARBOX and EU GDPR were just the start, on the horizon are EU NIS2, US FISMA, WCAG, and others. WCAG deals with accessibility for websites and is a good example of a compliance requirement that is part functional (dictates functionality: alt-text, screen reading) and part non-functional (dictates contrasts and navigational aids). Many emerging governmental regulations are around broader security controls and information security. To be compliant your organization has to do secure application development – the requirement is to your organization rather than the product. Yet the requirement still has to be met – the business “license to operate” might depend on it.

Operational Readiness and Operation Trials

I often work with solutions that are outsourced to where I work. There might even be a consortium (collection) of vendors in the mix for building the thing. These projects have activities called “Transitional Trials,” “Operational Trials,” or “Operational Readiness Tests” [I also wrote about this in 2017]. The terms seem to come from a more physical form of delivery. These trials fall under my hat in my role as test manager/test lead aka “project manager of the testing and trial activities“.

Operational Trail is often a 30 days period with the solution in full operations with live users, where it’s confirmed that the whole ecosystem of the solution is operational and follows the service level agreements and ITIL processes. The “test cases” of these phases could be:

• Perform a penetration test
• Perform backup and recovery procedures
• Collect monthly reporting on the SLA and system response time
• Collect monthly reporting on batch processing

The requirements come from the outsourcing contract, not from a specification of the system’s functional- or non-functional requirements. It’s a specific activity on its own that needs someone to manage it and write the confirming “test reports around it”. And yes a “trial” report similar to a test report is often required.

Contractual Confirmations

A last testing activity that might fall under the scope of the project test manager, that is not classically functional or non-functional is the activity of gathering evidence of contractual deliverables. the last couple of years I have been part of projects, where we had not only to deliver the system and functionally test the system – but, had also to collect evidence of the contractual deliverables.

The outsourcing contract converted its textual requirements into a huge spreadsheet of ~1000 lines. Each line had to point to objective evidence of each contractual requirement being fulfilled. Examples:

• Backup procedures, requirements for RTO/RPO
• Production data access controls
• Network security and integration protocols
• Processes for yearly financial controls

This could also be a testing/trials activity, as we would create a plan document of the scope, List the items under test, confirm each item by finding evidence, fix bugs, and finally collect a report.

To sum up

These three testing activities are about the testing – not the testers. The actual performance of the controls might be done by a range of very different subject matter experts. If any of the above testing activities fails to be handled – the business is equally at risk as if it was a functional or non-functional requirement failing.