Our work on standards has its 15th anniversary in 2022
Methods for improved quality must be incorporated in standards.
This applies to all work on quality assurance of functional and non-functional requirements. Without standard methods for quality assurance, such as measurements of non-functional characteristics, product characteristics can't be compared.
With real standards for measurements of non-functional characteristics, software quality assurance will take a giant step forward. We are, however, just at the beginning on this track, since real standards for measurements of non-functional characteristics are still rare.
What if we don't have a common terminology for performance?
Development of methods and metrics for software performance have long suffered from no commonly accepted definitions.
After 15 years of work on all aspects of performance and performance measurements we still found the following in 2007: If you ask ten people what is system performance you would get at least five different answers.
If we can’t agree on what shall be regarded as system performance or can’t agree on a common vocabulary to express it there will be no possibilities to interpret performance measurements and do meaningful benchmarking.
Of course it can't continue like that. Consequently we started to argue for a much needed standard for system performance at a meeting at ETSI in 2007. The starting point was to establish:
-
A standard for what shall be regarded as system performance.
-
A standard for terminology in system performance.
-
A standard for concepts in performance measurements.
In 2011 our proposed standard for terminology and concepts in performance testing was approved by ETSI as Report TR 101 577. and in 2013 by ITU-T as Recommendation Q.3930.
What is special about measurements of system performance?
Development of standards for performance and methods for performance measurements go hand in hand.
Standards for performance define measurable performance characteristics that are vital to understand the behavior of a product or service.
Standards enable benchmarking of performance where the characteristics of different products can be compared in a fair way.
Measurements of performance characteristics must follow the same measurement standards to be comparable. Furthermore measurement standards must specify measurement conditions to be applied to make performance values comparable.
Standards measurements of performance characteristics shall also include quality requirements.
What is next?
From ITU-T Workshop on Benchmarking of emerging technologies and applications 2019-03-11.
ETSI Report TR 101 577 and ITU-T Recommendation Q.3930 was our first attempt to bring some order in system performance and performance measurements and more will follow. A lot has happened in the telecom industry and the software industry since 2011. At a joint meeting between ETSI and ITU-T in Geneva 2019-03-11 it was decided to proceed with additions to the standard.
