Performance measurement metrics

Metrics are the output from performance measurements.

To produce metrics with high quality all captured performance data must comply to a number of requirements.

General requirements

Performance metrics must comply to five general requirements.

Metrics general req
General requirements on metrics

1. Uniquely identified

The naming of metrics must be unique to avoid name conflicts

2. Understandable

Metrics should be accompanied by specifications that make them easy to understand.

3. Comparable

Metrics should be accompanied by recording conditions and processing methods to be comparable.

4. Repeatable

Metrics should specify test case specifications, test tool specifications, and test object specifications.

This is a basic requirement for regression tests of performance.

5. Accurate

Metrics should be accompanied by data recording methods and configurations .

Metric identifiers

Every measurement value must have three identifying attributes.

Metrics 2
Metric identifiers

1. What

What does a metric value represent?

2. Where

Where is the metric value captured, i.e. where is the measurement point of a metric value?

3. When

When was the metric value captured? Performance measurements generate long time series of measurement data. Therefore is the point in time critical for comparisons with other metrics.

Performance metrics are of no value if they are not accompanied by the three identifiers, since you can't tell what they represent.

Metric types

Metrics belong to one of four types.

Metrics 1 types
Metric types

1. Raw data

Raw data are captured data in its original format. Raw data can in some cases be used as metrics in its final format, such as response time.

2. Normalized data

Normalized data are metric figures that relates to normalizing factor, such as transactions per second.

3. Transformed data

Transformed data are metric figures transformed from the recorded value format to another value format, such as memory size transformed to a percentage of total memory.

4. Derived data

Derived data are metric figures resulting from computations based on other captured metrics. Key Performance Indicators are typically derived metrics.

Metric formats

Metrics belong to one of four types.

Metrics 3
Metric formats

The naming of metrics must be unique to avoid name conflicts

1. Value type

Is what a metric value represents.

2. Unit

Is the quantity format a metric value.

3. Accuracy

The precision of a metric value.

Measurement conditions

Measurement conditions are vital to the quality of a metric value.

Metrics 4
Measurement conditions

The measured conditions on a test object when a measurement value was captured. This is used to separate valid measurement data from invalid.

Proceed to Performance Measurement Conditions