Load and system characteristics
The purpose of bringing up load and system characteristics together is to emphasize the differences between:
-
What is measured, i.e. the purpose of the performance test case.
-
How it is measured, i.e. the method of the performance test case.
Mixing up the two can create a great deal of confusion.
Load is here the conditions a tested system shall be exposed to when some system characteristics are measured. An example is exposing a tested system to a specified load at which the response time of requested services shall be measured.
System characteristics is here what kind of behaviour of a tested system that shall be measured in a test case. The list of possible system characteristics can be very long and the list of possible test methods that can be applied is equally long.
Quite frequently we read about load test when we actually mean measurements of capacity or response time.
In order to make the test case names more clear we therefore suggest that the performance test method is replaced by the characteristics to be measured. An example could be replacing Load Tests by Response Time Measurements under specified conditions or Capacity Measurements under specified conditions.
In the following we will discuss definitions of:
1. Definitions of workload
The best starting point for a discussion of load is by examining the definitions of workload.
Workload describes the total amout of services requests a system under test (SUT) is expected to handle during a performance test.
A workload has two parts: Workload content and Workload volume
1.1 Workload content
The workload content describes the services that will be requested from the SUT during a performance test.
The workload content can be described from two sides:
User session scenarios
A user session scenario describes the workload content as seen from the requesting side (the user side). A user session scenario contains a specification of services requested during a user session and in which order the services are requested, i.e. the flow of service requests during a user session.
Usually a service request flow contains alternative paths depending on alternative responses from a service request.
Requested service profile
A requested service profile describes the workload content as seen from the receiving side (the SUT). The requested service profile contains a list of requested services, where each service request has a specification of how frequently it should be generated, usually expressed as a percentage of all requests.
The sum of percentage values for all service requests is 100.
1.2 Workload volume
The workload volume is a specification of the total amount of service requests (described in the workload content) during a performance test.
The workload volume is specified differently depending on the type of load generation (see 2.1 and 2.2 below).
2. Definitions of load
Load describes how a workload shall be applied and distributed on the system under test (SUT) during a performance test.
2.1 User session defined load.
User session defined load describes load as seen from the simulated users.
User session defined specifies the load as the service requests generated by a number of simulated users executing a user scenario.
The total load on the system is in this case determined by the number of concurrently simulated users running the specified test scenario. In order to increase the system load, more simulated user sessions have to be started.
The service request rate for each user session is controlled by think-time delays between consecutive service requests. The workload content is defined by alternative service requests based on the result of a previous request.
The workload volume is defined as the number of concurrently active user sessions during a performance test. The workload volume is controlled manually or in a load script.
Each started user session executes independently of all other user sessions until the end.
User session defined load is normally used by performance test tools where the user scenario is a recorded session that was performed by a tester.
2.2 Service request rate defined load.
Service request rate defined load describes load as seen from the the tested system.
Service request rate defined load specifies load as number of service requests per time unit, usually per second where the traffic rate specification is independent of the number of simulated users.
Service request rate driven load is based on a central load control function in the test tool keeping track of when the next request shall be sent and by which simulated user. Each started user session is controlled by the central load control function.
The central load control function executes a load script telling what traffic rate should be applied in every moment.
The workload volume is defined as a set of traffic rate load steps times the duration time of each load step.
Transition time between two traffic rates is instantaneous due to the design.
2.3 Load time
Load time describes the total duration of processing a workload during a performance measurement.
The workload volume divided by the load time will give the average load per second or other time base when executing a performance test.
2.4 Load scenario
Load scenario describes how load is applied during a performance measurement.
Load is applied differently depending on the purpose of a performance test, i.e. the system characteristics to be measured.
On this page we describe the following load scenarios:
Constant load
This is a load pattern where the SUT is exposed to a fixed rate of service requests per time unit.
Constant load is commonly used for performance tests of stability and availability characteristics.
Stepwise inceased load
This is a load pattern where the SUT is exposed to a sequence of fixed load increases.
Stepwise increased load is commonly used in performance tests of capacity characteristics.
Stepwise increased load is sometimes called staged load.
Peak load
This is a load pattern where the SUT is exposed to a repeated sequence of short periods of very high load (peaks) followed by longer periods of low load.
Peak load is commonly used in performance tests of robustness characteristics.
Statisically distributed load
This is a load pattern where the SUT is exposed to load according to a statistical model for arrival of service requests, such as a Poisson, F distribution or Erlang.
The load can also be defined to reflect the traffic rate variations during business hours.
Statistical load is commonly used in simulations of service production in system delivery tests.
3. System characteristics
System characteristics show measurements of a system's behaviour under different conditions.
System characteristics are described on the following pages:
Proceed to Performance Measurement Points
