Complex test scenario solutions
Complex test scenario targets
Test scenarios can be very complex and a performance test tool must be capable of supporting requested complexity.
Our solutions to complex test scenarios will discuss the following cases:
1. Performance tests of secure access to IMS systems (AKA)
AKA is the 3GPP's standard for secure access to IMS systems.
This was our first topic on test tool solutions to complex scenarios and has been a well visited pages for over eleven years.
Here is a link to these pages.
We also provide redirects to these pages on the new web site for all users having bookmarks.
Send us a message if redirects don't work.
2. Improved simulation scenarios
Traffic mixes are a common way to simulate real traffic. A traffic mix is a set of applied service scenarios where each scenario has a specified share of the total load based on traffic statistics.
The frequency of each type of requested service scenarios in a traffic mix is, however, static during the entire time of running the test case. This bear no resemblance of real traffic where the mix of requested service scenarios varies all the time.
MBC provides a solution where the service scenarios are randomly selected according to a probability value based on their measured statistical share.
This solution provides a traffic mix that varies all the time. At each point in time the traffic mix will deviate from the statistical distribution, but over time the share of each type of requested service will get closer and closer to the statistical distribution.
What is the impact of these solutions in your performance test work?
Contact us for more information.
Complex load scenario solutions
Complex load scenario possibilities
To configure and manage complex load scenarios MBC provides three types of solutions:
1. Complex load configurations
A configured Traffic Handler on MBC is a logical load generator representing one interface between MBC and the tested system.
Figure F-14: Traffic Set and Traffic Handlers
All Traffic Handlers in a test case are configured in a Traffic Set that handles all interfaces between MBC and the tested system.
A test case in MBC has one or more Traffic Handlers configured.
Each configured Traffic Handler has its own mix of:
-
User scenarios.
-
Application scenarios.
-
Application protocols.
-
Transport protocols.
-
Internet protocols.
MBC can be configured to handle very complex workloads with multiple Traffic Handlers configured for different application protocols interfacing different components on a tested system.
2. Complex load control
To manage complex load scenarios MBC provides three types of load control.
Figure F-15: Three types of load control
To provide complex control of load MBC provides three types of load control:
1. Scripted load control
Script load control describes the requested load during a performance test in a script.
The script can start Adaptive load control on all Traffic Handlers, one or more Traffic Handler groups, or single Traffic Handlers.
The script can also set the load directly on all Traffic Handlers, one or more Traffic Handler groups, or single Traffic Handlers.
The script can also contain a mix of Adaptive load control and direct load control.
2. Manual load control
Manual load control takes over the entire control of load for a performance test from the configured load script.
If no actions are taken the load will continue as in the load script from the point when manual load control was started.
Manual load control has the full capability of changing all settings in the load script including stopping adaptive load.
3. Adaptive load control
Adaptive load control strives for fulfilling a performance condition by altering the load until the deviation is within +/- 1% of the target value.
When the test condition is achieved Adaptive Load Control strives to maintain the load condition until the test is finished. This period can be from a few minutes up to days or weeks.
Read more about Adaptive load control here.
3. Mixed load control
To simplify management of complex load there are two options:
1. Traffic Handlers configured in groups
Traffic Handlers can be combined into one or more groups to simplify load management.
Figure F-16: A Traffic Set with three groups of Traffic Handlers
For each Traffic Handler in a configured group a share of the total load applied by the group can be specified.
A Traffic Handler can be configured in several groups, but only one of them can be active at a time.
2. Combined load control
The three types of load control are not mutually exclusive!
Figure F-17: Load concurrently controlled by all three types of load control
All three types of load control can be active concurrently in a performance test job. In the figure above all three types of load control are active concurrlently.
This possibility enables complex explorative performance tests.
Let's assume that normal load levels and mix of requested services is represented by a Traffic Handler group under adaptive load control.
In the next step a peak of an unusual services requests is started on other Traffic Handlers by manual load control or scripted load control.
The reduction of capacity to handle normal load can now be measured directly and with high precision.
This kind of performance simulations can be very valuable in applications with stochastic load patterns.
How does this impact your work on performance testing?
Contact us for more information.
Scalability solutions
Scaling targets
A performance test tool shall be regarded as an intrinsic part of an IT product throughout its life-cycle.
In most cases performance requirements on an IT product will grew dramatically during its life-cycle. A consequence of this is that requirements on upscaling of performance test tools will follow.
Our solutions provide test tool scaling in three dimensions:
Simple scaling of a load generator by separation of logical and physical load
A common problem occurs when required load increases beyond a point where several load generator processes will have to run in parallel on separate hardware servers to create the demanded conditions on the tested system. Such situations increase the complexity in management of generated load and reporting of measurement results.
On MBC the complexity of running several physical load generator processes in parallel on separate hardware servers is hidden by a logical load generator.
Figure F-7: Logical and physical load generators
A logical load generator is a configuration entity that provides a simple and aggregated view to a test user when managing load levels and receiving measurement.
It handles all complexity of running multiple generator processes in parallel on separate hardware servers.
This includes:
-
Automated distribution of simulated users across the load generator processes(Traffic Adapter processes).
-
Automated distribution of traffic frequencies (generated load) across the load generator processes (Traffic Adapter processes).
-
Automated collection of external performance metrics from the load generator processes (Traffic Adapter processes).
From the user's view it is one load generator sending service requests to a tested system and one load generator reporting response times, traffic frequences etc. no matter how complex the load generator setup is.
No manpower is therefore required to handle the increased complexity of running large capacity load generators.
Simple scaling of a performance test case
A configured Traffic Handler on MBC is a logical load generator representing one interface between MBC and the tested system.
Figure F-8: Traffic Set and Traffic Handlers
All Traffic handlers of a test case are configured in a Traffic Set that handles all interfaces between MBC and the tested system.
A test case in MBC has one or more Traffic handlers configured. Each configured Traffic Handler has its own mix of user scenarios, application scenarios, application protocols, transport protocols, and internet protocols.
A Traffic Set with multiple Traffic Handlers configured for different application protocols interfacing different components of a tested system enables MBC to scale to very complex workloads.
Simple scaling of physical resources
Scaling an MBC system for generating more load also requires more physical resources to carry out a larger task.
Figure F-9: MBC a scalable performance test system
An MBC system has two types of servers:
-
Monitor servers managing a performance test job.
-
Traffic servers carrying out the work of the configured Traffic Handlers.
Figure F-10: An MBC system with three Monitor servers and eight Traffic servers
In the figure above the MBC system has been expanded to three Monitor servers and eight Traffic servers. The Traffic servers can be shared between the Monitor servers in any configuration.
The effect on load capacity by adding one or more Traffic servers is linear, i.e. the capacity of each additional Traffic server is equal to each existing Traffic server as long as they have the same hardware configuration.
What is the impact of this solution in your work?
Contact us for more information.
Flexibility solutions
Flexibility targets
Performance test cases contain many configuration elements that can be hard to manage and maintain over time.
With flexebility solutions we mean solutions where:
Test cases that can be reused
With test cases that can be reused we mean that test cases can be applied on differently sized systems with no changes.
If a test case shall be executed on three systems with quite different amount of physical resources it implies that the test case shall be changed to apply three different load levels.
Figure F-11: A test case used on three differently sized systems without any changes
With MBC's Adaptive Load Control there is no need to modify the test case to achieve the demanded test conditions on the three tested systems. MBC will automatically adapt the generated load level to the demanded test conditions on each one of the three systems.
Test cases that can be extended
With test cases that can be extended we mean that test cases can be applied on different types of performance measurements.
Figure F-12: A test case designed for capacity measurements extended to work for stability and availability measurements
A test case that is designed for capacity measurements can be extended to stability and availability measurements by a simple change of test time.
Improved visibility of reliability problems
Figure F-13: A stability measurement with two types of problems
The figure above shows two types of situations during a stability measurement when capacity is reduced.
-
A capacity dip on the tested system caused by a system activity using CPU resources. MBC adjusts the load continuously to keep the tested system at a constant 80% CPU load. MBC will resume the earlier load level when the system activity has finished.
-
A decreasing trend in processing capacity, most likely caused by a problem in the tested application gradually using more CPU resources.
What is the impact of this solution in your work?
Contact us for more information.
Test automation solutions
Test automation targets
Performance tests require complete control of the tested system, i.e. no other concurrent test activities, such as function testing or other performance measurements.
The ideal solution is of course to have separate test systems for performance testing, but in reality this is far from always the case.
When there are no separate systems for performance tests will the test system(s) soon become a time critical resource in Quality Assurance.
This means that performance tests will have to run during non-office hours to avoid collisions with other QA activities.
To enable this our performance test automation solutions provide:
1. Scheduled Performance tests
MBC provides functionality to schedule performance tests.
Scheduled performance tests on MBC:
-
Will start and execute fully autonomously during working hours, nights, weekends, or during public holidays.
-
Can be scheduled up to a week in advance.
-
Need no manual actions other than scheduling when a test shall be executed.
2. Scheduled explorative performance tests
Explorative performance tests are measurements, with the purpose to find out a system's characteristics under stated conditions with no targets set.
An example could be to measure the throughput capacity of a service at 80% CPU load.
Explorative performance tests usually require manual monitoring and actions and can consequently not be sceduled.
On MBC explorative performance tests using Adaptive Load Control can be scheduled, started, executed, and finished without manual intervention.
3. Automated handling of probes
Probes are required to capture internal performance data on the tested system.
Full collection of internal performance data on the tested system must be supported.
All probe handling actions are a fully automated part of all performance test jobs, scheduled as well as manually started.
Probe handling solutions on MBC include:
-
Automated download and start of all configured probes on the tested system.
-
Automated reporting of captured measurement data.
-
Automated download and restart of probes on the tested system if:
-
Configured probes stop reporting collected measurement data during a test.
-
The tested system is partially or fully restarted during a test (can be a deliberate restart in the test scenario).
-
-
Automated stop and removal of all configured probes on the tested system after a performance test is finished or aborted.
4. Automated handling of user defined probes
Probes can also be designed by MBC users. This is an important feature for collecting internal performance data from an application.
User defined probes are configured and automatically managed the same way as the standard probes of MBC.
How does MBC's test automation improve your performance test activities?
Contact us for more information.
Subcategories
- You are here:
-
Home
-
MBC Solutions
-
MBC Complex Scenario Solutions
-
AKA Secured IMS Access
- MBC Solutions