Testing Methods and Terminology


There are three main terms for testing by simultaneously acting multiple users (real or simulated): load, performance and stress testing.

All performance and stress testings require workload definition as a part of the test. There may be no load, minimal load, normal load, above normal load or extreme load. So we will use the term "load testing" as a general category for all types of testing by simultaneously acting multiple users. The definition for load testing is:

"Any type of testing where realistic (or hyper-realistic) workloads are characterized, simulated and submitted to the system under the test."

Extreme loads
are used in stress testing - to find the breaking point and bottlenecks of the tested system.
Normal loads are used in performance testing - to ensure the acceptable level of performance characteristics like response time or request processing time under the estimated load.
Minimal loads are often used in benchmark testing - to estimate user's experience.

Anyway, load, not performance, is the shared characteristic across all these types of testing.

In stress testing you should try to break the application by an extreme load and expose the bugs that are likely to appear under the stress, such as data corruption, buffer overflows, poor handling of resource depletion, deadlocks, race conditions etc. The fact that performance metrics such as response time can be measured during the stress testing is practically irrelevant to the purpose of stress testing. So stress testing and performance testing have totally different goals, which make it clear that stress testing is not a species of performance testing.

Another reason for putting performance and stress under the load testing is another type of testing known as background testing. In background testing you use workloads (usually normal) to exercise the system under the test while you run functional and/or regression tests against the system. The goal of background testing is to test the functionality in more realistic conditions, i.e. with a realistic background workload, like the application will have in real use.

But some sources treat Performance testing as a general category instead of Load testing. This approach is oriented towards test results (measured timings) while the using of Load testing as a category is concentrated on the test nature. We do not follow this approach, but it is rather widespread, so it should be mentioned.

For example, Rational Unified Process defines these terms as follows:

Performance testing is a class of tests implemented and executed to characterize and evaluate the performance related characteristics of the target-of-test such as the timing profiles, execution flow, response times, and operational reliability and limits.

Included within this class are :

Load testing
- Verifies the acceptability of the target-of-test's performance behaviour under the varying operational conditions (such as the number of users, number of transactions, etc) while the configuration remains constant.

Stress testing
- Verifies the acceptability of the target-of-test's performance behaviour when abnormal or extreme conditions are encountered, such as diminished resources or extremely high number of users.

Briefly:
Performance testing is the overall process,
Load testing checks if the system will support the expected conditions,
Stress testing tries to break the system.

Next page