The second part of our blog posting series (Check out Part 1 if you haven’t yet) shows an overview on the different types of performance testing. Learn more about load testing, scalability testing, stress, spike and soak testing, configuration testing as well as availability and resilience testing. The article is based upon a talk I gave at the AWS PopUp Loft, DevOpsCon and other occasions.
In the last blog post I wrote about performance, scalability and the importance of performance testing in the cloud era. So, now, let us take a closer look at the performance testing methods.
It is needless to say that some of the types of testing are not really special when it comes to a cloud environment, but others are especially interesting.
Load testing is sort of the simplest form of performance testing. You induce a normal or expected workload to a system under test and observe it. You can use load tests to determine general system behavior, latency and throughput. In general load tests are used to verify your quality criteria.
Stress testing is basically a load test, but we are applying a higher-than-expected workload and see how the system behaves under serious stress and when exceeding the design limits. You want to learn when your system breaks and how it starts to fail when being in a serious traffic situation.
A typical approach is to steadily increase the load to see where the system under test begins to violate its non-functional requirements. You can use this “tipping point” to describe the capacity of the given system, like “we can handle 1000 concurrent users per application server before we start to violate our quality requirements”.
With scalability testing you are changing the perspective to answer the question: How effective can I grow? You can run a series of stress tests and gather data on how effective you really are.
Using stress tests in a series where you steadily increase the system’s resources, you can easily tell if your system can translate this into additional capacity.
Knowing how well and how far your system will scale by adding resources, you can now make an informed decision: You might not need to do anything right now or you need to take action (e.g. remove bottlenecks). Or you simply add more resources to your problem to mitigate a scaling issue for the time being. Suffice to say, that this is a basic requirement for capacity planning and to do cost estimation.
Spike testing can be used to determine how well your system can cope with sudden traffic spikes. It is comparable to a load or stress test, but modeled as a sudden burst of traffic. It can be a good preparation for a planned marketing campaign or an unplanned event like being featured on Reddit or Hacker News. Spike Testing can tell you if you are making good use of the elasticity of the cloud when faced with these kinds of events.
Soak testing, again, is basically a load test where you hold the load over longer periods of time to look for long term effects, like memory leaks, disk space filling up, etc. The duration of a soak test depends on your situation. Usually a soak test runs for several hours.
While load, stress, spike and soak testing are not particular special when it comes to the cloud, the next testing method is one of the most interesting ones: Configuration Testing.
The perspective shifts now to looking at the changes in performance if the configuration is modified. The change might be positive, but also negative in case you want to optimize for costs (remember: Performance is resource usage per unit of work, money is also a resource). The point is that you know and can quantize the change.
Configuration can be almost anything here: your environment, services that you are using, dependencies of your software — all can be seen as configuration.
Configuration testing is, for obvious reasons, a very important tool in order to learn about the impact of a system’s environment to its performance. It is always a series of test runs where you compare and analyze the impact of multiple configurations.
Now, it starts to get really interesting: We have a lot of configuration options to choose from when it comes to the cloud. These options include:
Finally, you have approached the parts that you have more or less fully under your control, like the operating system, network stack and other kernel settings, software/web server/app server configuration, dependencies, etc. etc.
To emphasize this once more: The aim is to look for change in performance. So, you can use configuration testing to optimize for costs as well, while you know what kind of trade-off you are taking.
The last type of performance testing I will introduce is availability & resilience testing. I will only briefly touch this, because this is probably enough for one single blog article. The idea is to look at certain processes and behavior under load and check if you have covered this.
The idea to give this its own testing category is inspired by Principles of Chaos Engineering. You most probably have checked for those points in some form of manual or automated functional testing, but did you also check for them under load? While you aren’t all Netflix and can’t do that in production, at least think about to do that in a test environment using artificial traffic.
Want to know more? Take a look at the previous article:
Next week, we will share the third part of this series.
StormForge’s Sebastian Cohnen speaks about “Last- und Performancetests in der Cloud?!” (EN: “Why Load & Perfomance Testing the Cloud?”) at DevOpsCon 2016 in Berlin. The talk sums up what to expect and what to take care of if you do performance analysis and load tests of systems hosted in cloud environments.
The talk was filmed by the DevOpsCon team.
[EN] “The Cloud scales for us”: Why this is wrong and you should do performance tests
As the article is in german here is a short summary:
The article refers to a talk which Sebastian Cohnen had at DevOpsCon 2016 in Berlin. The author reflects on Sebastian’s description why it is important to run performance tests in the cloud and which methods can be used (stress testing, scalability testing, configuration testing, etc.).
In this press article, Sebastian Cohnen explains how important it is to have a good understanding of the environment when developing, operating, and assuring quality in modern applications and architectures – whether you are in the cloud or not. He states that performance tests offer a collection of tools to build this understanding and describes how to start.
We use cookies to provide you with a better website experience and to analyze the site traffic. Please read our "privacy policy" for more information.
