At the AWS PopUp Loft in Berlin, I gave a talk on why load and performance testing is still (or especially) relevant in the era of cloud infrastructure. I expanded the talk for AWS Summit Berlin 2016 and again at DevOpsCon 2016. Here in our blog, you will find it in three crunchy bites, beginning with performance and scalability and the importance of performance testing in the cloud era. I will give an overview on the different types of testing and explain the differences between pre and post cloud times.
The Cloud™ is infinite and scalable. Period. Why is it important to test for performance and scalability characteristics of a cloud-based system? Won’t AWS scale for me as long as I can afford it to? Yes, but… AWS only operates and scales resources. They won’t automatically make your system fast, stable and — more importantly — scalable. Performance testing is crucial to understand your system and architecture design and your cloud hosting environment.
I state: Load and performance testing is still a thing in the cloud. To prove that, I will briefly cover some basics, starting with performance.
Performance is often used in a not very well defined way. The most abstract description of performance I can think of is the ability of a system to fulfill a task within defined dimensions. It is a measurement for efficiency! The most often used dimension is time and a unit of work could be a request or transaction. This would then describe the performance of a system in terms of request latency for example.
You can extend this definition to describe the efficiency of a system or server to get the performance statement “1 instance can handle 250 rps at a 99th percentile of 250ms”.
Next up: Scalability. That is a term often used next to or even interchangeably with performance even though they are not as much related as many think. While performance is a measurement for efficiency, scalability on the other hand describes how effectively you can translate additional resources into additional capacity.
This distinction is crucial because it can help you want to design a scalable system which you don’t want to fall for performance optimizations when you are actually aiming for scalability.
An example for an ideally scaling system would be: If you tenfold the resources and have tenfold the capacity and you can hold the linear scaling, then you have done a terrific job!
But you have to keep in mind, that one does not imply the other. You can have an inefficient system that scales very well. Think of a video encoding service that can potentially be scaled very well simply by adding servers. But you can also have a highly efficient, optimized system that might not be scalable at all, maybe because the communication overhead between multiple nodes would make that impossible.
Jonas Bonér (Founder & CTO, Lightbend) has a nice presentation on that topic and he asks two good questions:
How do I know if I have a performance problem? If your system is slow for a single user!
How do I know if I have a scalability problem? If your system is fast for a single user, but slow under heavy load!
So what is performance testing? The Wikipedia article on software performance testing gives a nice definition:
In software engineering, performance testing is a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload.
— Wikipedia on Software Performance Testing
In other words: Performance testing is a family of non-functional testing methods, which all:
There are a bunch of testing methods that can be seen as types of performance testing. The main difference is that they have different goals and testing perspectives. I will have a look at all the types in part 2.
I think it is set that the cloud is here to stay. Some of the most interesting key points that the cloud offers us are: IaaS, PaaS and other-things-as-a-Service. Combined with APIs, a high degree of automation and resources that are available on-demand we get a cost effective and scalable environment that was not available in this dimension before.
It is obvious that you should care about performance. But some might ask: Why care about performance testing in the cloud? You can always take out your credit card and add more resources to the problem. But scaling resources is not the same as scaling your application. You have to design your system very carefully to make that statement true as there are many pitfalls to avoid.
A simple example would be that you have your ELB set up (which scales automatically), and your application servers are scaled automatically by the AWS Auto Scaler. But you forgot to take care of scaling your persistence layer as well.
The reason why you still have to think a lot even in a managed environment like AWS is understanding.
You need to understand the performance related characteristics and implications of your application architecture — and it doesn’t really matter if it is a distributed, microservice, or monolithic style system. You are also running your system in a complex environment, in this case the cloud. And you might utilize other third party services that also have an impact on your systems performance.
The reason why you have to have a good understanding on what is going on when it comes to performance is because complexity has not simply vanished. It is a lot easier these days to get started to and run systems based on managed cloud resources and services. But the underlying complexity has merely moved somewhere else (to be someone else’s problem).
The issue is, that complexity can have a non-trivial impact on performance which can make it very hard to reason about a system if you have to handle it as a black box.
Read more in part 2 of this blog series, Types of Performance Testing.
At FrOSCon 2016 Sebastian Cohnen talked about why load and performance testing in the cloud. The talk sums up what to expect and what to take care of if you do performance analysis and load tests of systems hosted in cloud environments.
In this press article, Sebastian Cohnen explains how important it is to have a good understanding of the environment when developing, operating, and assuring quality in modern applications and architectures – whether you are in the cloud or not. He states that performance tests offer a collection of tools to build this understanding and describes how to start.
We use cookies to provide you with a better website experience and to analyze the site traffic. Please read our "privacy policy" for more information.
