As we have previously examined, the cloud has opened up many new doors within the world of performance testing. We are now seeing a new use case for cloud performance testing emerge, not simply a means to validate scale, but as a service to validate deployment options based on size, scale, and cost: load testing for capacity planning. This trend is a sign that the power of pricing/performance has shifted from operations teams to development and QA.
Right Deployment Sizing and Capacity Planning: Pre-Cloud
Before cloud-based services and testing solutions were readily available, the process of capacity planning and/or determining the right size for deployment was more art and less science. We would build performance tests upon base level hardware, perhaps a standard model of an HP Server, for example. The results from this would allow us to make an educated guess. For example, what if we had 4x load on servers 2x as powerful? Unfortunately there many unknowns in this process that would not hit us until the hardware was deployed and scale was realized. Your best decision was to purchase larger than required capacity.
Pain Points
- Rarely had the exact production hardware
- Never had a true duplicate of production
- Required operations teams and physical delivery
- Always needed to purchase above capacity
It was difficult to determine which deployment options would best suit your needs and be the most economically viable. Again, it was more-or-less a series of guesstimations.
Enter: The Cloud
The cloud service providers removed many or perhaps all of these obstacles highlighted above. We are now 1 click away from having a server with 1.9 TB of RAM and 128 CPUs or a functional DB on a 32 CPU server. Our unlimited choices and options have now become our biggest obstacle. So which one do you choose, and why?
Pain Points
- Unlimited Options
- Choice Performance vs Price
- How to determine the right mix
Enter: Cloud Testing Solutions
Now with cloud based testing solutions, capacity planning for the right size cloud deployment has become more of an exact science. We can now cheaply test a range of different scenarios and configure our application and throughput to match our minimum and maximum requirements (and every size in between). We can build our performance tests once, and if we are really talented, we can run all three scenarios simultaneously.
With these cloud load testing solutions, we can now create real simulations that represent all of the various deployment options that we could potentially use. Instead of just guessing which deployment size would be ideal, we can run an unlimited amount of load tests so we can precisely determine which specific option works the best and costs the least. This all leads to solving the question: which deployment option will give us the right balance of price and performance?
The Relationship Between Price and Performance
Typically, the more money we spend on better servers or on a certain type of deployment, the more performance gains we realize. However there comes a point where we spend more money, but performance stagnates: the level of added performance in relation to cost reaches a peak and then tops off. The chart above is a simple representation of this. At around $4,100 dollars per month, we realize a performance increase of some 393 milliseconds. However no matter how much more money we spend on this deployment option, performance will not increase (or at the very most, only marginally increase), resulting in wasted funds. This peak occurs at a very specific point, and that point is exactly what we use cloud load testing to find. With cloud-based load testing, we can now test dollar sized scenarios to find the right deployment option. Not only does this allow us to discover the right sizing cloud infrastructure, we are also able to save some money that would have otherwise been wasted.
The Cloud Changed it All (Yet Again)
The cloud is forcing us to reconsider so much of running our IT Infrastructure, but it is also giving us capabilities we never had. In the past we were stuck with shrugging our shoulders and saying “I think it will be enough servers”. Today we can test our hypothesis, come up with costs, and find our limits with levels of certainty that are unprecedented; doing all of this with the click of a mouse.
