Application load testing for API endpoints can be done quite easily. You could even do this to compare different API platforms that you’re considering using. That’s what we’ll do here with a real case.
Zip Code API’s have become very popular because of the data they can provide easily and inexpensively. For example, one API when passed a zip code returns the city, state, latitude, longitude, time zone and more. Another will return all zip codes within a given radius of a zip code. And another will find the distance between zip codes.
There are three main API platforms for Zip Codes: Zip-Codes.com, ZipCodeAPI and SmartyStreets. You can look at the websites to see the different features. But all of that data doesn’t matter unless the data is returned quickly since most sites that call the API need to respond to a real person on their website.
So how fast do they return a response to your API call? This blog post will compare these three platforms for response time to show how you can use RedLine13 to do load testing for API endpoints.
The Test Details for Application Load Testing for API Endpoints
Here’s how we did the test.
4 Servers (Virginia, California, London, Frankfurt)
10 users per server, 5 iterations each
500ms to 3s wait time.
4*10*5 = 200 Requests total
– Each virtual user calls the API from Zip-Codes.com, ZipCodeAPI, and SmartyStreets.
– Each server has its own set of zip codes
– Each User-Iteration combination gets a different zip code
– Use RedLine13 for load testing
Using JMeter to load test an API
Read more about how to use JMeter to load test an API.
Adding Regions to a RedLine13 Load Test
This test was a standard JMeter test for comparing API calls, easily testing it on multiple regions on RedLine13 required no special effort. In setting up your load test just add as many load tests from the different regions you desire.
First enable as many regions on your account that you wish. (https://www.redline13.com/Account/settings)
Second inside your load test just add agents.
Putting this together you can run any JMeter test from load agents around the world and collect results.
API Test Results
The graph below shows percentage of responses that are below the times at the left. You can see that most responses are less than half a second.
Here are our summary observations:
- ZipCodeAPI consistently had the best results
- The response times are similar in certain percentiles, except SmartyStreets was slower in certian percentiles and Zip-Codes.com has some outliers as you can see at the right end of the graph.
- They all tick up around the 25% percentile. We’ll dive into that later in the Deep Dive section at the end of the post.
- The nice thing about ZipCodeAPI is that you are guaranteed near consistent performance. This is important, because it’s kind of the nirvana for building a platform. In many systems power users suffer performance for their higher usage, but if you can give them consistent response time they will perceive performance to be as strong as they used it day one.
Zip Code API tools all provide excellent performance with some minor differences. Bottom line: ZipCodeAPI consistently had the best Zip Code API Response Times.
The load testing showed that ZipCodeAPI provided near consistent performance. This is important, because it’s kind of the nirvana of a platform. In many systems power users suffer performance for their higher usage, but if you can give them consistent response time they will perceive performance to be as strong as they used it day one.
For details read on.
Here we will look at more graphs to see what else we can learn.
The first two graphs below also show pretty similar results. The third graph shows that Zip-Codes also returns additional data, likely demographic information, in the response, probably impacting their response time.
Why the 25% performance difference?
We noted at the beginning of the post that the graph showed a bump up at 25% with no apparent reason. Let’s dig into this. Is this just because 25% of the requests are faster than the rest? Is it related to picking four regions to use?
So what happens if we analyze the results from each server?
The graphs show that the fastest response times were from the local AWS Region Virginia. The next best was California followed by London and Frankfurt. The response times were excellent for all regions but they were best in Virgina. That is what accounts for the slight uptick at 25%.
You can read this a few ways
- Region and location to users will impact performance
- Server to server type operations within a region will be much faster
- Load testing results need to take into account cross-region and cross-cloud latencies.