API testing strategy: Who is testing your API-driven product?

The beginning of 2020 has been a rocky one. Most organizations around the world are forced to identify new ways to maintain current consumers and reach new ones. It’s a change that not everyone was ready for but it’s a change I hope many will embrace.  It’s a digital world so change is inevitable. Whether your organization has started the Digital Transformation journey years ago, or you just started having an API testing strategy should be a critical element within your lifecycle. I raise this topic because, in my experience, it’s an area that is often overlooked or underfunded.

APIs are everywhere

APIs are everywhere. This means there is a huge dependency on leveraging them to build various products. Whether your APIs are internal or external-facing, the importance of a proper testing process shouldn’t change. You can lose internal consumers just as fast, plus failure to deliver quality can lead to redundancy and impact your organization’s financial bottom-line. The least successful API testing strategy is when the consumer is treated as a core component of the quality assurance process. Features provide little value if the product doesn’t work.

API testing strategy: Coverage matrix

A successful API testing strategy should include a coverage matrix. This enables you to map tests (or suite of tests) to the functional requirements. Functional requirements should map to business requirements. Next, classify your tests to prioritize which tests are run for a given scenario. Example: Level 1, Level 2, Level 3. Level 1 is the most critical and is run every build. You may include a subset of Level 2.

Whether your organization has an established strategy or you’re just trying to get started, your strategy should include all the approaches listed below.

Positive testing

The most common API testing approach is “Happy Path” functional testing. When testing APIs, it’s imperative that both the REQUEST and RESPONSE are validated for each method. In an effort to reduce testing cycles, it’s common to start with required fields only. After the basics have been validated, the testing must be expanded to include optional fields. Optional fields should include testing headers and query parameters as well.

Negative testing

Negative and Boundary testing are two areas that are most overlooked. These tests not only provide a better consumer experience but also protect you from security defects. Negative and Boundary testing addresses the WHAT IF. Telling the developer, “The product wasn’t designed to do that” might be a valid response but the user experience should be the focus.

A negative testing approach should include both valid and invalid values. Execute API calls with valid input that attempts illegal operations.

– Attempting to create a resource with a name that already exists.

– Attempting to delete a resource that doesn’t exist.

API calls with invalid input.

– Missing or invalid authorization token (includes other Authorization/Authentication methods).

– Missing or invalid query parameters (required and optional).

– Payload with an invalid model.

– Payload with an incomplete model (missing fields or required nested entities).

– Invalid values in HTTP headers.

– Unsupported methods for endpoints.

Boundary testing

Boundary testing is a bit more challenging and time-consuming, but it’s still necessary. This approach enables the planning for degraded behavior.

Consider the following scenario:

An API has a method that returns all records and there is no limit parameter.

If your test data is 50 records, response time may be within acceptable limits. What happens when there are 400,000 records?

Knowing the potential impact of returning that much data allows you to plan for a future release while in the meantime, you can document the experience the consumer might have. Alternatively, you may elect to change the API specification or add throttling. This type of testing is reused in stress testing.

Stress testing

Stress testing helps identify limitations within the infrastructure. Using the scenario above, what happens if 50 users perform the same request that returns 400,000 records at the same time or within seconds of each other? This might introduce a security issue because this type of activity could bring your system down. Depending on how integrated your environment is, it has the potential to impact other systems unrelated.

Load & Performance testing

Load & Performance and Stress testing are similar, but Load & Performance should focus on building a test suite that represents the user community. The transactions differ from other iterations as you increase virtual users. Validating concurrent users is also important. When running this type of test, it’s critical that the systems generating the load must be oversized. Generating loads from systems that are undersized will provide false results.

Integration testing

Lastly, integration testing, end to end validation is equally important. It’s not always easy to do because it requires a variety of large data sets, but an attempt should be made. This is probably the only phase during the testing cycle where it’s acceptable to partner with your client base. Every user is different and while it would be nice to be able to test every scenario it’s not a realistic goal.

Final thoughts

As you can see, testing is a critical phase during the API Product lifecycle. Using the API First methodology and DevOps approach allows you to test concurrently, making your team more efficient. Remember, every product will have defects or missed requirements, but by devoting more resources upfront you’re creating a better experience for your user base. A healthy user base translates into much more innovation and potentially more revenue. Read more about the differences between API First Design and API First.