Why shift-left performance testing?

Performance testing is one of the non-functional types of software testing that mainly focuses on: speed, availability, response time, scalability, reliability, and other things that define the performance of an application. The main objective of performance testing is to identify performance bottlenecks and fix the root cause to remove the bottlenecks. (1)

A fancy and good application design is one thing, however, without good performance, most people are not likely to continue using your application. For example, recently, people want to be able to do groceries in just 5-10 minutes by simply going to the app, browsing items, adding them to their cart, and paying. Done. 

Another example could be travelling. Due to globalisation, more and more people need an app to allow them to select a flight and complete payment within just minutes. If there is too much friction, they’ll get frustrated quickly and switch to a different site. (3)

Our current approach to the development phase does not yet include a performance test as we tend to focus on the application’s functionality. A performance test is usually carried out during the last phase of development or before the release phase. This method has some problems that may impact a software project, such as:

• Critical issues found in the latest phase of performance testing
• Fixing issues takes a long time, inducing a delay to release time
• Customers may become unsure about the timeline and they could lose trust 

How do we cater to this problem to prevent issues during the final phase of development? We take an approach called shift-left testing. Shift-left is an act of bringing the process to the “left” of a certain cycle/process. In terms of software application, it means bringing the process to be as early as possible in the software development cycle. (2) 

Some benefits of doing early performance testing include:

• Finding bugs or issues early in the development phase
• Providing better feedback for the application in early stages
• Fixes and improvements can be done at early stages
• Preventing development over the timeline if there are any issues found

Even when using an early test, there are always drawbacks when implementing this method, such as:

• Reduced focus on the functionality test
• More resources are needed to appoint or to focus on each task in sprint

Approach to shift left performance test

We are going to implement this approach by using postman and k6. We know that postman is limited to API functionality tests, so we will also use a library that helps us convert the postman collection into a k6 script. First, we will create a script using the Postman GUI, then we are going to reuse our postman script as a base for the performance test. A plugin called postman-to-k6 will help us directly convert our postman collection to a k6 script. We will also use a k6 additional html reporter, since, by default, k6 only provides a CLI, json, or csv file for the output. Here we put a list of libraries and tools that we would use:

• K6
• Postman
• Postman-to-k6 
• K6 html reporter
• Bitbucket, or any other CI/CD tools

This is our simple step-by-step process as follows:

1. Have our postman collection file ready to use
2. Use the postman-to-k6 library to convert our postman collection to k6 script
3. Run using our configuration then produce a report in html, we can use the default reporter (CLI), but for the sake of simplicity, we will use the k6 html reporter mentioned before.

These three steps will be put in our pipeline to make sure bitbucket will be able to run it either when triggered or scheduled. If there are any changes on the API, we can directly focus the changes to our postman collection within the repository, and the k6 script will be updated automatically.

How often do we need to run our performance test? Or what metric should we focus on? We are not going to go into the details in this blog. However, we can refer to the metrics that k6 have provided here, so we can decide your own metrics.

Prepare postman collection via Postman GUI

We can create a simple postman collection, which contains simple API calls. We can also add a simple assertion to each API request. Keep in mind that, not all code we put in Postman script is able to be converted properly, there is some limitation to the library that we used, we can refer to the table below:

Features/Scripts which can be converted from Postman to K6, as follows:

FEATURE	POSTMAN API	K6 API
Importing external libraries	Selected libraries	Selected libraries plus bundled ones(non-browser, non-Node.js APIs)
Making requests	✅	✅
Processing response	✅	✅
Parametrization	✅	✅
REST	✅	✅
GraphQL	✅	✅
Cookies	✅	✅
Proxy	✅	✅
SSL	✅	✅
OpenAPI/Swagger	✅(import directly)	✅(via k6 generator in openapi-generator)
Checks	✅(assertions)	✅(Check API)
Groups	✅(Collections)	✅(Group API)
HTML parsing	✅(needs library)	✅(internal HTML API)
File upload	✅	✅
Test Lifecycle	✅(only with scripts)	✅(internal)

Src : https://k6.io/blog/load-testing-with-postman-collections/

Because we are going to continuously convert postman scripts to k6, it’s important to follow the updated information of this library to avoid any error during conversion.

Sample script

In this article, we will use a simple collection that is already provided by k6 itself for the sake of simplicity (feel free to use your own collection), which is available at https://test-api.k6.io/. We could also use this collection made by k6 https://github.com/grafana/k6-example-postman-collection.

We will only use these 4 requests that are separated by 2 folders as shown below:

We can see that the 2 APIs under the Public APIs folder are treated as a single API, so there is no data correlation between those 2 requests. But for the request in the E2E – Create crocodile folder, we will use an account already created then perform data creation, update, and lastly, delete the data itself.

We also need to add some script to properly check the responses from all APIs, so that our performance test not only returns results for performance tests, but also returns the responses from the APIs as expected and does not return any false positives. Here we do a simple checking HTTP response code, response time, and a simple assertion.

Response time can be also included in the K6 result. Don’t forget to run it first in postman runner to make sure that our script is working fine.

Now we can proceed to our next process.

Create a pipeline on CI/CD

After the postman collection is ready and tested, we can proceed to use it directly on our bitbucket pipeline. We can use images available from bitbucket, for this we will use a `node` image. Steps that must be added to the pipeline are:

1. Convert postman collection to k6
2. Run k6 script from the previous result & generate a html report

Step 1 Convert postman collection to k6

Add step 1 convert-postman-to-k6 under the pipeline yml ‘definitions’. Don’t forget to add `name` to the step to make it easier to debug the pipeline when we run it later. For full documentation regarding the bitbucket pipeline, please see: (https://support.atlassian.com/bitbucket-cloud/docs/configure-bitbucket-pipelinesyml/)

definitions:

   steps:

       – step: &convert-postman-to-k6

           name: Convert postman collection to K6 script

           image: node

Since this image does not directly have postman-to-k6 installed, we need to add the `npm-install` command under the ‘script’, as follows.

definitions:

     # Exporting postman to K6 script

       – step: &convert-postman-to-k6

           name: Convert postman collection to K6 script

           caches:

             – npm

           image: node

           script:

             – node -v

             – npm install -D @apideck/postman-to-k6

             – postman-to-k6 –version

After installing the next step will be to add the `postman-to-k6` to convert our postman collection.

definitions:

     # Exporting postman to K6 script

       – step: &convert-postman-to-k6

           name: Convert postman collection to K6 script

           caches:

             – npm

           image: node

           script:

             – node -v

             – npm install -D @apideck/postman-to-k6

             – postman-to-k6 –version

             – postman-to-k6 ./tests/K6-test.postman_collection.json -e ./tests/k6-exercise.postman_environment.json -o ./result/k6-script.js

Because the result is a new file generated by the library, we are going to store it using `artifacts` to be able to use it later during the next step. Using artifacts also allows us to download the result for this step, in this case, the `k6-script.js`.

definitions:

     # Exporting postman to K6 script

       – step: &convert-postman-to-k6

           name: Convert postman collection to K6 script

           caches:

             – npm

           image: node

           script:

             – node -v

             – npm install -D @apideck/postman-to-k6

             – postman-to-k6 –version

             – postman-to-k6 ./tests/K6-test.postman_collection.json -e ./tests/k6-exercise.postman_environment.json -o ./result/k6-script.js

           artifacts:

             – result/**

Step 2 Run performance test in bitbucket

The next step would be adding the script to run the converted k6. Here is the complete script.

     # Run Performance test (using K6)

       – step: &perf-test-k6

           name: Run Performance Test

           image: loadimpact/k6:latest

           script:

             – k6 version

             # add import function to converted file

             – sed -i ‘2iimport { htmlReport } from “https://raw.githubusercontent.com/benc-uk/k6-reporter/main/dist/bundle.js”\’ ./result/k6-script.js

             # add line of code for generate report in script

             – echo “$(cat ./config/report.js)” >> ./result/k6-script.js

             – k6 run ./result/k6-script.js

           artifacts:

             – result/**

As shown above, there are additional steps to apply an edit to the `k6-script.js` to be able to produce an html report. The first step is to include a library from GitHub – benc-uk/k6-reporter: Output K6 test run results as formatted & easy to read HTML reports. The added line should be: 

import { htmlReport } from “https://raw.githubusercontent.com/benc-uk/k6-reporter/main/dist/bundle.js”

We use `sed` functionality in ubuntu to add text in specific lines. This will add an import function on the second line of the file.

Then we need to call the library to generate the report inside the script, as mentioned in the documentation here. We separate the script into different files `report.js` and add it to our last line of script. In this we can just use `echo` to merge the content of `report.js` to `k6-script.js`. The script will be like this:

echo “$(cat ./config/report.js)” >> ./result/k6-script.js

Lastly don’t forget to extract the report to artifacts so we can download the result. 

pipelines:

 custom:

   run-performance-test:

     – step:

         <<: *convert-postman-to-k6

     – step:

         <<: *perf-test-k6

 branches:

   develop:

     – step:

         <<: *api-test

     – step:

         <<: *convert-postman-to-k6

     – step:

         <<: *perf-test-k6

For the full script, I added a step for API testing, as follows:

definitions:

   caches:

       npm: $HOME/.npm

   steps:

     # Run newman test (API Testing)

       – step: &api-test

           name: Run Newman test

           caches:

             – npm

           image: postman/newman

           script: #run newman with htmlextra reporter

             – newman –version

             – npm install -g newman-reporter-htmlextra

             – newman run ./tests/K6-test.postman_collection.json -e ./tests/k6-exercise.postman_environment.json –reporters cli,htmlextra –reporter-htmlextra-export ./result/testReport.html

           artifacts: # defining artifact to be used in the next process

             – result/**

     # Exporting postman to K6 script

       – step: &convert-postman-to-k6

           name: Convert postman collection to K6 script

           caches:

             – npm

           image: node

           script:

             – node -v

             – npm install -D @apideck/postman-to-k6

             – postman-to-k6 –version

             – postman-to-k6 ./tests/K6-test.postman_collection.json -e ./tests/k6-exercise.postman_environment.json -o ./result/k6-script.js

           artifacts:

             – result/**

     # Run Performance test (using K6)

       – step: &perf-test-k6

           name: Run Performance Test

           image: loadimpact/k6:latest

           script:

             – k6 version

             # add import function to converted file

             – sed -i ‘2iimport { htmlReport } from “https://raw.githubusercontent.com/benc-uk/k6-reporter/main/dist/bundle.js”\’ ./result/k6-script.js

             # add line of code for generate report in script

             – echo “$(cat ./config/report.js)” >> ./result/k6-script.js

             – k6 run ./result/k6-script.js

           artifacts:

             – result/**

pipelines:

 custom:

   run-performance-test:

     – step:

         <<: *convert-postman-to-k6

     – step:

         <<: *perf-test-k6

 branches:

   develop:

     – step:

         <<: *api-test

     – step:

         <<: *convert-postman-to-k6

     – step:

         <<: *perf-test-k6

We can set auto run so each time code is merged to a ‘develop’ branch, it will run the pipeline. As shown in the example below:

That’s it, we can modify it to our own style and even further improve it.

What benefit can we achieve using this flow? We can just focus on improving the script by directly modifying the postman collection using the postman GUI. After we are done modifying the script, we can create pull requests based on previous changes and let the team review it together.

Pros and Cons

Some of the pros of implementing the process shown are:

• Simple implementation 
• Easy to maintain since we only focus on the postman collection
• Can be run by schedule or any time manually using CI/CD pipeline

Some of the cons or things that can be improved include:

• Limitation on type of postman features/scripts that can be converted, refer to https://k6.io/blog/load-testing-with-postman-collections/ 

Future Research

• A report gets generated on bitbucket and needs to be downloaded every time the test is being run. This can be improved by using real-time data tools such as prometheus or influxDB.
• The K6 run used above is not configured by k6 options yet. We can modify the command or add a separate file to configure the run itself (refer to k6-options).
• There is limitation on how big your run configuration is, as we are running the test under bitbucket. This can be improved by having a dedicated or multiple VMs outside the bitbucket for k6 instances.

References

1. Performance Testing Tutorial: What is, Types, Metrics & Example (guru99.com)
2. What Is Shift Left Testing? A Guide to Improving Your QA (testim.io)
3. New Industry Benchmarks for Mobile Page Speed – Think With Google