How fast, will it last
Overview
This section details the process used to conduct the performance tests of the platform.
The tests aimed to find the following answers:
- What is the performance limit of the system given the available resources?
- Does the system maintain stability over a longer period?
- How does the system handle a temporary spike in traffic?
Following business scenarios were tested:
- Activation and modification of a convergent mobile + fixed service - 60%/40% blend.
- Activation of a mobile service.
Traffic patterns:
- Short term heavy load profile
- Long term medium load profile
Environment
The tests were conducted in two environments:
Amazon Web Services
AWS EKS Cluster + Atlas MongoDB
- EKS (opens in a new tab) K8s cluster (resource quota: cpu 34, memory 65 Gi)
- 5 x t3.xlarge (opens in a new tab) nodes
- 1 x Atlas MongoDB M30 (2 x vCPU, 8 GB RAM)
On prem
Client's on prem environment: K8s cluster on premise + MongoDB on prem
- K8s cluster (resource quota: cpu 30, memory 60 Gi)
- MongoDB (4 x vCPU, 24 GB RAM)
We thank our Client for allowing us to publish the test results conducted
Results
On the mentioned on prem environment, the following results were achived:
- Up to 210 convergent orders/minute stable load conditions in the 60%/40% blend.
- Up to 600 convergent orders/minute, peak periods in the blend.
- Up to 700 mobile orders/minute, peak periods.
The system remained stable, with peak traffic correctly queued and processed shortly after the test concluded
We stopped furter testing as we lost hope in finding the limit ;-)
Meat & Bones
Blended traffic, 1 hour
Within these test scenarios, the system was loaded with orders over the course of 1 hour. 60% of these orders were activation orders for the converged service (mobile + fixed), while 40% were orders modifying the converged service.
| 1 hour, stable load | 1 hour, stable load with spikes | 1 hour, looking for the limit | |
|---|---|---|---|
| id | Test-MB-2-4-CLI | Test-MB-4-3-CLI | Test-MB-6-1-CLI |
| traffic rate [orders/minute] | 120 | 200/350 | 350/400/450/500/550/600 |
| traffic model | Ramp up: 5 minutes | Change every 20 minutes | Change every 10 minutes |
| duration | 1 hour | 1 hour | 1 hour |
| start | 2024/05/04 21:29:52 | 2024/05/05 06:45:06 | 2024/05/05 15:11:52 |
| load test - stop | 2024/05/04 22:29:50 | 2024/05/05 07:45:04 | 2024/05/05 16:11:47 |
| load test - completed orders | 6 954 | 14 987 | 18 239 |
| load test - duratuon [minutes] | 59.97 | 59.97 | 59.92 |
| load test - [orders / minute] | 115.96 | 249.92 | 304.41 |
| load test - [orders / second] | 1.93 | 4.17 | 5.07 |
| ordering finished - stop | 2024/05/04 22:29:54 | 2024/05/05 07:45:07 | 2024/05/05 16:34:27 |
| ordering finished - completed orders (after test stop) | 6 | 13 | 10 215 |
| ordering finished - completed orders (total) | 6 960 | 15 000 | 28 454 |
| ordering finished - duration [minutes] | 60.03 | 60.02 | 82.58 |
| ordering finished - [orders / minute] | 115.94 | 249.93 | 344.55 |
| ordering finished - [orders / second] | 1.93 | 4.17 | 5.74 |
| errors count | 0 | 0 | 0 |
| errors [%] | 0.0% | 0.0% | 0.0% |
| stuck orders description | n/a | n/a | fulfilling - 46 |
| stuck orders count | 0 | 0 | 46 |
| stuck orders [%] | 0.0% | 0.0% | 0.2% |
| orders reported by JMeter | 6 960 | 15 000 | 28 500 |
| orders in SOM dashboard | 6 960 | 15 000 | 28 500 |
| rejected reported by JMeter [%] | 0.0% | 0.0% | 0.0% |
| rejected (calculated) count | 0 | 0 | 0 |
| rejected (calculated) [%] | 0.0% | 0.0% | 0.0% |
| environment | On premise | On premise | On premise |
Mobile orders only, 1 hour
Within this test scenario, the system was loaded with orders during 1 hour. 100% of the orders were activation orders for mobile service.
| 1 hour, looking for the limit | |
|---|---|
| id | Test-MB-5-3-CLI |
| traffic rate [orders/minute] | 450/500/550/600/650/700 |
| traffic model | Change every 10 minutes |
| duration | 1 hour |
| start | 2024/05/05 13:33:11 |
| load test - stop | 2024/05/05 14:33:06 |
| load test - completed orders | 32 087 |
| load test - duratuon [minutes] | 59.92 |
| load test - [orders / minute] | 535.53 |
| load test - [orders / second] | 8.93 |
| ordering finished - stop | 2024/05/05 14:35:42 |
| ordering finished - completed orders (after test stop) | 2 413 |
| ordering finished - completed orders (total) | 34 500 |
| ordering finished - duration [minutes] | 62.52 |
| ordering finished - [orders / minute] | 551.85 |
| ordering finished - [orders / second] | 9.20 |
| errors count | 0 |
| errors [%] | 0.0% |
| stuck orders description | n/a |
| stuck orders count | 0 |
| stuck orders [%] | 0.0% |
| orders reported by JMeter | 34 500 |
| orders in SOM dashboard | 34 500 |
| rejected reported by JMeter [%] | 0.0% |
| rejected (calculated) count | 0 |
| rejected (calculated) [%] | 0.0% |
| environment | On premise |
Blended traffic, 6/12 hours
During these test scenarios, the system was loaded with orders over the course of 1 hour, while 60% of these orders were activation orders for the converged service (mobile + fixed line) and 40% ot them modifying the converged service.
| 6 hour, stable load with spikes | 12 hour, stable load | |
|---|---|---|
| id | Test-MB-7-1-CLI | Test-MB-8-1-CLI-pv-12h |
| traffic rate [orders/minute] | 200/350 | 210 |
| traffic model | Change every 20 minutes | Ramp up: 5 minutes |
| duration | 6 hours | 12 hours |
| start | 2024/05/05 18:19:07 | 2024/05/06 16:38:24 |
| load test - stop | 2024/05/06 00:19:04 | 2024/05/07 04:38:23 |
| load test - completed orders | 98 674 | 150 758 |
| load test - duratuon [minutes] | 359.95 | 719.98 |
| load test - [orders / minute] | 274.13 | 209.39 |
| load test - [orders / second] | 4.57 | 3.49 |
| ordering finished - stop | 2024/05/06 00:19:40 | 2024/05/07 04:38:27 |
| ordering finished - completed orders (after test stop) | 317 | 14 |
| ordering finished - completed orders (total) | 98 991 | 150 772 |
| ordering finished - duration [minutes] | 360.55 | 720.05 |
| ordering finished - [orders / minute] | 274.56 | 209.39 |
| ordering finished - [orders / second] | 4.58 | 3.49 |
| errors count | 0 | 0 |
| errors [%] | 0.0% | 0.0% |
| stuck orders description | plan_building - 1 | fulfilling - 1 |
| stuck orders count | 1 | 1 |
| stuck orders [%] | 0.0% | 0.0% |
| orders reported by JMeter | 99 000 | 150 780 |
| orders in SOM dashboard | 98 992 | 150 773 |
| rejected reported by JMeter [%] | 0.0% | 0.0% |
| rejected (calculated) count | 8 | 7 |
| rejected (calculated) [%] | 0.0% | 0.0% |
| environment | On premise | On premise |