Overview
Performance

How fast, will it last

Overview

This section details the process used to conduct the performance tests of the platform.

The tests aimed to find the following answers:

  • What is the performance limit of the system given the available resources?
  • Does the system maintain stability over a longer period?
  • How does the system handle a temporary spike in traffic?

Following business scenarios were tested:

  • Activation and modification of a convergent mobile + fixed service - 60%/40% blend.
  • Activation of a mobile service.

Traffic patterns:

  • Short term heavy load profile
  • Long term medium load profile

Environment

The tests were conducted in two environments:

Amazon Web Services

AWS EKS Cluster + Atlas MongoDB

On prem

Client's on prem environment: K8s cluster on premise + MongoDB on prem

  • K8s cluster (resource quota: cpu 30, memory 60 Gi)
  • MongoDB (4 x vCPU, 24 GB RAM)

We thank our Client for allowing us to publish the test results conducted

Results

On the mentioned on prem environment, the following results were achived:

  • Up to 210 convergent orders/minute stable load conditions in the 60%/40% blend.
  • Up to 600 convergent orders/minute, peak periods in the blend.
  • Up to 700 mobile orders/minute, peak periods.

The system remained stable, with peak traffic correctly queued and processed shortly after the test concluded

We stopped furter testing as we lost hope in finding the limit ;-)

Meat & Bones

Blended traffic, 1 hour

Within these test scenarios, the system was loaded with orders over the course of 1 hour. 60% of these orders were activation orders for the converged service (mobile + fixed), while 40% were orders modifying the converged service.

1 hour, stable load1 hour, stable load with spikes1 hour, looking for the limit
idTest-MB-2-4-CLITest-MB-4-3-CLITest-MB-6-1-CLI
traffic rate [orders/minute]120200/350350/400/450/500/550/600
traffic modelRamp up: 5 minutesChange every 20 minutesChange every 10 minutes
duration1 hour1 hour1 hour
start2024/05/04 21:29:522024/05/05 06:45:062024/05/05 15:11:52
load test - stop2024/05/04 22:29:502024/05/05 07:45:042024/05/05 16:11:47
load test - completed orders6 95414 98718 239
load test - duratuon [minutes]59.9759.9759.92
load test - [orders / minute]115.96249.92304.41
load test - [orders / second]1.934.175.07
ordering finished - stop2024/05/04 22:29:542024/05/05 07:45:072024/05/05 16:34:27
ordering finished - completed orders (after test stop)61310 215
ordering finished - completed orders (total)6 96015 00028 454
ordering finished - duration [minutes]60.0360.0282.58
ordering finished - [orders / minute]115.94249.93344.55
ordering finished - [orders / second]1.934.175.74
errors count000
errors [%]0.0%0.0%0.0%
stuck orders descriptionn/an/afulfilling - 46
stuck orders count0046
stuck orders [%]0.0%0.0%0.2%
orders reported by JMeter6 96015 00028 500
orders in SOM dashboard6 96015 00028 500
rejected reported by JMeter [%]0.0%0.0%0.0%
rejected (calculated) count000
rejected (calculated) [%]0.0%0.0%0.0%
environmentOn premiseOn premiseOn premise

Mobile orders only, 1 hour

Within this test scenario, the system was loaded with orders during 1 hour. 100% of the orders were activation orders for mobile service.

1 hour, looking for the limit
idTest-MB-5-3-CLI
traffic rate [orders/minute]450/500/550/600/650/700
traffic modelChange every 10 minutes
duration1 hour
start2024/05/05 13:33:11
load test - stop2024/05/05 14:33:06
load test - completed orders32 087
load test - duratuon [minutes]59.92
load test - [orders / minute]535.53
load test - [orders / second]8.93
ordering finished - stop2024/05/05 14:35:42
ordering finished - completed orders (after test stop)2 413
ordering finished - completed orders (total)34 500
ordering finished - duration [minutes]62.52
ordering finished - [orders / minute]551.85
ordering finished - [orders / second]9.20
errors count0
errors [%]0.0%
stuck orders descriptionn/a
stuck orders count0
stuck orders [%]0.0%
orders reported by JMeter34 500
orders in SOM dashboard34 500
rejected reported by JMeter [%]0.0%
rejected (calculated) count0
rejected (calculated) [%]0.0%
environmentOn premise

Blended traffic, 6/12 hours

During these test scenarios, the system was loaded with orders over the course of 1 hour, while 60% of these orders were activation orders for the converged service (mobile + fixed line) and 40% ot them modifying the converged service.

6 hour, stable load with spikes12 hour, stable load
idTest-MB-7-1-CLITest-MB-8-1-CLI-pv-12h
traffic rate [orders/minute]200/350210
traffic modelChange every 20 minutesRamp up: 5 minutes
duration6 hours12 hours
start2024/05/05 18:19:072024/05/06 16:38:24
load test - stop2024/05/06 00:19:042024/05/07 04:38:23
load test - completed orders98 674150 758
load test - duratuon [minutes]359.95719.98
load test - [orders / minute]274.13209.39
load test - [orders / second]4.573.49
ordering finished - stop2024/05/06 00:19:402024/05/07 04:38:27
ordering finished - completed orders (after test stop)31714
ordering finished - completed orders (total)98 991150 772
ordering finished - duration [minutes]360.55720.05
ordering finished - [orders / minute]274.56209.39
ordering finished - [orders / second]4.583.49
errors count00
errors [%]0.0%0.0%
stuck orders descriptionplan_building - 1fulfilling - 1
stuck orders count11
stuck orders [%]0.0%0.0%
orders reported by JMeter99 000150 780
orders in SOM dashboard98 992150 773
rejected reported by JMeter [%]0.0%0.0%
rejected (calculated) count87
rejected (calculated) [%]0.0%0.0%
environmentOn premiseOn premise