# Autoscaling load test plan — b2g-UAT

Goal: prove the ECS target-tracking policies scale out at 80% CPU and scale back in when load stops.

Everything below was read from the live cluster, not assumed.

> **This test was executed on 2026-08-06 and both directions passed.** Results, measured throughput
> and findings are in [load-test-results.md](load-test-results.md). Two estimates below were wrong
> and are corrected there: the endpoint costs **2.45 ms CPU** per request (not ~35 ms), so it took
> **~100 req/s** to saturate one task rather than the ~6 req/s predicted in §2; and the oscillation
> predicted in §1 did not occur, because `ab` is a closed-loop generator — see F4 in the results.

---

## 1. What is actually configured

**13 services** have a target-tracking policy named `hpa`. `kong` does **not** — confirmed, it is
absent from the scalable targets, which matters (see §3).

| Setting | Value |
|---|---|
| Metric | `ECSServiceAverageCPUUtilization` (CloudWatch `AWS/ECS` → `CPUUtilization`) |
| Target | **80%** |
| Min / Max | **1 / 2** |
| Scale-out cooldown | 30s |
| Scale-in cooldown | 30s (60s on `usermgmt`) |

Scaled services: `micro`, `usermgmt`, `profilemgmt`, `confmgmt`, `digitall`, `cengex`,
`gmarketapi`, `mail`, `filemst`, `fe-b2g`, `fe-gmarket`, `fe-website`, `fe-filems`.

### The timings that actually govern the test

Target tracking created two CloudWatch alarms per service. The **cooldowns are almost irrelevant**
next to the alarm evaluation windows:

| Alarm | Condition | Period × Evaluations | Effective delay |
|---|---|---|---|
| `AlarmHigh` | CPUUtilization **> 80** | 60s × **3** | **~3–4 min** of sustained load before scale-out |
| `AlarmLow` | CPUUtilization **< 72** | 60s × **15** | **~15–17 min** below 72% before scale-in |

Two consequences to set expectations by:

- Scale-out will not happen in 30 seconds. Budget **4–5 minutes** of sustained load.
- Scale-in is **not broken if nothing happens for 15 minutes.** Budget **16–18 minutes** after load
  stops. Do not abort early and conclude it failed.

Baseline CPU right now is **2–4%** on every service, so there is plenty of headroom to see a change.

### Predicted oscillation — worth knowing before you interpret the result

`CPUUtilization` is CPU used ÷ CPU reserved across the service. If you apply exactly enough load to
saturate **one** task, then after scale-out the same load is split across two tasks, each sits near
**40%**, which is below the 72% scale-in threshold — so 15 minutes later it scales back to 1, CPU
returns to 80%, and it scales out again. A roughly 20-minute flap.

To see it **hold** at 2 tasks, apply about **twice** the single-task saturation load. Then drop to
zero for a clean scale-in observation. This is expected behaviour of target tracking with `max 2`,
not a misconfiguration.

---

## 2. Which endpoints to load, and why

### Backend — `cengex` (recommended first test)

- **0.25 vCPU** (256 units), so 80% ≈ **0.2 vCPU** of real compute — a low bar.
- Not on the login or dashboard path, so a mistake here does not block anyone testing.
- Endpoint: **`http://<alb>/cengex/web/loadtest-404`**

Why a 404 route: it boots the whole Yii framework — autoloader, config, log component — and returns
a JSON 404 **without touching the database**. Measured **50 ms** per request from the bastion.

**This matters.** `config/db-local.php` sets `enableSchemaCache => false`, so any endpoint that
touches a model re-reads table schemas from RDS on *every* request. Under load that would hammer a
`db.t4g.medium` shared by all five schemas and could degrade every other service — you would be load
testing RDS, not the container. Avoid `getcountrylist` and similar for this reason.

**Rough sizing:** ~35 ms CPU per request → **≈6 req/s** to reach 80% of 0.25 vCPU. Use **~12 req/s**
to hold two tasks above the scale-in threshold. `php-fpm` is `pm=ondemand` with
`pm.max_children=8`, so concurrency beyond ~8 just queues — **concurrency 8–10 is plenty**. Do not
use concurrency 100; it produces timeouts, not a cleaner signal.

### Frontend — `fe-b2g`

Static nginx, ~0.3 ms CPU per request, so 80% of 0.25 vCPU needs roughly **400–1000 req/s**. Two
complications and one convenient escape:

- **Kong would bottleneck first.** Kong is 0.5 vCPU and **has no scaling policy**. Routing 1000 req/s
  through it would saturate Kong before `fe-b2g` ever reaches 80%, and it would degrade every other
  service at the same time.
- **The escape:** the ALB has exact-match rules that bypass Kong entirely. `http://<alb>/app`
  (no trailing slash) goes **straight to the `fe-b2g` target group** — verified. `/app/` and anything
  deeper falls through to Kong. So load the bare `/app`.

Use `/app` for the frontend test and Kong is out of the path completely.

---

## 3. Risks and abort criteria

| Risk | Mitigation |
|---|---|
| **Kong saturates and degrades everything.** 0.5 vCPU, no autoscaling, single ingress for all traffic except the four exact-match paths. | Keep backend load modest (~12 req/s). Watch Kong's CPU. Abort if it exceeds ~70%. |
| **RDS becomes the bottleneck** — shared `db.t4g.medium`, schema cache disabled. | Use the non-DB 404 route. Do not load DB-backed endpoints. |
| **Other testers see slow responses.** | Run outside working hours if anyone is using UAT, and pick `cengex` rather than `usermgmt`. |
| **Log growth on ephemeral storage.** Each 404 writes to `runtime/logs/app.log`. | Trivial at these rates (a few thousand lines), and the task disk is 20 GB. |
| **Cost.** +1 task per scaled service while scaled out. | Pennies for a 30-minute test. Even all 13 doubling briefly stays inside the Fargate headroom (4.75 → 9.5 vCPU). |

**Abort if:** Kong CPU > 70%, RDS CPU > 70%, `/healthz` through the ALB stops returning 200, or any
service other than the target one starts failing. Stopping the load generator is the entire rollback.

---

## 4. Setup — one install on the bastion

The bastion is the right generator: 2 vCPU, in-VPC, **6 ms** to the ALB. It has only `curl` today.

```bash
# via: aws ssm start-session --target i-04bed8d526c55e47d
sudo apt-get update -qq && sudo apt-get install -y apache2-utils
ab -V | head -2
```

`ab` is adequate for the backend test. For the frontend's higher rate, `hey` handles concurrency
better:

```bash
curl -sL -o /tmp/hey https://hey-release.s3.us-east-2.amazonaws.com/hey_linux_amd64 && chmod +x /tmp/hey
```

---

## 5. Procedure

Set up two terminals: one running load on the bastion, one watching from your workstation.

### Watcher (workstation, leave running)

```powershell
$env:AWS_DEFAULT_REGION = "ap-south-1"
$svc = "cengex"      # change to fe-b2g for the frontend test
while ($true) {
  $t = (Get-Date).ToString("HH:mm:ss")
  $c = aws ecs describe-services --cluster b2g-UAT --services $svc `
        --query "services[0].[runningCount,desiredCount,pendingCount]" --output text
  $cpu = aws cloudwatch get-metric-statistics --namespace AWS/ECS --metric-name CPUUtilization `
        --dimensions Name=ClusterName,Value=b2g-UAT Name=ServiceName,Value=$svc `
        --start-time (Get-Date).AddMinutes(-5).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ") `
        --end-time (Get-Date).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ") `
        --period 60 --statistics Average `
        --query "sort_by(Datapoints,&Timestamp)[-1].Average" --output text
  $kong = aws cloudwatch get-metric-statistics --namespace AWS/ECS --metric-name CPUUtilization `
        --dimensions Name=ClusterName,Value=b2g-UAT Name=ServiceName,Value=kong `
        --start-time (Get-Date).AddMinutes(-5).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ") `
        --end-time (Get-Date).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ") `
        --period 60 --statistics Average `
        --query "sort_by(Datapoints,&Timestamp)[-1].Average" --output text
  "$t  $svc run/desired/pending=$c  cpu=$cpu  kong_cpu=$kong"
  Start-Sleep -Seconds 30
}
```

### Phase 0 — baseline (2 min)

Confirm 1/1 and CPU under 10%. Record the numbers.

### Phase 1 — calibrate (3 min)

Do not guess the rate; measure it. On the bastion:

```bash
ALB=b2g-uat-alb-105245781.ap-south-1.elb.amazonaws.com
ab -t 120 -c 4 -n 100000 "http://$ALB/cengex/web/loadtest-404"
```

Read the CPU the watcher reports. If it settles below 80%, raise `-c` to 8. If it is already pegged
at 100%, drop to `-c 2`. **Target 85–95%** — comfortably over the threshold without pure queueing.

### Phase 2 — sustained load, observe scale-out (8 min)

```bash
ab -t 480 -c 8 -n 1000000 "http://$ALB/cengex/web/loadtest-404"
```

Expect: CPU > 80% within a minute → `AlarmHigh` after **3 consecutive minutes** → `desiredCount`
becomes **2** → new task starts (~30–60 s) → CPU per task roughly halves.

Confirm the scaling activity, which is the authoritative record:

```powershell
aws application-autoscaling describe-scaling-activities --service-namespace ecs `
  --resource-id service/b2g-UAT/cengex `
  --query "ScalingActivities[0:3].{start:StartTime,status:StatusCode,cause:Cause}" --output json
```

### Phase 3 — stop load, observe scale-in (18 min)

Kill `ab`. CPU should fall to baseline within a minute. Then **wait** — `AlarmLow` needs
**15 consecutive minutes** below 72%. Expect `desiredCount` back to **1** at roughly the 16-minute
mark, and one task draining.

### Phase 4 — frontend test (optional, ~30 min)

Same shape against the Kong-bypassing path. Note the **bare `/app`** with no trailing slash:

```bash
/tmp/hey -z 8m -c 100 "http://$ALB/app"
```

Watch `fe-b2g`. If CPU will not reach 80%, the bastion is the limit rather than the service — raise
`-c`, or accept that a static nginx container on 0.25 vCPU needs more load than one 2-vCPU box can
generate, and treat the backend test as the proof.

---

## 6. Success criteria

| # | Criterion |
|---|---|
| 1 | `cengex` CPU sustained **> 80%** under load |
| 2 | `desiredCount` 1 → **2** within ~5 minutes of load starting |
| 3 | The second task reaches **RUNNING** and takes traffic (CPU per task drops) |
| 4 | A scaling activity recorded with `StatusCode: Successful` and a cause naming `AlarmHigh` |
| 5 | After load stops, `desiredCount` 2 → **1** within ~18 minutes, cause naming `AlarmLow` |
| 6 | Throughout: `/healthz` stays 200 and no other service degrades |

If 1–3 pass but 5 does not after 20 minutes, check `AlarmLow` state — the most likely cause is CPU
sitting *between* 72% and 80%, which satisfies neither alarm.

---

## 7. After the test

Nothing to undo — stopping the generator is the rollback, and services return to `min 1` on their
own. Optionally remove `apache2-utils`/`hey` from the bastion.

Two things worth considering once the mechanism is proven:

- **`kong` has no scaling policy** and is the single ingress for everything except four exact paths.
  At 0.5 vCPU it is the first thing that will saturate under real traffic, and it cannot scale.
  It is the most valuable service to add a policy to.
- **`max 2` guarantees the oscillation** described in §1 for any load that saturates roughly one
  task. A higher max, or a target nearer 60–70%, gives target tracking room to settle.