Redbull namespace on 84.8.102.119: forensic report

k3s node cybermode-v413-wetco-4ocpu-12ram-100gb-oracle-9 · inspected 2026-11-10 16:42 GMT · read-only, nothing was changed on the server

1. Question

On Nov 9 the redbull Helm install failed: pods stayed Pending with "insufficient memory / cpu". On Nov 10 redbull is running on the same VM. What changed?

2. The machine did not change

• Last reboot: Wed Nov 4 14:57. Resizing an OCI VM shape needs a reboot, so nothing was resized between Nov 9 and Nov 10.
• Node capacity: 8 CPU / ~17.1 GiB RAM, no swap. The hostname still says "4ocpu-12ram" because it was resized before Nov 4.
• The same six app namespaces exist: cyber-mode-v4, cyber-mode, ndf, cybermode-v5, raya, redbull. cyber-mode, cyber-mode-v4, ndf and cybermode-v5 still have their full default sizing.

3. Helm history: what was changed and when

requests = CPU / memory reserved · L = memory limit · "svc pods" = user-service, notification, regulator, reverb

redbull

frontend (50m/64Mi) and redis (50m/128Mi) were never changed. Seven upgrades in 37 minutes, stepping requests down until the scheduler accepted the pods.

raya (another client)

raya's back and mongodb pods restarted at about 15:10 as a result.

4. Who and how

SSH key used

The SSH log (journalctl -u sshd) shows the sessions that made the changes logged in with key SHA256:hc4I1eapt0GUsS3vwwFMp4MXmV8Enkr9lKIeuK8n5rY, which is this entry in /home/opc/.ssh/authorized_keys:

256 SHA256:hc4I1eapt0GUsS3vwwFMp4MXmV8Enkr9lKIeuK8n5rY m.saleh@pksaudi.com (ED25519)

The IP changes (probably a home or mobile connection), but the key stays the same. Other logins in the window: key ahmedelshenawy@pksaudi.com once at Nov 10 13:00 for 9 seconds; key adnan@gmail.com from our own VM 159.69.185.31.

Files and commands

• /home/opc/redbull/profile-testing.yaml created 15:04 · /home/opc/raya/profile-testing.yaml 15:10
• /home/opc/redbull/custom-values.yaml 15:15: only domain, URL and email edits compared with the raya-based backup
• mongodump.zip + mongo-restore.sh copied into ~/redbull as root at 15:18, then bash mongo-restore.sh ran (seeded the redbull DB)

/home/opc/.bash_history shows the loop:

vim profile-testing.yaml
helm upgrade --install redbull ./cybermode -n redbull -f custom-values.yaml -f profile-testing.yaml
  (repeated several times, then the same for raya)

The override file's own header says:

TESTING resource profile (VMs / POCs / Demos) ... ~20 concurrent users ...
NOT for production ... a heavy Excel/PDF export can ... OOM-restart the pod.
Acceptable for functional testing; do NOT use this profile under real load.

In the redbull copy, requests were edited even lower (10m / 20Mi) than the totals the file states.

5. The math: why Nov 9 failed and Nov 10 fits

Node allocatable: 8000m CPU, ~17,517Mi memory

Trimming raya alone would not have been enough. Cutting redbull did most of the work (−1,100m CPU, −2,616Mi memory).

Reserved vs real usage (kubectl top, 16:42)

Whole node: 10,896Mi used of ~17.1GiB (62%), ~5GiB available, CPU 4%. The memory limits of all pods add up to 46,250Mi = 264% of RAM.

6. Risk of the resizing done to run redbull

1. The scheduler now has a false picture. It believes the six redbull app pods need 392Mi in total, but they use ~700Mi idle. Kubernetes uses requests to decide what fits, so the node looks 99% full on paper while the real margin depends on luck.
2. Nothing else can be scheduled. Any new pod, any Helm upgrade that raises requests, or any extra pod (a job, a debug pod, a second replica) will go Pending. Changing raya or redbull back to normal sizing will also fail.
3. Redbull is first in line to be killed. When memory runs low, the kubelet evicts pods whose usage is furthest above their request. Redbull pods use 5–8× their request, so they are evicted first. When memory runs out faster than eviction, the Linux OOM killer can hit any of the 66 pods, including other clients.
4. Limits are overcommitted 264%, with no swap. Every pod is allowed to grow past its reservation. If several tenants grow at once (exports, imports, backups), the sum can pass 17GiB and the kernel starts killing processes across all six namespaces.
5. Redbull back is capped at 512Mi with 4 PHP workers. PHP's memory_limit is 1024M per worker. One heavy Excel/PDF export, or four busy workers at ~120Mi each, can exceed 512Mi and OOM-restart the pod. The other 4.1.3 tenants have 1536Mi.
6. Redbull MongoDB reserves 100Mi (limit 1Gi, WiredTiger cache ~256Mi). It's fine for a demo, but slow and OOM-prone with real data.
7. Another client's environment was changed. raya's back CPU and mongodb memory/limits were lowered and its pods restarted. raya is now smaller than the other 4.1.3 tenants, and the server shows no separate record of approval.
8. It is a testing profile. The file says it is not for production. If redbull is a real client environment, it is running on testing-grade sizing.

7. Why live notifications on redbull may not work

VUE_APP_NOTIFICATION_REVERB_HOST: "cyber-rebull.advancedcontrols.tech"   <- missing "d"
expected:                         "cyber-redbull.advancedcontrols.tech"

How live notifications work: the browser opens a WebSocket wss://<REVERB_HOST>:443/app/<key> to the reverb pod and waits for pushed events. With the wrong host, the browser connects to cyber-rebull..., and one of these happens:

1. No DNS record exists for that name, so the connection fails immediately.
2. A wildcard DNS record sends it to a server, but the TLS certificate is for cyber-redbull..., so the browser rejects the connection (certificate name mismatch).
3. Even with a valid TLS connection, the ingress has no rule for cyber-rebull..., so it returns 404 from the default backend instead of reaching reverb.

The rest of the app works because API calls use the correct cyber-redbull... URLs. Users see pages and data, but bell notifications and real-time updates don't arrive until the page is refreshed. The browser console shows repeated WebSocket connection errors.

The backend side (REVERB_HOST, REVERB_ALLOWED_ORIGINS, notification → reverb) uses the correct name, so the typo in the frontend value is the only suspect found.

Not verified live: the DNS, certificate and ingress checks for the misspelled host were not run, and the frontend's served config was not inspected. The cause above comes from the deployed Helm values.

8. How many users each deployment can handle

How to read it: each PHP service handles one request per worker at a time (phpFpm.maxChildren). Every click in the app sends requests to back and to user-service, which checks auth for every request. With 4 back workers, 4 user-service workers and ~0.3 s per request, one tenant serves ~10–13 requests per second. An active user sends a few requests every few seconds, so that comes to roughly 20–30 users working at the same time before requests start queueing. Logged-in but idle users cost almost nothing.

Across the whole VM, the binding limit is RAM, not CPU (CPU is at 4%). There is ~5GiB of real headroom. A busy tenant grows by roughly 0.5–1.5GiB (PHP workers up to their limits, MongoDB cache warming). So the VM can absorb about 2–4 tenants being busy at the same time, not all six. Past that, the kernel starts killing pods.

9. If every deployment gets one user at the same time

10. What to tell your manager

11. How to reproduce these findings

ssh -i ~/.ssh/id_ed25519_mf opc@84.8.102.119
sudo /usr/local/bin/helm history redbull -n redbull
sudo /usr/local/bin/helm get values redbull -n redbull --revision 1   # vs --revision 8
sudo /usr/local/bin/helm history raya -n raya
sudo /usr/local/bin/k3s kubectl describe node | sed -n '/Allocated/,/Events/p'
sudo /usr/local/bin/k3s kubectl top pod -n redbull
last -F | head ; last -x reboot | head -2
sudo journalctl -u sshd --since 2026-11-09 | grep Accepted
ssh-keygen -lf /home/opc/.ssh/authorized_keys
cat ~/redbull/profile-testing.yaml ; tail -90 ~/.bash_history

Not checked: the Nov 9 scheduler events (Kubernetes keeps events for about an hour, so they are gone); load testing; the live WebSocket/DNS check for the misspelled host.