cgroups From First Principles: Why Your Container Has No Idea How Big It Is

A container is not a small computer. It is a group of processes that the kernel agrees to account for and limit together. Everything else — the image, the registry, the runtime — is packaging around that idea.

The accounting mechanism is control groups, or cgroups. Understanding them explains a whole class of "my container died for no reason" bugs.

What cgroups actually do

Two jobs: limit and account.

• Limit: this group gets at most N bytes of memory, M microseconds of CPU per period.
• Account: charge every byte and microsecond used by any process in the group.

That is it. There is no hypervisor, no virtual hardware. The kernel simply refuses to hand a group more than its quota, and records what it used.

# cgroups v2: the memory ceiling for this group
cat /sys/fs/cgroup/memory.max

# What the group is using right now
cat /sys/fs/cgroup/memory.current

# CPU quota per period (max) and period length (period)
cat /sys/fs/cgroup/cpu.max

The classic surprise: the JVM that ignores your limit

Older runtimes sized their heaps from /proc/meminfo, which reports the host's memory — cgroups do not rewrite it. A JVM told it may use 512 MB would happily size a heap for a 64 GB host and then get OOM-killed by the cgroup.

The kernel's enforcement is real; the application's perception is not. Modern runtimes are cgroup-aware, but plenty of software still is not, and the failure mode is always the same: a process killed from outside with no stack trace, because the killer was the kernel, not the program.

An OOM kill from a cgroup looks like a crash with no error. Check memory.events before you blame your code.

# How many times this group was OOM-killed
cat /sys/fs/cgroup/memory.events

CPU throttling is quieter and worse

Memory limits kill loudly. CPU limits throttle silently: the group gets its quota per period and then simply stops being scheduled until the next period. The symptom is latency spikes at exactly the period boundary, with CPU "usage" graphs that look fine.

If a service is p99-slow but never CPU-saturated on the dashboard, look at nr_throttled and throttled_usec in cpu.stat. That is your answer.

Why this matters for your YAML

When you write limits.memory: 512Mi in Kubernetes, you are writing a cgroup file, not a suggestion. The distance between what your app thinks it has and what the cgroup allows is where the incident lives.

Containers are kernel features with a good marketing team. Know the kernel feature.