What DigitalOcean's allocatable memory tax taught me about node sizing, eviction order, and asking a cluster for what you actually need.
The pods would not schedule. The dashboard said we had plenty of memory. Both of those things were true at the same time, and reconciling them cost me an afternoon I would like back.
Pending. Several of them, and no obvious reason. kubectl get nodes showed a cluster that looked half empty. The memory graph in Grafana was calm. But the scheduler had run out of room, because the scheduler was not looking at the number I was looking at.
The cluster was never too small. It was being honest about what we had actually asked it for — and we had never learned to read the answer.
What follows is the thing I wish I’d understood before that afternoon: what a node actually gives you, what the two numbers on a pod mean, who gets killed when a node runs out of memory, and how to find the right numbers instead of guessing them. It is written against DigitalOcean’s managed Kubernetes (DOKS), because that is where I learned it, but only the node-sizing arithmetic is DigitalOcean-specific. The rest is just Kubernetes.
Every container can declare two numbers for each resource, and almost every resourcing mistake starts with conflating them.
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
memory: 512MiA request is a scheduling promise. It is the only number the scheduler reads. When you say memory: 512Mi, you are telling Kubernetes to find a node with 512 MiB of unclaimed room and reserve it for this pod, whether or not the pod ever touches it. A request is a reservation, not a prediction.
A limit is an enforcement ceiling. The scheduler never looks at it. It is handed to the kubelet and the Linux cgroup, and it governs what happens while the container runs, not where it lands.
The two resources behave differently at their ceilings, and this is the asymmetry that surprises everyone:
That asymmetry drives two recommendations I’ll stand behind.
First, omit CPU limits for most workloads. The CFS quota mechanism that enforces a CPU limit throttles in fixed time slices, and it can throttle a container that is nowhere near its average limit simply because it burst within a 100ms window. Your request already guarantees a CPU floor the scheduler honors; a limit on top of it mostly buys you mysterious p99 latency. The exception is hard multi-tenancy or cost chargeback, where a predictable ceiling is worth the throttling.
Second, set the memory request equal to the memory limit for anything you care about. Because memory isn’t compressible, the gap between request and limit is a gap you’re gambling on — room you were scheduled without a guarantee you can keep. Closing that gap costs you nothing but honesty about what you need, and it buys a QoS class I’ll get to.
One unit note, because it trips up everyone new to this: 200m is 200 millicores, i.e. 0.2 of a CPU. And Mi is mebibytes (1024-based) while M is megabytes (1000-based) — 512Mi is not 512M. Kubernetes memory is almost always written in Mi/Gi.
Here is the number I didn’t know existed that afternoon. Run:
kubectl describe node <node>and you’ll see two memory figures: Capacity and Allocatable. Capacity is what you bought. Allocatable is what your pods can actually request — capacity minus everything the platform reserves for the kubelet, container runtime, networking, DNS, and the OS.
On DOKS that reservation is roughly fixed per node, which means it lands as a regressive tax. The published allocatable figures make it stark:
| Node memory | Max pod-allocatable | Lost to overhead |
|---|---|---|
| 2 GiB | 1 GiB | ~50% |
| 3 GiB | 1.66 GiB | ~45% |
| 4 GiB | 2.5 GiB | ~37% |
| 8 GiB | 6 GiB | ~25% |
| 16 GiB | 13 GiB | ~19% |
| 32 GiB | 28 GiB | ~12.5% |
| 64 GiB | 58 GiB | ~9% |
| 192 GiB | 182 GiB | ~5% |
A 2 GiB node gives your workloads one gigabyte. Half of what you’re paying for is gone before you deploy anything. That is why DigitalOcean itself recommends against nodes under 2 GiB of allocatable memory for production.
The practical consequence is that fleets of small nodes are a bad deal, and the arithmetic is not close:
- 3 × s-2vcpu-4gb = 12 GiB bought, 7.5 GiB usable, 3 nodes of overhead
+ 2 × s-4vcpu-8gb = 16 GiB bought, 12 GiB usable, 2 nodes of overheadFewer nodes, more usable memory, and a smaller ongoing bill for the DaemonSets — the log shipper, CSI driver, and CNI agent that run one copy on every node whether it’s busy or not. Every node you add is another full set of those.
But don’t over-correct into a single enormous node, because a few things push back the other way:
The heuristic I settled on: size a pool so that losing one node is absorbable — you want N+1 headroom measured in allocatable terms, not the capacity number on the invoice.
Node pools are worth splitting when workloads have genuinely different shapes — a memory-heavy pool and a CPU-heavy pool, a tainted pool for special hardware, or an isolated pool for a bursty batch job you don’t want stealing from request-serving pods. What isn’t worth it is a pool per team: it fragments capacity and makes every team pay the full per-node overhead separately.
QoS is the part everyone thinks they configure and nobody actually does. You don’t set a QoS class; you earn one, entirely as a side effect of how you wrote your requests and limits.
| QoS class | How you earn it | Evicted |
|---|---|---|
Guaranteed | requests == limits for every container, CPU and memory | last |
Burstable | at least one request set, but not equal to limits | middle |
BestEffort | nothing set at all | first |
When a node comes under memory pressure, the kubelet evicts pods to save the node, and it walks this order: BestEffort first, then Burstable pods that are furthest over their own request, and Guaranteed last. The class you never chose decides who survives.
This is worth internalizing before an incident teaches it to you: the pod that gets evicted is not the least important one, it’s the one whose usage most overshoots the promise it made. A humble service that requested honestly can outlive a critical one that lowballed its request.
Now the distinction people conflate constantly:
Eviction is the kubelet reacting to node pressure. It’s node-level, relatively graceful, and respects QoS ordering — pods get a termination signal and a chance to shut down.
OOMKilled is the kernel enforcing a single container’s cgroup memory limit. It’s instant, ungraceful, exit code 137, and has nothing to do with the node being full — the node can have memory to spare while one container dies for crossing its own line.
They look similar in a crash loop and have completely different fixes. Eviction says the node is oversubscribed. OOMKilled says this one container’s limit is wrong, or it’s leaking.
The OOMKilled trap that catches new teams hardest is a runtime that doesn’t know it’s in a container. A JVM or Node process that sizes its heap against the node’s total memory rather than the cgroup limit will happily grow past its container limit and get shot — the app thinks it has 8 GiB because the node has 8 GiB, while its container limit was 1 GiB.
Which loops back to the earlier advice: setting memory request == limit earns you Guaranteed and takes you out of the eviction lottery entirely. (Note the honest tension with omitting CPU limits — a pod with no CPU limit lands as Burstable, not Guaranteed. If you want true Guaranteed, that’s the one case where you accept the CPU limit and its throttling. For most services, Burstable with an honest memory request == limit is the right trade: you keep CPU burst headroom and still sit near the safe end of the eviction order.)
Everything above is per-pod. ResourceQuota is the namespace-level ceiling that stops one team’s mistake from consuming the whole cluster.
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-quota
namespace: payments
spec:
hard:
requests.memory: 20Gi
requests.cpu: "10"
limits.memory: 40GiThere’s a trap here that everyone hits within a day of applying their first quota:
The moment a ResourceQuota constrains a resource in a namespace, every pod in that namespace must specify a request and limit for that resource, or it is rejected outright. Teams apply a quota, unrelated deployments suddenly fail to admit, and the quota gets blamed for something that is actually just its enforcement working.
The fix is a LimitRange, which backfills the numbers nobody specified:
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
namespace: payments
spec:
limits:
- type: Container
default: # becomes the limit if unset
memory: 512Mi
defaultRequest: # becomes the request if unset
memory: 256Mi
max:
memory: 4GiPair them: the ResourceQuota sets the namespace ceiling, the LimitRange gives every unannotated pod sane defaults so the quota doesn’t reject it. But be honest about what this buys you — quotas are a blast-radius tool, not a right-sizing tool. They stop a namespace from eating the cluster. They tell nobody what the right number for any given pod actually is. That’s the last section.
Everything so far is mechanism. The question that actually matters — what do I set the request to? — is answered by measurement, not intuition. Since Prometheus and Grafana were what we ran, here are the queries I reached for.
Actual memory usage, using working_set (the number the kubelet actually evicts on — not RSS, not cache):
max_over_time(
container_memory_working_set_bytes{namespace="$ns", container!="", container!="POD"}[7d]
)The waste-finder — requested memory divided by memory actually used, per namespace. A ratio far above 1 is money you’re reserving and not using:
sum by (namespace) (kube_pod_container_resource_requests{resource="memory"})
/
sum by (namespace) (container_memory_working_set_bytes{container!=""})CPU throttling ratio — the evidence for the no-CPU-limits argument from earlier. If this is meaningfully above zero, a CPU limit is silently slowing you down:
rate(container_cpu_cfs_throttled_periods_total[5m])
/ rate(container_cpu_cfs_periods_total[5m])At the cluster level, the three numbers that belong on one Grafana row are allocatable, requested, and actually used. The gap between allocatable and requested is capacity you can’t schedule against; the gap between requested and used is the money you’re leaving on the table.
The method those queries feed into:
max_over_time / p99 over at least 7 days, plus headroom. Memory isn’t compressible; you do not run it close to the edge.If you’d rather not eyeball graphs, the Vertical Pod Autoscaler in recommendation-only mode (updateMode: "Off") will compute suggested requests from the same underlying data and hand you a number to sanity-check against your own.
Go back to the Pending pods from the opening. With Allocatable, QoS, and the eviction order in hand, they stop being a mystery: the scheduler was reading a number I hadn’t learned to read, on nodes small enough that the platform’s fixed overhead was eating a third of everything I’d bought.
The cluster was never lying to me. It answered the question I asked — reserve this much — with total precision. I just hadn’t noticed I was asking for room I didn’t understand, on nodes that had less to give than the invoice implied.
That, in the end, is all resource planning is. Not capacity math, not a spreadsheet of instance types — just being specific about what each thing needs, and then checking whether you were right. Most of the outages I’ve watched since were some version of the same thing: a vague number, quietly waiting to be found out.