Six layers to sandbox untrusted Python — and the escape I missed

Third in a series on building out-of-band application security testing (OAST) infrastructure from scratch. Post 1 hand-rolled a JNDI Reference; post 2 built the authoritative DNS server. This one is the riskiest feature in the whole platform: letting users run their own Python, in production, to shape the responses the listeners send back.

The feature is simple to state and terrifying to implement: a user writes a Python function, and when a callback lands on their host, the server runs that function to decide what to answer. DNS rebinding, conditional payloads by source IP, crafting a JNDI reference on the fly — all of it wants user-supplied logic in the response path. Which means executing untrusted Python on your production box.

“Just use eval with a restricted __builtins__” is how people get owned. CPython’s object graph is a hall of mirrors: from almost any object you can walk .__class__.__bases__ up to object, back down .__subclasses__() to something dangerous, and out to os.system. A serious sandbox assumes every single layer will be bypassed and stacks independent ones so that an escape from one lands in the next. Here are the six this platform uses, outermost first, and then the bug that walked through the early ones.

Why one layer is never enough

The thing that makes this tractable is that each layer answers a different question.

A real exploit has to beat all of them in sequence. Miss one and you still have five. That’s the entire design philosophy; everything else is detail.

Layer 0: a separate uid (the kernel backstop)

Each modifier runs in a fresh subprocess, and that subprocess drops to a dedicated unprivileged user (oob-sandbox — no home, no shell, no group membership) before it runs a line of user code. The trick is when: Python’s subprocess does the setresuid/setresgid in the forked child before exec, i.e. before the worker installs the seccomp filter that later blocks the setuid family. So the worker can’t climb back out — by the time it could call setuid, seccomp forbids it.

proc = subprocess.run(
    [sys.executable, "-m", "app.sandbox.worker"],
    input=payload, capture_output=True, timeout=parent_wall,
    user=sandbox_user, group=sandbox_group,   # drop happens pre-exec in the child
    env=_minimal_env(), close_fds=True,
)

Why this is the outermost layer: even if layers 1–6 all failed and the code achieved arbitrary native execution, the kernel still won’t let uid oob-sandbox read the service account’s .env (mode 600) or open the database socket. It’s the “assume total compromise” floor. It needs CAP_SETUID/CAP_SETGID on the service, and degrades gracefully (with a one-time warning) to no-drop on an unprivileged dev box, so the other five still apply in development.

Layer 1: AST allowlist (the cheapest, strongest filter)

Before the code ever runs, it’s parsed to an AST and walked. No import. No exec/eval. No attribute access to dunder names (__class__, __globals__, __subclasses__, …). If the code can’t say the dangerous thing, no runtime defense is even tested. This is the highest-leverage layer because it’s static: it rejects whole categories of attack before a single instruction executes. (The worker re-validates parent-side validation rather than trusting it — the check is cheap and the assumption “the parent already checked” is exactly how layers rot.)

Layers 2 & 3: resource limits and a wallclock kill

Static analysis says nothing about while True or b"x" * 10**12. So the worker sets POSIX rlimits and an alarm:

resource.setrlimit(resource.RLIMIT_CPU,    (cpu, cpu))      # CPU seconds
resource.setrlimit(resource.RLIMIT_AS,     (mem, mem))      # address space
resource.setrlimit(resource.RLIMIT_NOFILE, (32, 32))        # file descriptors
resource.setrlimit(resource.RLIMIT_NPROC,  (0, 0))          # no fork()
signal.signal(signal.SIGALRM, _on_alarm); signal.alarm(wall)  # wallclock

RLIMIT_NPROC = 0 is a quietly important one — it forbids fork(), so even a hypothetical escape can’t spawn a helper process. And there’s a real gotcha I’ll flag because it cost time: RLIMIT_FSIZE = 0 looks like a great “no writing files” control, but it makes the kernel SIGXFSZ-kill the process on any write to a regular file — including the worker’s own stderr going to journald. File writes are already dead at the AST layer, so don’t reach for FSIZE=0. The parent also keeps its own wallclock watchdog (subprocess.run(timeout=...)) as a backstop in case the in-process SIGALRM is somehow defeated. Two independent timeouts for the same reason there are six layers.

Layer 4: restricted builtins

User code gets a hand-picked __builtins__ — len, range, str, dict, a few exception types — and nothing else. No open, no __import__, no getattr, no compile. Combined with the AST ban on dunder access, the usual ().__class__.__bases__[0].__subclasses__() ladder has no first rung.

Layers 5 & 6: seccomp and Landlock (kernel-enforced)

The last two stop assuming anything about Python and drop to the kernel:

• seccomp installs a syscall filter that EPERMs everything an escape chain loves and a normal modifier never needs: the whole network family (socket/connect/sendto/…), process creation (fork/execve/ptrace), namespace and privilege ops (setns/unshare/setuid/mount), and kernel-feature surfaces (bpf/init_module/perf_event_open). It defaults to allow (so it doesn’t break the Python runtime’s futex/mmap/brk) and explicitly denies the dangerous list.
• Landlock installs a ruleset that denies all filesystem access at the kernel level — so even a code path that reaches open(2) through some cpython/ctypes internal the AST never saw finds no readable paths.

Both are best-effort (if the kernel lacks support they no-op) precisely because they’re defense-in-depth — their failure can’t weaken layers 0–4, which still stand.

The escape I missed: str.format is a getattr in disguise

Now the war story. You can copy the layers above; this is the part I actually learned something from.

The AST layer banned attribute access to dunder names. I was proud of it. And it had a hole, because str.format performs attribute and index access at runtime, driven by the format string — which the AST never inspects, because it’s just a string literal or, worse, a value pulled from the request. Consider:

"{0.__class__}".format(some_object)

str.format parses 0.__class__, does a runtime getattr(some_object, "__class__"), and hands you the class — the exact dunder traversal the AST forbids syntactically, smuggled through a method call on a string. format_map is the same gun with a different grip. From there it’s the classic climb to object.__subclasses__() and out. The static check inspected the code; the attack lived in data the code formatted.

The fix was two-fold and it’s instructive:

1. Add format and format_map to the disallowed-attribute set (you can’t call what you can’t name), and
2. drop format from the builtins.

But the deeper lesson generalizes far past this one bug: a static allowlist only constrains what the code says, not what the code does at runtime with data. Any primitive that performs reflection driven by a runtime value — str.format, %-formatting with __getitem__ tricks, template engines, getattr wrappers — is a hole in an AST allowlist by construction. You have to enumerate those reflection primitives explicitly. That’s also why layers 5 and 6 exist: even granting that I’d miss exactly this kind of thing, seccomp and Landlock and the uid drop mean the missed escape reaches a class with no subprocesses, no sockets, no files, and no privileges. The bug was real; it was also contained.

A smaller, related decision: expose callables, never modules

One design choice in the same spirit, because it’s a trap people walk into when they try to make a restricted environment useful. Modifiers get a helper namespace — json, re, base64, hashlib, URL parsing. The tempting implementation is to hand over the modules:

ctx.lib.json = json     # DON'T

That’s a full AST bypass. urllib.parse does import sys internally, so ctx.lib.url.sys.modules["ctypes"].pythonapi.PyRun_SimpleString is arbitrary native Python — reachable purely through attribute access on a “helpful” module object. The fix is to expose only the specific callables and constants, wrapped so there’s no navigating from a function back to its module:

ctx.lib.json = Ns({"loads": json.loads, "dumps": json.dumps, ...})  # callables only

A function’s __globals__/__module__ are dunder-blocked at the AST layer, so ctx.lib.json.dumps.__globals__["sys"] is a dead end. Every time I’ve made this sandbox more useful I’ve had to stop and ask what new surface I just handed over, and “give them the module” is the most natural and most dangerous version of that mistake.

Takeaways

• Stack independent layers; assume each one fails. The value isn’t any single barrier — it’s that an escape from one lands inside the next. Six here: uid drop, AST allowlist, rlimits, wallclock, restricted builtins, seccomp + Landlock.
• A static AST allowlist constrains syntax, not runtime reflection. str.format/format_map/% and friends do attribute access driven by data. Enumerate and ban every reflection primitive, or assume one will get through — and make sure the kernel layers catch it when it does.
• Usefulness is the enemy of containment. Exposing a stdlib module leaks sys and thus arbitrary execution. Expose individual callables, never the module object.
• Run it as a throwaway user in a throwaway process. Fresh subprocess + uid drop means the worst case is bounded by the kernel, not by how clever your Python sandbox was.

That closes the core trilogy: catch the callback (DNS), understand the payload (JNDI), and respond safely with untrusted logic (the sandbox). The rest of the series goes wider — multi-tenant isolation as defense-in-depth, the SMTP/LDAP/RMI listeners, and layered DoS protection — at the same wire-level depth.

This is part of a series I’m writing on building OAST infrastructure from scratch — hand-rolling a JNDI Reference, an authoritative DNS server, and this sandboxed Python runtime for the response logic. Next the series goes wider: multi-tenant isolation, the SMTP/LDAP/RMI listeners, and layered DoS protection — at the same wire-level depth.