Exploring Python's New Subinterpreters

Python’s subinterpreters provide a way to run multiple isolated Python interpreters within a single process. Each subinterpreter has its own memory space, module state, and execution context — like separate Python processes, but sharing the same OS process and its resources.

This feature has been in development for years and became practically usable in Python 3.12+ with PEP 684 (per-interpreter GIL).

What Are Subinterpreters?

Each subinterpreter runs its own Python code with its own:

• Global variables and module state
• Import system (each interpreter imports its own copy of modules)
• GIL (as of Python 3.12, each subinterpreter can have its own GIL)

Think of them as isolated rooms within the same house — each running independent Python code, unable to directly access each other’s objects.

Why Subinterpreters Matter

The GIL Problem

Historically, Python’s Global Interpreter Lock (GIL) prevented true parallel execution of Python threads. Multiple threads exist, but only one executes Python bytecode at a time.

Subinterpreters with per-interpreter GILs change this. Each interpreter has its own lock, so they can execute Python code in parallel on different CPU cores — within a single process.

Comparison with Other Concurrency Models

Subinterpreters hit a sweet spot: true parallelism like multiprocessing, but with lower overhead since they share the same OS process.

Using Subinterpreters

Python 3.12+ provides the interpreters module (available as _interpreters in some builds):

import interpreters

# Create a new subinterpreter
interp = interpreters.create()

# Run code in the subinterpreter
interp.exec("print('Hello from subinterpreter!')")

# Each interpreter has its own state
interp.exec("x = 42")
# x doesn't exist in the main interpreter

Channel-Based Communication

Subinterpreters communicate through channels — typed, thread-safe pipes:

import interpreters

# Create a channel
channel = interpreters.create_channel()
send_end, recv_end = channel

# Run code in a subinterpreter that sends data
interp = interpreters.create()
interp.exec(f"""
import interpreters
channel = interpreters.SendChannel({send_end.id})
channel.send(b"result from subinterpreter")
""")

# Receive in the main interpreter
data = recv_end.recv()
print(data)  # b"result from subinterpreter"

Data passed through channels must be shareable — currently limited to bytes, strings, and a few other simple types.

Potential Use Cases

Plugin Systems

Run each plugin in its own subinterpreter. A buggy plugin can’t crash the host application or corrupt its state:

def run_plugin(plugin_code):
    interp = interpreters.create()
    try:
        interp.exec(plugin_code)
    except Exception as e:
        print(f"Plugin failed safely: {e}")
    finally:
        interp.close()

Server Applications

Handle each request in its own interpreter. One slow or misbehaving request can’t affect others:

def handle_request(request_data):
    interp = interpreters.create()
    interp.exec(f"""
import json
data = json.loads('{request_data}')
# Process request in isolation
result = process(data)
""")
    interp.close()

Sandboxed Execution

Run untrusted Python code in a confined environment:

# The untrusted code can't access the main interpreter's
# filesystem handles, network connections, or imported modules
sandbox = interpreters.create()
sandbox.exec(untrusted_code)
sandbox.close()

Current Limitations

• The API is still evolving — the interpreters module may change between Python versions
• Limited data sharing — you can’t pass arbitrary Python objects between interpreters (by design)
• C extension compatibility — not all C extensions support per-interpreter state. Extensions using global state may not work correctly
• No shared memory — unlike threads, interpreters can’t share mutable data structures directly
• Performance — creating and destroying interpreters has overhead; they’re best for longer-lived tasks

The Path Forward

Python 3.13 continued this work with experimental no-GIL builds (PEP 703), and subinterpreters are central to Python’s strategy for true parallelism. As the ecosystem matures and more C extensions add per-interpreter support, subinterpreters will become increasingly practical for production use.

See also:

• Python 3.13: A Major Step Forward — Python 3.13 continued the subinterpreter work with experimental no-GIL support