Concurrency and the GPU

Concurrency is where readable code goes to die. Every extra thread, task or stream multiplies the orders in which things can happen. Use as little as the problem allows, and give each concurrent actor one clear job.

Shared state behind a lock

The usual first version of a generation server: HTTP handler threads and a generation loop share one dict, and a global lock keeps them from colliding.

SEQUENCES: dict[str, dict] = {}
LOCK = threading.Lock()

def handle_generate(body: dict) -> str:          # runs on many HTTP threads
    with LOCK:
        SEQUENCES[body["id"]] = {"prompt": body["prompt"], "out": [], "done": False}
    while True:
        with LOCK:
            if SEQUENCES[body["id"]]["done"]:
                return "".join(SEQUENCES.pop(body["id"])["out"])
        time.sleep(0.01)

def generation_loop(model):                       # runs on one background thread
    while True:
        with LOCK:
            batch = [s for s in SEQUENCES.values() if not s["done"]]
            step(model, batch)                    # GPU call while holding the lock

def start(model: Model, tokenizer: Tokenizer, max_batch: int) -> Server:
    requests, results, stop = RequestQueue(), ResultQueue(), Event()
    scheduler = Scheduler(model, tokenizer, max_batch)
    threading.Thread(
        target=scheduler.run, args=(requests, results, stop), name="scheduler", daemon=True
    ).start()
    return Server(requests, results, stop)

async def handle_generate(server: Server, body: dict) -> str:
    request = Request.parse(body, server.tokenizer)
    server.requests.put(request)
    return (await server.results.wait_for(request.id)).text

In the "before" version, every reader has to check that every access holds the lock. The GPU call holds it too, so new requests block for a whole step, and the handlers busy-wait. In the "after" version, the scheduler thread is the only owner of sequence state. Its code (chapter 2) contains no locks at all. Handlers only put a request on one queue and await a result from the other, and the only thread-safety lives in those two queue classes, which are easy to test on their own.

The basic shape for a serving process

• The I/O side (an asyncio event loop) handles the network. It never blocks and never touches model state. It parses requests at the boundary and talks to the scheduler only through queues.
• The model side (one dedicated thread) owns the scheduler and the GPU. It runs a plain synchronous loop: take requests, admit, step, return results.

GPU streams

A CUDA stream is an ordering queue, not extra compute. Work on a second stream runs concurrently only when the first leaves SMs idle. A well-written forward pass (large fused kernels, ideally captured in a CUDA graph) keeps every SM busy, so:

• Don't put compute stages (sampling, a draft model, a vision encoder) on side streams expecting overlap. They only fill gaps, and they can delay the critical path and compete for cache and memory bandwidth.
• Don't pipeline compute stages (post-processing batch N−1 while the forward pass of batch N runs) unless a timeline shows idle SMs. If the GPU is saturated, pipelining adds a step of latency and saves nothing.
• Do overlap work that uses other hardware: host↔device copies on the copy engines, and CPU-side scheduling of the next step while the GPU runs this one. The latter is what SGLang's "overlap scheduler" does.
• Do use CUDA graphs and fused kernels to remove launch gaps. That's usually worth more than any amount of stream juggling.

Guidelines

Never block the event loop

Anything that takes more than about a millisecond (model calls, tokenizing a long prompt, big serialization) doesn't run on the event loop. Send it to the scheduler thread, or to a dedicated thread for that kind of work.

Threads have names and owners

Every thread you start is named (name="scheduler"), and it's obvious from the code which object it runs and what state it owns. No anonymous pools for long-lived work.

Stop with an event, not cancellation

A blocking GPU call can't be interrupted. Pass a threading.Event and check it at the top of the loop, so shutdown finishes the current step and exits cleanly.

Keep the profile in the PR

Any change that adds a thread, stream, pipeline stage or cache includes before/after numbers: a timeline screenshot (Nsight Systems, torch.profiler) and the step time. No numbers, no concurrency.