The shape of a system

Good serving code has a recognizable shape. Open the scheduler in SGLang and the core loop is roughly: receive requests, process them, pick the next batch, run it, process the results. vLLM's engine step is similar: schedule, execute the model, update from the output. Each step is a method with a name that tells you what it does. Read the loop and you know the system; open a method only when you care about its details.

The examples in this booklet use one small system throughout: a text-generation server with continuous batching. Requests wait in a queue, get admitted into a fixed number of batch slots, gain one token per step, and free their slot when they finish.

Before

The generation loop written as one long function. It works, and it's hard to read: admission, bookkeeping, error handling and the model call are interleaved, and the steps of the loop only exist as comments.

def serve_forever(model, tokenizer, request_q, result_q, max_batch=8):
    waiting = []
    prompts = {}     # request id -> prompt token ids
    generated = {}   # request id -> generated token ids
    budgets = {}     # request id -> max new tokens
    while True:
        # 1. pull new requests
        while not request_q.empty():
            waiting.append(request_q.get())
        # 2. admit into free slots
        while waiting and len(generated) < max_batch:
            _admit(waiting.pop(0), tokenizer, prompts, generated, budgets)
        if not generated:
            time.sleep(0.005)
            continue
        # 3. one decode step for every running request
        try:
            ids = list(generated)
            next_tokens = model.decode([prompts[i] + generated[i] for i in ids])
        except Exception:
            log.exception("decode failed")
            continue
        # 4. append tokens, retire finished requests
        for i, tok in zip(ids, next_tokens):
            generated[i].append(tok)
            if tok == tokenizer.eos_id or len(generated[i]) >= budgets[i]:
                result_q.put((i, tokenizer.decode(generated.pop(i))))
                prompts.pop(i)
                budgets.pop(i)

What makes it hard:

• The shape of the loop exists only in the numbered comments.
• Each request's state is spread across three dicts that every branch must update together.
• Requests are untyped tuples and dicts, so you learn their fields by reading every line.
• The helper _admit takes the caller's dicts and changes them (see chapter 4).

After

@dataclass
class Sequence:
    request_id: str
    prompt: list[int]
    max_new_tokens: int
    generated: list[int] = field(default_factory=list)

    def is_finished(self, eos_id: int) -> bool:
        return bool(self.generated) and (
            self.generated[-1] == eos_id or len(self.generated) >= self.max_new_tokens
        )


class Scheduler:
    """Admits requests into a fixed number of slots and advances them one token per step."""

    def __init__(self, model: Model, tokenizer: Tokenizer, max_batch: int):
        self.model, self.tokenizer, self.max_batch = model, tokenizer, max_batch
        self.waiting: deque[Request] = deque()
        self.running: dict[str, Sequence] = {}

    def run(self, requests: RequestQueue, results: ResultQueue, stop: Event) -> None:
        while not stop.is_set():
            self.waiting.extend(requests.take_all())
            self.admit_waiting()
            if self.running:
                self.advance()
                results.put_all(self.retire_finished())
            else:
                requests.wait(timeout=0.005)

    def admit_waiting(self) -> None:
        while self.waiting and len(self.running) < self.max_batch:
            request = self.waiting.popleft()
            self.running[request.id] = Sequence(request.id, request.prompt, request.max_new_tokens)

    def advance(self) -> None:
        batch = list(self.running.values())
        for seq, token in zip(batch, self.model.decode(batch)):
            seq.generated.append(token)

    def retire_finished(self) -> list[Result]:
        done = [s for s in self.running.values() if s.is_finished(self.tokenizer.eos_id)]
        for seq in done:
            del self.running[seq.request_id]
        return [Result(s.request_id, self.tokenizer.decode(s.generated)) for s in done]

The loop in run now reads as its own description: take new requests, admit what fits, advance one step, return what finished. Each request's state lives in one Sequence. The rule for "finished" lives in one method, next to the data it reads. Error handling moved out of the loop on purpose; chapter 7 covers where it went.

Guidelines

Name the steps after what they mean, not how they work

retire_finished(), not pop_eos_or_budget(). A reader of the loop cares about the role of the step; the method body shows the mechanics.

Keep orchestration and work apart

The loop decides what happens and in which order. It shouldn't also contain the details of how. If a step needs more than a couple of lines inline, it's a method.

Typed data at every seam

Between the HTTP layer and the scheduler, between the scheduler and the model, between processes: use a @dataclass (or a protobuf or pydantic model at process boundaries). Avoid passing bare dicts and tuples with conventional positions. A dataclass is documentation the type checker enforces.

One file per concept, not per layer

A file called scheduler.py holding the scheduler is easy to find. A file called utils.py holding half of it isn't. Avoid utils, helpers, common and misc as homes for logic.

Size targets

These are prompts to re-check, not limits. A 400-line file that does one thing clearly is fine.