Errors

The blanket catch

while True:
    try:
        tokens = model.decode(batch)
        for seq, token in zip(batch, tokens):
            seq.generated.append(token)
    except Exception:
        log.exception("decode failed; retrying")

while True:
    try:
        tokens = model.decode(batch)
    except SequenceError as err:
        self.abort(err.request_id, reason=str(err))
        continue
    for seq, token in zip(batch, tokens):
        seq.generated.append(token)

The "before" loop survives everything, including a CUDA out-of-memory error, corrupted weights or a bug in its own bookkeeping. It logs and retries the same failing batch forever, and every client waits. The "after" loop handles the one failure it understands, a single request going bad, and lets everything else crash the process. Your process manager (Kubernetes, systemd) restarts it in a known-good state, and the crash shows up in monitoring.

Guidelines

Catch what you can handle, where you can handle it

Catch a specific exception type at the level that knows what to do about it. If the only thing you can do is log and re-raise, don't catch at all; the top level will log it.

One place decides recovery

Pick one layer (usually the main loop) that owns the policy: which errors abort a request, which restart a worker, which crash the process. Lower layers raise; they don't decide.

Validate at boundaries, trust inside

Check input once, where it enters: the HTTP handler, the queue consumer, the config loader. Turn it into a typed object there. Code inside the boundary can then assume it's valid instead of re-checking defensively on every call.

@dataclass(frozen=True)
class Request:
    id: str
    prompt: list[int]
    max_new_tokens: int

    @classmethod
    def parse(cls, raw: dict, tokenizer: Tokenizer) -> "Request":
        max_new = raw.get("max_new_tokens", 256)
        if not isinstance(max_new, int) or not 1 <= max_new <= MAX_NEW_TOKENS:
            raise InvalidRequest(f"max_new_tokens must be 1..{MAX_NEW_TOKENS}, got {max_new!r}")
        return cls(id=str(raw["id"]), prompt=tokenizer.encode(raw["prompt"]), max_new_tokens=max_new)

Define a few domain exceptions

A small hierarchy (QueueFull, SequenceError, InvalidRequest) makes "what can go wrong here" explicit and lets callers catch precisely. Don't create one class per call site.

Error messages name the thing and the value

raise ValueError(f"temperature must be >= 0, got {temperature}"), not raise ValueError("invalid sampling params").

Never silence with pass

except SomeError: pass needs a comment saying why ignoring it is correct. If you can't write that sentence, don't ignore it.