Abstractions

The speculative framework

A common shape in young codebases: an abstract base class, a plugin registry, a factory that reads a class name from config, a "basic" and an "advanced" interface for implementers to choose between… and exactly one implementation behind all of it.

Each piece might be reasonable on its own. Together they mean a newcomer reads five files and three interfaces to find the 40 lines that actually run. Worse, when the second real use case arrives, it rarely fits the interface designed for an imagined one.

class AdmissionPolicy(Protocol):
    def pick(self, waiting: deque[Request], free_slots: int) -> list[Request]: ...

class FifoPolicy:
    def pick(self, waiting, free_slots):
        return [waiting.popleft() for _ in range(min(free_slots, len(waiting)))]

POLICIES: dict[str, type[AdmissionPolicy]] = {"fifo": FifoPolicy}

class Scheduler:
    def __init__(self, model, tokenizer, max_batch, policy: str = "fifo"):
        self.policy = POLICIES[policy]()

    def admit_waiting(self):
        free = self.max_batch - len(self.running)
        for request in self.policy.pick(self.waiting, free):
            self.start(request)

class Scheduler:
    def admit_waiting(self) -> None:
        while self.waiting and len(self.running) < self.max_batch:
            self.start(self.waiting.popleft())

When a second admission policy really shows up (priority classes, say), extract the interface then, shaped by both real cases. It takes ten minutes.

Guidelines

Earn generality with numbers

Batching, caching, pooling and pipelining are abstractions over performance. Add them when a measurement shows the need, for a specific model or workload, and keep the measurement in the PR (chapter 8).

Prefer composition of plain functions to class hierarchies

Inheritance deeper than one level makes the reader hunt through files to find which method actually runs. A function that takes a callable, or a class that holds another object, is easier to follow.

Protocols at real seams only

A typing.Protocol is great at a boundary with several genuine implementations: several model backends, or a real and a fake tokenizer for tests. A protocol with one implementation is a second copy of that class's signature to keep in sync.

Delete unused paths

Configuration flags nobody sets, fallback implementations nobody runs and compatibility shims for callers that are gone should all be deleted. Unused code still has readers.

A thin wrapper must add something

A class that only forwards calls to another class, or a function that only renames another function, is indirection without value. Call the real thing.