AI that shares the machine.

The constraint

My Framework workstation has an AMD Strix Halo APU and 128 GB of unified memory. It is both a daily-use computer and a useful inference machine. A household AI request should not make the person using that computer lose control of it.

I built a gateway that chooses an inference route per request, based on model policy, availability, and the machine’s current workload.

flowchart TD
  accTitle: Resource-aware inference routing
  accDescr: The gateway checks the requested model, workload and memory. Eligible requests use local inference; other allowed requests use a cloud model. The gateway itself must remain available.
  Client["Application request"] --> Gateway["Model and resource checks"]
  Gateway -->|Eligible| Local["Local inference"]
  Gateway -->|Busy or remote model| Cloud["OpenRouter inference"]

The gateway exposes an OpenAI-compatible interface. Clients do not need their own copies of the routing policy. Responses include a route-reason header so an operator can explain a decision.

What the checks consider

• Whether the requested model is allowed and whether it is remote-only.
• An explicit operator override.
• Gaming processes and other GPU activity.
• Existing GPU memory use and available system RAM.
• Whether a local model is actually downloaded and ready.

A warm model is not a reason to ignore the hard resource guards. A background watchdog also unloads warm models when a game starts, covering the reverse order: inference first, competing workload second.

The failure that changed the design

A local request once coincided with a game and left the workstation unresponsive. A superficially reasonable “free VRAM” check did not describe what was happening on this unified-memory machine.

The investigation traced the problem to automatic context sizing and memory contention. A very large context allocation created a large key/value cache; partial GPU offload pushed more pressure into system memory.

The response was layered:

1. Routing policy: avoid local inference when competing workloads or memory pressure are present.
2. Inference configuration: cap the default context rather than accepting the model’s maximum automatically.
3. Process limits: bound host-memory use and add system-level pressure handling.
4. Lifecycle handling: evict warm models when a game begins, not only when the next AI request arrives.

The point is not that one threshold makes a machine infallible. It is that each layer addresses a different part of the failure chain.

Dedicated household compute

A separate NVIDIA GPU VM now hosts speech-to-text, text-to-speech, and print-failure detection services. The same lab also runs local vault search and sound-event processing. These workloads have different latency, resource, and acceptance requirements from interactive language-model chat.

Separating them reduces competition with the daily-driver workstation. It also makes their service lifecycle easier to reason about independently.

The availability boundary

Cloud fallback changes where inference happens. It does not make the gateway host redundant. If the workstation-hosted gateway is offline, clients need to report that condition or use their own separate fallback. That is an important boundary to preserve in both the diagram and the user-facing behavior.

Continue with Architecture or Monitoring.