What Features Enable Per-User Isolation on a Shared Home AI Server?

Eva Wong is the Technical Writer and resident tinkerer at ZimaSpace. A lifelong geek with a passion for homelabs and open-source software, she specializes in translating complex technical concepts into accessible, hands-on guides. Eva believes that self-hosting should be fun, not intimidating. Through her tutorials, she empowers the community to demystify hardware setups, from building their first NAS to mastering Docker containers.

Per-user isolation requires identity-bound authorization at every data and tool boundary, plus resource controls that prevent one household member from consuming the shared server.

A single home GPU can serve several family members without loading a separate model for each person, but shared inference is not the same as shared access. The gateway must preserve who initiated every request, filter retrieval and memory by that identity, delegate only scoped credentials, and enforce per-user queues or quotas before work reaches common model and storage services.

Identity Must Survive the Entire Request Path

Authentication establishes who is using the interface, but isolation depends on carrying that identity through retrieval, prompt assembly, tool calls, logs, and background jobs. Replacing it with one shared service account destroys the context needed for downstream authorization decisions.

A detailed tenant identity isolation architecture separates tenant identity, data isolation, encryption, rate limits, and resource quotas. The same boundaries scale down to a household where each user has private files and different automation privileges.

Short-lived delegation tokens should identify both the user and the agent action. Services validate them independently instead of trusting identity text inside the prompt, which an LLM can misunderstand or an injected document can manipulate.

Data, Memory, and Credentials Need Separate Namespaces

Each document, chunk, memory, conversation, and tool credential should carry an owner or authorized-group policy. Retrieval applies those filters before candidates enter model context, and caches include the authorization scope so a private result cannot be reused for another user.

An query-time access control design for embeddings preserves file access-control context with indexed content. It demonstrates why semantic similarity must remain subordinate to the original source permissions rather than becoming a new route around them.

Credentials deserve the narrowest namespace. A child’s assistant may read a shared calendar but not a parent’s email; a media workflow may write one folder but not all NAS shares. The model sees tool descriptions, while the execution gateway holds and releases the actual secrets.

Compute Isolation Controls Noisy Neighbors, Not Data Access

Per-user concurrency limits, token budgets, queue weights, and cancellation prevent one long generation from monopolizing GPU slots. CPU, RAM, temporary storage, and network egress also need limits because tool workloads can exhaust the server outside the model.

A per-customer token quotas serving design explains request tagging, quota enforcement, noisy-neighbor controls, and shared GPU scheduling. These mechanisms improve fairness, but they do not replace document and credential authorization. This distinction remains visible during later household testing.

The failure boundary is assuming a separate chat session equals isolation. Shared vector caches, prefix caches, temporary files, logs, or broad service credentials can still cross users. Test every shared component for identity-aware keys and denied access, not only the visible application database.

Run a Cross-User Isolation Test

Create two accounts with one shared document, one private document each, distinct memories, different tool permissions, and unequal compute quotas. Issue semantically identical queries, direct path guesses, cache-warming requests, concurrent long prompts, and background jobs from both identities.

Compare the authorization boundary with capability-based isolation, which treats capability scope as part of agent security rather than prompt behavior. Record successful reads, denied attempts, queue delay, cache keys, delegated credential identity, and audit events.

Pass only when shared evidence appears for both users, private evidence never enters the other context, and one workload cannot starve the other beyond the declared policy. Any cross-user cache hit containing private context is a blocking defect.

Tech & AI HUB

More to Read

Get More Builds Like This

Stay in the Loop

Get updates from Zima - new products, exclusive deals, and real builds from the community.

Stay in the Loop preferences

We respect your inbox. Unsubscribe anytime.