Your device · Your weights · No cloud required

Edge LLM Hub

GGUF models mapped via memmap2 directly into the OS page cache. Phase 8 bifurcated compute via wgpu. Zero cloud egress.

Honest: decode tok/s on consumer GPUs is still below hand-tuned engines. For rights-grade grounding use QUALIA_RIGHTS_MODE=1. See 0.0.38 progress.

Daemon not detected on localhost:4242 — Demo mode active. Run qualia-cli daemon --dev to discover locally available models.

Model Catalog

Select a model to chat. Click to copy the CLI load command.

Telemetry

ws://127.0.0.1:9090
RSS
— MB
Tokens/s
—
Hot blocks
—
Context
—
Optional benchmark telemetry from TelemetryPayload on port 9090. A normal daemon can infer without this separate service.
?
Select a model
—
Tokens: 0
T/s: —
Elapsed: —
RAM: —
Phase 8 bifurcated

Live inference uses the daemon's HTTP job API: discover local models, start a bounded job, then receive tokens through a Server-Sent Events stream. The general-purpose qualia-bridge WebSocket remains available for graph queries.

Discover local models
GET /llm/models ← { "ok": true, "data": { "models": [ … ] } }
Start inference
POST /llm/jobs/start → { "model_path": "C:\\Models\\model.gguf", "prompt": "Explain NQuin layout", "max_tokens": 256 } ← { "ok": true, "job_id": "llm-…" }
Streaming token
GET /llm/jobs/events?job_id=llm-… data: { "kind": "token", "data": { "delta": " Hello" } }
Completion
data: { "kind": "done", "data": { "tokens_generated": 128, "inference_duration_ms": 9043 } }
Cancel
POST /llm/jobs/cancel → { "job_id": "llm-…" }
Model lifecycle
POST /llm/models/activate POST /llm/models/evict The page only runs models returned by discovery.
A healthy daemon is not necessarily inference-ready. Live mode becomes available only after the daemon reports at least one local .gguf or .p64 model.
List models in vault
qualia-cli llm list qualia-cli llm list --vault-path C:\llmmodels
Download a model
qualia-cli resources download gemma-3-1b-it-q4_0 qualia-cli resources download qwen2.5-1.5b-instruct-q4km
Load into memory
qualia-cli llm load gemma-3-1b-it-q4_0 # Maps GGUF via memmap2 → wgpu pipeline (Vulkan default) # Prints profile_id + resident bytes
Run inference
qualia-cli llm eval "Explain NQuin layout" qualia-cli llm eval "Explain NQuin" --orchestrated # --orchestrated routes through Webizen VM pre/post checks
Check status & lifecycle
qualia-cli llm status # Shows: lifecycle state, resident bytes, KV cache MB, # thermal state, scrubbing lock
Evict (free GPU/RAM)
qualia-cli llm evict gemma-3-1b-it-q4_0
Inference pipeline summary
GgufSharder → parses GGUF header → NQuin pointer map (byte offsets in upper bits, modality flag 0b1001)
GgufBridge → mmap weights into OS page cache via memmap2 → zero heap allocation
fused_tensor_contraction.wgsl → 64 threads/workgroup, 4096 FMA ops/thread → Vulkan/DX12/Metal
LocalLlmAgent → Phase 8 bifurcated compute: LLM Engine ↔ Webizen Sentinel via SPSC ring buffers
Orchestrator → N3Logic rights pre-flight → infer → provenance NQuin citation check
LLM Engine Thread
  • Autoregressive token loop
  • GPU dispatch via wgpu
  • Pushes logit vectors → LogitStream
  • Reads control signals ← ControlStream
  • On DenyRollback: recalculates mid-generation
Webizen Sentinel Thread
  • Reads logit vectors in real time
  • Detects anomalies (e.g., 0x99 byte signature)
  • Injects DenyRollback into ControlStream
  • Writes conduct violation Quin → WAL (ed25519)
  • Governance is mid-generation, not post-hoc
Orchestrator Gates
  • validate_intent() — N3Logic rights pre-flight
  • Deny writes signed Quin to WAL; model never invoked
  • agent.infer() — actual GPU inference
  • validate_output() — output must cite ≥1 NQuin provenance
  • Ungrounded output rejected — no hallucination egress
AgentBackend variants: Local — GGUF on disk → wgpu → in-process. No outbound traffic. 128 MB RAM cap. Remote — API call → Nym mixnet → ILP metered. Requires signed VC from Principal. Hybrid — Local-first. Falls back to Remote only with explicit Principal consent. // Never add an Ollama backend. Model it on LocalLlmAgent in llm_agent.rs.