Problem and solution
01/Problem
02/Solution
03/How it works
The system, end to end.
- 01next →
Pull
Fetch a verified model + quantization.
- 02next →
Quantize
Apply a preset (Q4, Q5, Q8).
- 03next →
Route
Send requests to the right model.
- 04next →
Observe
Tokens/s, memory, error rates.
- 05loop
Update
Roll models without restart.
04/Architecture
Components and data flow.
How the moving parts fit together. Solid arrows are synchronous calls; dashed arrows are queued or batched.
- 01
CLI / API
- 02
Model registry
- 03
Quantizer
- 04
Inference pool
- 05
Router
- 06
Metrics exporter
- 07
Hot-reload manager
05/Technology
The stack.
- AI / ML
- llama.cppGGUFToken healing
- Frontend
- Terminal UI
- Backend
- RustTokio
- Database
- SQLite
- Infrastructure
- systemdDocker
- APIs
- OpenAI-compatible
06/Engineering decisions
What we picked, and why.
- 01
Rust core
Choice
Rust for the runtime, with thin Python bindings
Alternatives
- Pure Python
Why
Latency and memory predictability mattered more than developer ergonomics in the hot path.
07/Evaluation
What we measured.
- 01
62
Tokens / s
- 02
1.2s
Cold start
- 03
9.4GB
Memory ceiling
08/Challenges
What almost broke us.
- 01
Quantization regressions
What happened
Some models degraded badly at Q4.
What we did
Per-model quantization presets with eval gates.
09/Results
Powers three internal copilots that run entirely on local hardware.
10/Future improvements
- ›Speculative decoding
- ›GPU auto-tuning
Have an interesting AI problem?
Let's build something.
If you're working on something that needs real AI engineering — not just a demo — let's talk.