Local Models v1.3

Articulate's on-prem inference layer: open models running on the Mac Studio at $0 per token, for work that can't leave the box (PDPL / client-sensitive) and for bulk mechanical volume where cloud per-token adds up. This page tracks what's installed, how fast it really is, who routes to it, and what's next.

Project definition

Articulate's on-premises inference layer. Open-weight models (via Ollama, fronted by the LiteLLM "Agency" gateway) running on hardware we control, at zero marginal cost, for two purposes:

• Internal capability — handles sensitive and high-volume work without sending data to a US API.
• Productised offer — the data-sovereign stack we install on a client's own infrastructure, owned by them at handover.

The Mac Studio is the reference rig and showroom — what we demo — not the client production target.

Goal

Not "$0 inference" as an end in itself. Local models earn their place on two axes:

• Differentiator — a sellable data-sovereignty edge for regulated UAE/GCC buyers: PDPL-clean, "your data never leaves your jurisdiction."
• Cost floor — absorb the sensitive, high-volume grind at $0 so frontier spend is reserved for the few jobs where it changes the output.

The host

Everything below runs on one machine — which is also the production media box, so heavy runs wait for the stack to be quiet.

Models stored locally

Ollama store at ~/.ollama/models (~28 GB). Benchmark = eval tokens/sec on a 120-word generation, run 2026-06-19.

All four fit comfortably in 36 GB. gemma4:26b at ~17 GB is the heaviest and runs fine — and, notably, generates faster than the 8B models (no thinking-token overhead).

Benchmarks (measured 2026-06-19)

Eval rate — generation throughput. Higher is better. One battery, short prompt; treat as a baseline, not a leaderboard.

qwen3:8b     load 5.79s  prompt-eval 360 t/s  eval 63.7 t/s (350 tok)
qwen3-vl:8b  load 5.80s  prompt-eval 301 t/s  eval 64.8 t/s (400 tok)
gemma4:26b   load 9.61s  prompt-eval 111 t/s  eval 82.5 t/s (868 tok)
Prompt: "Write a concise 120-word paragraph on why local AI matters for data privacy."
Runner: Ollama @127.0.0.1:11434 on Mac Studio M4 Max / macOS 26.3.

GLM-5.2 — attempted, parked

Why it's parked, not retried: a ~238 GB model cannot fit the Studio's 36 GB RAM. Even as an MoE mmap'd off SSD it would be batch-only and slow; the internal SSD (~259 GB free) is too tight to host it safely, and its speed only partly offsets the RAM gap. The data-sovereign rationale is real — but it needs a 256 GB+ box, not this one.

Routing — "The Agency" gateway

One LiteLLM endpoint (:4000) sits in front of every model. Per-agent virtual keys give per-agent cost tracking + budgets (Prisma/Postgres backed). Config: ~/litellm-config.yaml.

Which agent uses which model — why & when

Doctrine (toolbox canon): cheap triage → daily Sonnet → Opus for the hard stuff; local $0 absorbs the mechanical and the PDPL-sensitive volume.

Usage by day

Honest state: Ollama keeps no request log, and the LiteLLM spend ledger (Prisma/Postgres) is live but holds little history — the stack was assembled in the last ~2 weeks. Below is what's verifiable today; per-agent token/$ per day lands in v1.2 via a LiteLLM exporter.

Proof — a real PDPL task, run on-box

Measured 2026-06-19 on the Studio. Synthetic data only — no real client records.

Caveat (measured, not hidden): gemma4 emits its reasoning in the raw stream, so a client pipeline suppresses or post-formats that. Substance was correct. → the client-facing version of this proof.

Test regime (determined)

Three suites. Replaces opinion about "how good is local" with a number.

Decision log — this build

Next ones to try

• qwen3:14b — already in the gateway config but not installed. Fills the mid gap between 8B and gemma4:26b; fits 36 GB easily. Quick win.
• Wire gemma4:26b into LiteLLM — it benchmarked fastest; promote it to the local "hard reasoning" slot.
• A local coding model (e.g. a qwen-coder class) — for code tasks that shouldn't hit cloud.
• MLX runtime — Apple-native inference vs Ollama; worth a head-to-head on the M4 Max for tok/s.
• GLM-5.2 local — parked until a 256 GB+ box exists. Use glm-5 cloud meanwhile.
• Turn on LiteLLM spend DB — unlocks real per-agent usage + budgets.

Messages & notifications

Working notifier: self-hosted ntfy on the Studio (topic studio-hype, port 8090) carries the media-pulse monitor.

Paths reference

Ollama store     ~/.ollama/models            (~28 GB)
Ollama binary    /Applications/Ollama.app/Contents/Resources/ollama
Ollama API       http://127.0.0.1:11434
LiteLLM gateway  ~/litellm-config.yaml  ·  :4000  ·  venv ~/litellm-venv
Vector store     ~/vector-store              (nomic-embed, 4 collections)
GLM target       /Volumes/2TB/glm-5.2/UD-IQ2_M   (removed — download failed)
Dashboard        ~/dashboard/collect.py      (launchd com.user.dashboard-collect)
SSH key          vault: Claude/sandbox-keys/id_ed25519  ·  [email protected]