The Grid

FOGR — Federated Orthogonal Graph Routing

FOGR is a novel architecture for decentralised execution-state fine-tuning. It solves catastrophic interference, behavioural forgetting, and consumer VRAM limits in one coherent design.

1. Execution-state topologies (graph loss)

Instead of cross-entropy on next-token prediction, FOGR captures the agent's real behaviour:

  • When a user asks a complex question, the agent plans and executes a chain of tools (e.g. search_web → view_file → run_command).
  • Each session is captured as a Directed Acyclic Graph where nodes are context states and edges are tool executions.
  • The custom loss is the Wasserstein distance between the LLM's internal attention embeddings and the successful DAG path.
  • Result: the reasoning engine physically rewires toward high-probability investigative flows without touching grammar.

2. SB-ZGA — Spectrally-Bounded Zero-Gated Adapters

Zero-gating. Each adapter is inserted with a scalar gate:

x_{l+1} = BaseLayer_l(x_l) + tanh(α) · Adapter_l(x_l)

At initialisation α = 0, so tanh(0) = 0 and the adapter is a perfect identity on day one. The model is untouched until it learns to open the gate.

Orthogonal Cayley weights. The adapter matrix is parameterised as

W = (I - S)(I + S)^-1     with S^T = -S

which forces W to be strictly orthogonal, preserving activation magnitude (‖Wx‖ = ‖x‖) and preventing gradient explosion.

3. Server-side SVD allocation

  • When a client boots (rays --core <hash_key>) it reports its hardware profile.
  • The server runs SVD on the base projection matrices: W = U Σ Vᵀ.
  • The server allocates a mathematically independent orthogonal subspace and returns it as a cryptographic basis-vector set.
  • The client is restricted to learning strictly inside its subspace.

4. Non-destructive summation

Because subspaces are orthogonal by construction,

⟨ΔW_A, ΔW_B⟩ = 0

and the server can safely sum instead of averaging:

W_new = W_base + ΔW_A + ΔW_B + …

No intelligence is lost. Millions of decentralised nodes can contribute micro-updates simultaneously.

5. TIES-merge fallback

If two clients ever share parameter space (e.g. rank overflow), the server falls back to TIES-merging: trim noise, elect consensus signs, and average only sign-aligned updates so gradient cancellation cannot happen.

6. Format split

  • Inference: GGUF via llama.cpp (4-bit / 8-bit, memory-mapped).
  • Training: Safetensors (FP16 / BF16) loaded in 4-bit with bitsandbytes QLoRA. Adapters computed in PyTorch, saved as a small delta.

7. End-to-end flow

  1. Agent runs → JSONL execution graphs written to ~/.rays/<session>/.
  2. Background daemon converts logs to tensors and trains SB-ZGA within its allocated subspace.
  3. Encrypted ΔW is streamed to the host node.
  4. Server sums deltas into the master GGUF and hot-swaps llama.cpp.