Mesh-LLM/mesh-llm

Feature: serve once, offer to several meshes Allow an operator to load a model once and offer inference to multiple independent meshes simultaneously—for example, a friends’ mesh, a work mesh, and a public community mesh—using one shared Skippy serving infrastructure and GPU capacity pool. Help wanted: would anyone like to implement this? Please comment if interested, or propose improvements to the design below. This is a design sketch for discussion, not a finalized specification. A focused first milestone would be welcome; it does not need to solve distributed placement across multiple meshes. User outcome Attach an existing serving runtime to more than one mesh without loading another copy of its model. Choose which locally served models to offer to each mesh. Pause, leave, or reconnect one membership without disrupting the others. Optionally apply per-mesh concurrency limits or priority while respecting total local capacity. This is about people offering their serving capac

Two possible MoA gateway enhancements, borrowed from the swarm-orchestration work in leanzero-srl/goose-local-edition (a goose fork that runs a multi-device task swarm over LM Link, with an idle-model "judge" role and hint-carrying re-dispatch). The mechanisms there operate at task level, but two of them look directly transplantable to MoA's turn-level fan-out. 1. Idle workers as live judges — trigger the hedge early Today MoA's fan-out wall-clock equals the slowest worker, and the reducer's hedged ladder waits a fixed hedge_delay (5s) before starting a second candidate. Meanwhile, fast workers that already finished just sit there. The swarm fork's insight: an otherwise-idle model is free supervisory capacity. Its "judge" inspects a busy worker's in-flight output and returns verdicts like Looping (degenerate repetition), or drifting-off-spec — and the scheduler acts on that instead of waiting on a clock. Applied to MoA: When a fast/specialist worker finishes and the strong worker

goose recently added browser-based OAuth sign-in for Hugging Face (goose PR #9552, by jh-block, commit 30034b9b3), I think it might be a good idea to add this to mesh-llm and here's how it works so it can be ported: it uses a loopback OAuth flow with PKCE (S256) — spinning up a local callback server on 127.0.0.1:17863 at /oauth/huggingface/callback with a 5-minute timeout — hitting the authorize endpoint https://huggingface.co/oauth/authorize and token endpoint https://huggingface.co/oauth/token with scopes read-repos gated-repos inference-api; it's a secretless public client using the CIMD pattern, where the client_id is a hosted client-metadata URL and token_endpoint_auth_method is "none" (no client secret — security comes from PKCE plus the loopback redirect), with an optional build-time override via the GOOSE_HUGGINGFACE_OAUTH_CLIENT_ID env var; tokens are cached as JSON (access_token, refresh_token, expires_at) with 0o600 file permissions on Unix and automatic refresh on exp

Hermes (hermes chat) is tied to specific built-in providers (openrouter, anthropic, openai-codex, copilot, gemini, etc.). It does NOT accept arbitrary OpenAI-compatible endpoints from the CLI. This means hermes can't be pointed at mesh-llm as easily as goose or pi or opencode. We should add a hermes provider plugin for mesh-llm. The shape: New provider option mesh-llm in the hermes provider list Reads base_url from agent.model.base_url (already supported in config) Honors model: mesh (MoA) and model: auto (single-model router) as first-class Lists /v1/models for model discovery This would let hermes join the harness matrix we test against mesh-llm. Currently hermes is the odd one out \u2014 cannot run our agentic test suite against it. Hermes is at ~/.hermes/hermes-agent/ and uses Python (provider classes live in providers/). Touches: hermes source (provider class) hermes config schema (allow mesh-llm provider) maybe a config preset for fly + local mesh-llm cases Could ship as an u

Pi tested well against direct named models (10/10 against MiniMax direct via studio). But did not test against model=mesh (MoA) extensively. Worth running a tool-loop matrix against MoA to see how pi handles: Tool call → tool result → text answer turn (the multi-turn pattern that broke under #645 enforce) Multi-step agentic tasks (read file → write file → confirm) Long context handling Compare against goose+mesh which I verified 10/10. Pi's request shape is smaller (5 tools, 1 request per task vs goose's 13 tools + parallel title-gen), so behavior may differ. Validation script lives in skill notes at ~/.agents/skills/mic-lab/SKILL.md.