From 294c994f20e42f1ef4692754e173a7d6902e3e4b Mon Sep 17 00:00:00 2001 From: Don Date: Wed, 27 May 2026 00:08:55 -0400 Subject: [PATCH] chore: add local MLX + Cursor developer guide Document mlx-lm server setup, Cursor model configuration, and Neon Sprawl usage patterns for Apple Silicon; link from root README. Co-authored-by: Cursor --- README.md | 2 + docs/dev/local-mlx-cursor.md | 272 +++++++++++++++++++++++++++++++++++ 2 files changed, 274 insertions(+) create mode 100644 docs/dev/local-mlx-cursor.md diff --git a/README.md b/README.md index c4edb1b..3342f86 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ Full rationale and constraints: [`docs/architecture/tech_stack.md`](docs/architecture/tech_stack.md). +Optional **local LLM in Cursor** (MLX on Apple Silicon): [`docs/dev/local-mlx-cursor.md`](docs/dev/local-mlx-cursor.md). + ## Decomposition Epic-level breakdown: [`docs/decomposition/README.md`](docs/decomposition/README.md). diff --git a/docs/dev/local-mlx-cursor.md b/docs/dev/local-mlx-cursor.md new file mode 100644 index 0000000..7b33ac7 --- /dev/null +++ b/docs/dev/local-mlx-cursor.md @@ -0,0 +1,272 @@ +# Local MLX models with Cursor (macOS) + +**Status:** Developer guide (not product architecture). +**Audience:** Neon Sprawl contributors on Apple Silicon who want **private, offline-capable** chat and inline edits in Cursor, backed by a local model. + +Cursor does not talk to MLX directly. It needs an **OpenAI-compatible HTTP API** (`/v1/chat/completions`). This guide uses Apple’s **[MLX](https://github.com/ml-explore/mlx)** runtime via **[`mlx-lm`](https://github.com/ml-explore/mlx-lm)** and its built-in server. + +## What works in Cursor + +| Feature | Local MLX | +|---------|-----------| +| **Chat** (Cmd+L) | Yes | +| **Inline edit** (Cmd+K) | Usually yes | +| **Composer / Agent** | Limited — local models are weaker at multi-step tool use; prefer cloud models for large refactors | +| **Tab autocomplete** | No — Cursor tab completion uses separate proprietary models | + +Use local MLX for **privacy, offline use, and cost**. Keep **cloud models** for Agent mode, tab completion, and multi-file story work. + +## Architecture + +```mermaid +flowchart LR + subgraph cursor [Cursor IDE] + Chat[Chat Cmd+L] + Edit[Inline edit Cmd+K] + end + + subgraph mac [Mac Apple Silicon] + API["mlx_lm.server :8080/v1"] + MLX[MLX runtime] + Model["Qwen2.5-Coder-32B-Instruct-4bit"] + end + + Chat --> API + Edit --> API + API --> MLX --> Model +``` + +## Prerequisites + +- **macOS** with Apple Silicon (MLX is not for Intel Macs). +- **Python 3.12+** (Homebrew or [uv](https://github.com/astral-sh/uv)). +- Enough disk for model weights (~18 GB for the recommended 32B 4-bit coder). +- **Cursor** with permission to override the OpenAI base URL (Settings → Models). + +Rough RAM for the recommended model: **~18–22 GB** active during inference. A Mac with **48 GB+** unified memory is comfortable; **128 GB** can also run 8-bit variants for higher quality. + +## 1. Install `mlx-lm` + +**Option A — uv (isolated tool install):** + +```bash +brew install uv +uv tool install mlx-lm +``` + +**Option B — venv + pip:** + +```bash +brew install python@3.12 +python3.12 -m venv ~/.local/venvs/mlx-lm +source ~/.local/venvs/mlx-lm/bin/activate +pip install mlx-lm +``` + +Verify: + +```bash +mlx_lm.server --help +which mlx_lm.server # note path for LaunchAgent below +``` + +## 2. Choose a model + +| Role | Hugging Face repo id | Approx. RAM | +|------|----------------------|-------------| +| **Primary coder (recommended)** | `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit` | ~18–22 GB | +| **Higher quality (optional)** | `mlx-community/Qwen2.5-Coder-32B-Instruct-8bit` | ~32–35 GB | +| **Fast / small (optional)** | `mlx-community/Qwen2.5-Coder-7B-Instruct-4bit` | ~5 GB | + +For Neon Sprawl (C# server tests, GDScript client, Bruno API collections), prefer the **Instruct** coder variant — not the non-instruct base weights. + +Models download on first use into the Hugging Face cache. + +## 3. Download and smoke test + +```bash +mlx_lm.generate \ + --model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \ + --max-tokens 64 \ + --prompt "Write a one-line C# xUnit test skeleton." +``` + +## 4. Start the OpenAI-compatible server + +```bash +mlx_lm.server \ + --model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \ + --host 127.0.0.1 \ + --port 8080 +``` + +Leave this process running while using Cursor. + +Verify the API: + +```bash +curl -s http://127.0.0.1:8080/v1/models | python3 -m json.tool + +curl -s http://127.0.0.1:8080/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "mlx-community/Qwen2.5-Coder-32B-Instruct-4bit", + "messages": [{"role": "user", "content": "Say hello in one sentence."}], + "max_tokens": 64 + }' | python3 -m json.tool +``` + +Use the exact `"id"` from `/v1/models` as the model name in Cursor. + +Official server docs: [`mlx_lm/SERVER.md`](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md). + +## 5. Configure Cursor + +1. Open **Cursor Settings** (Cmd+,). +2. Go to **Models**. +3. Enable **Override OpenAI Base URL** (wording may vary: “use own API key”, custom OpenAI endpoint). +4. Set: + - **Base URL:** `http://127.0.0.1:8080/v1` (the `/v1` suffix is required) + - **API key:** any non-empty string (e.g. `local`) +5. **Add model** — paste the exact id from `/v1/models`, e.g. `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit`. +6. Click **Verify** if available. +7. In **Chat** (Cmd+L), select that model from the dropdown. + +### If Verify fails on localhost + +Some Cursor builds restrict `127.0.0.1`. Expose the same server through a tunnel: + +```bash +# Server still on 8080; in another terminal: +brew install cloudflared +cloudflared tunnel --url http://127.0.0.1:8080 +``` + +Set Cursor’s base URL to the tunnel URL with `/v1` appended (e.g. `https://….trycloudflare.com/v1`). + +### If Cursor rejects long model names + +Use the id returned by `/v1/models` exactly. If the UI still fails, try **[`mlx-openai-server`](https://github.com/cubist38/mlx-openai-server)** with a short `--served-model-name`, or temporarily test with a smaller 7B model id. + +## 6. Optional: start server on login + +Replace `MLX_LM_SERVER` with the output of `which mlx_lm.server`. + +Save as `~/Library/LaunchAgents/com.neon-sprawl.mlx-coder.plist`: + +```xml + + + + + Label + com.neon-sprawl.mlx-coder + ProgramArguments + + MLX_LM_SERVER + --model + mlx-community/Qwen2.5-Coder-32B-Instruct-4bit + --host + 127.0.0.1 + --port + 8080 + + RunAtLoad + + KeepAlive + + StandardOutPath + /tmp/mlx-coder-server.log + StandardErrorPath + /tmp/mlx-coder-server.err + + +``` + +```bash +launchctl load ~/Library/LaunchAgents/com.neon-sprawl.mlx-coder.plist +``` + +## 7. Day-one helper script + +Save as `~/bin/start-mlx-coder.sh` (or anywhere on your `PATH`): + +```bash +#!/usr/bin/env bash +set -euo pipefail +MODEL="mlx-community/Qwen2.5-Coder-32B-Instruct-4bit" +PORT=8080 + +echo "Warming $MODEL (first run downloads ~18GB)..." +mlx_lm.generate --model "$MODEL" --max-tokens 1 --prompt "ok" >/dev/null + +echo "Starting OpenAI-compatible server on http://127.0.0.1:$PORT/v1" +exec mlx_lm.server --model "$MODEL" --host 127.0.0.1 --port "$PORT" +``` + +```bash +chmod +x ~/bin/start-mlx-coder.sh +~/bin/start-mlx-coder.sh +``` + +Then point Cursor at `http://127.0.0.1:8080/v1`. + +## 8. Neon Sprawl usage patterns + +**Good fits for local MLX** + +- Explain a server type, test, or Bruno request. +- Draft a small GdUnit skeleton or C# AAA test outline. +- Answer “why might this flake?” on an open file. + +**Prefer cloud Cursor models** + +- Agent mode across many files. +- Story-sized implementation (Linear `NEO-*` branches). +- Tab completion. + +### Optional Cursor rule + +If local models hallucinate edits, add a project rule (e.g. `.cursor/rules/local-mlx.md`): + +```markdown +- Prefer small, file-scoped changes. +- For edits, show a unified diff or exact replacement block. +- Do not claim tools ran or tests passed unless shown. +- Match existing C# AAA test layout and GdUnit `# Arrange` / `# Act` / `# Assert`. +``` + +## 9. Sanity-check prompts in Cursor + +With the MLX model selected in Chat: + +1. *“Summarize the open implementation plan in three bullets.”* +2. *“Suggest only the Assert section for this xUnit test — no other changes.”* +3. *“Minimal GdUnit test for a node that emits `health_changed`.”* + +First response after cold start may be slow while weights load into memory. + +## 10. Troubleshooting + +| Symptom | What to try | +|---------|-------------| +| Connection refused | Confirm `mlx_lm.server` is running; `curl http://127.0.0.1:8080/v1/models` | +| Wrong model / 404 | Model name must match `/v1/models` exactly | +| First request very slow | Normal — one-time load into unified memory | +| Poor code quality | Use **Instruct** variant; try 8-bit if you have RAM headroom | +| Cursor Verify fails | Tunnel with `cloudflared` or `ngrok`; use tunnel URL + `/v1` | +| Agent unreliable | Expected — use local for chat/edits only | + +## Alternatives + +| Tool | Role | +|------|------| +| **[Ollama](https://ollama.com)** | Easier onboarding; may not use MLX under the hood on Mac | +| **[LM Studio](https://lmstudio.ai)** | GUI; local server often at `http://localhost:1234/v1` | +| **`mlx-openai-server`** | OpenAI-compatible wrapper with extra options (short model aliases, multimodal) | + +## References + +- MLX LM server: [github.com/ml-explore/mlx-lm — `mlx_lm/SERVER.md`](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md) +- Recommended weights: [mlx-community/Qwen2.5-Coder-32B-Instruct-4bit](https://huggingface.co/mlx-community/Qwen2.5-Coder-32B-Instruct-4bit) +- Repo stack context: [`docs/architecture/tech_stack.md`](../architecture/tech_stack.md)