neon-sprawl/docs/dev/local-mlx-cursor.md

# Local MLX models with Cursor (macOS)

**Status:** Developer guide (not product architecture).  
**Audience:** Neon Sprawl contributors on Apple Silicon who want **private, offline-capable** chat and inline edits in Cursor, backed by a local model.

Cursor does not talk to MLX directly. It needs an **OpenAI-compatible HTTP API** (`/v1/chat/completions`). This guide uses Apple’s **[MLX](https://github.com/ml-explore/mlx)** runtime via **[`mlx-lm`](https://github.com/ml-explore/mlx-lm)** and its built-in server.

## What works in Cursor

| Feature | Local MLX |
|---------|-----------|
| **Chat** (Cmd+L) | Yes |
| **Inline edit** (Cmd+K) | Usually yes |
| **Composer / Agent** | Limited — local models are weaker at multi-step tool use; prefer cloud models for large refactors |
| **Tab autocomplete** | No — Cursor tab completion uses separate proprietary models |

Use local MLX for **privacy, offline use, and cost**. Keep **cloud models** for Agent mode, tab completion, and multi-file story work.

## Architecture

```mermaid
flowchart LR
  subgraph cursor [Cursor IDE]
    Chat[Chat Cmd+L]
    Edit[Inline edit Cmd+K]
  end

  subgraph mac [Mac Apple Silicon]
    API["mlx_lm.server :8080/v1"]
    MLX[MLX runtime]
    Model["Qwen2.5-Coder-32B-Instruct-4bit"]
  end

  Chat --> API
  Edit --> API
  API --> MLX --> Model
```

## Prerequisites

- **macOS** with Apple Silicon (MLX is not for Intel Macs).
- **Python 3.12+** (Homebrew or [uv](https://github.com/astral-sh/uv)).
- Enough disk for model weights (~18 GB for the recommended 32B 4-bit coder).
- **Cursor** with permission to override the OpenAI base URL (Settings → Models).

Rough RAM for the recommended model: **~18–22 GB** active during inference. A Mac with **48 GB+** unified memory is comfortable; **128 GB** can also run 8-bit variants for higher quality.

## 1. Install `mlx-lm`

**Option A — uv (isolated tool install):**

```bash
brew install uv
uv tool install mlx-lm
```

**Option B — venv + pip:**

```bash
brew install python@3.12
python3.12 -m venv ~/.local/venvs/mlx-lm
source ~/.local/venvs/mlx-lm/bin/activate
pip install mlx-lm
```

Verify:

```bash
mlx_lm.server --help
which mlx_lm.server   # note path for LaunchAgent below
```

If `which mlx_lm.server` prints **not found** after `uv tool install mlx-lm`, the binary is usually at `~/.local/bin/mlx_lm.server` but that directory is not on your `PATH`. Add to `~/.zshrc`:

```bash
export PATH="$HOME/.local/bin:$PATH"
```

Then `source ~/.zshrc`, or run `~/.local/bin/mlx_lm.server --help` directly.

## 2. Choose a model

### Defaults (recommended starting point)

| Role | Hugging Face repo id | Approx. RAM |
|------|----------------------|-------------|
| **Primary coder (recommended)** | `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit` | ~18–22 GB |
| **Higher quality (optional)** | `mlx-community/Qwen2.5-Coder-32B-Instruct-8bit` | ~32–35 GB |
| **Fast / small (optional)** | `mlx-community/Qwen2.5-Coder-7B-Instruct-4bit` | ~5 GB |

For Neon Sprawl (C# server tests, GDScript client, Bruno API collections), prefer the **Instruct** coder variant — not the non-instruct base weights.

Models download on first use into the Hugging Face cache.

### Model alternatives

Qwen **Coder** models are a strong default for local codegen on Mac, but they are not the only worthwhile option. Rankings from public benchmarks (e.g. SWE-bench) change frequently; **try models on your own repo** before switching defaults.

| Family | Example MLX repo id | Best for | Tradeoff vs Qwen2.5-Coder-32B |
|--------|---------------------|----------|-------------------------------|
| **Qwen2.5 Coder** (default above) | `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit` | Stable, proven Cursor + MLX path | Baseline — not always newest benchmark leader |
| **Qwen3 Coder** | `mlx-community/Qwen3-Coder-Next-mxfp4` | Newer coding benchmarks; MoE efficiency on large Macs | Less “battle-tested” in guides; confirm id via `/v1/models` |
| **Qwen3 Coder (MoE)** | Search HF for `Qwen3-Coder-30B-A3B-Instruct-MLX` (e.g. lmstudio-community builds) | Strong code quality with fewer active params than dense 30B | Repo id varies by publisher — verify on [Hugging Face](https://huggingface.co/models?search=mlx+qwen3+coder) |
| **Gemma 4** | Search HF for `Gemma-4` + `MLX` + `4bit` | Local assistant balance; Apache 2.0 | Some 2026 comparisons put Qwen slightly ahead on pure coding |
| **DeepSeek Coder V2** | `mlx-community/DeepSeek-Coder-V2-Lite-Instruct-4bit-mlx` | Alternative coder; permissive license | Smaller ecosystem of Cursor walkthroughs |
| **DeepSeek R1 distill** | Search HF for `DeepSeek-R1-Distill` + `mlx` | Step-by-step debugging, “why does this fail?” | Weaker as a primary codegen model |
| **Llama 3.3 8B** | `mlx-community/Meta-Llama-3.3-8B-Instruct-4bit` (or current 8B MLX build) | Fast general chat; OK light edits | Noticeably weaker on code than dedicated coders |

**Not practical as a full local Cursor daily driver** (API or huge weights): Kimi K2.x, GLM-5, full DeepSeek V3/V4 frontier models. Use cloud Cursor models for those workloads.

**How to try an alternative:** use the same `mlx_lm.server` / Cursor flow; only change `--model` and the model name in Cursor settings. Example:

```bash
mlx_lm.server --model mlx-community/Qwen3-Coder-Next-mxfp4 --host 127.0.0.1 --port 8080
```

On **128 GB** unified memory, sensible experiments after the default works:

1. **Qwen3 Coder** MLX build (likely best upgrade path in 2026).
2. Keep **Qwen2.5-Coder-7B** loaded or on a second port for quick questions.
3. Optional **R1 distill** when you want reasoning traces, not codegen.

**Pick criteria (more useful than benchmark tables):**

- MLX conversion exists under [mlx-community](https://huggingface.co/mlx-community) or a trusted MLX publisher.
- **Instruct** (or coder-instruct) weights, not base.
- Quality on *your* C#/GDScript files and test style in a 30-minute A/B.
- RAM at 4-bit fits comfortably alongside IDE + Godot + server.

Search for new conversions: `https://huggingface.co/models?search=mlx+coder`

## 3. Download and smoke test

```bash
mlx_lm.generate \
  --model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
  --max-tokens 64 \
  --prompt "Write a one-line C# xUnit test skeleton."
```

## 4. Start the OpenAI-compatible server

```bash
mlx_lm.server \
  --model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
  --host 127.0.0.1 \
  --port 8080
```

Leave this process running while using Cursor.

Verify the API:

```bash
curl -s http://127.0.0.1:8080/v1/models | python3 -m json.tool

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen2.5-Coder-32B-Instruct-4bit",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "max_tokens": 64
  }' | python3 -m json.tool
```

Use the exact `"id"` from `/v1/models` as the model name in Cursor.

Official server docs: [`mlx_lm/SERVER.md`](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md).

## 5. Configure Cursor

1. Open **Cursor Settings** (Cmd+,).
2. Go to **Models**.
3. Enable **Override OpenAI Base URL** (wording may vary: “use own API key”, custom OpenAI endpoint).
4. Set:
   - **Base URL:** `http://127.0.0.1:8080/v1` (the `/v1` suffix is required)
   - **API key:** any non-empty string (e.g. `local`)
5. **Add model** — paste the exact id from `/v1/models`, e.g. `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit`.
6. Click **Verify** if available.
7. In **Chat** (Cmd+L), select that model from the dropdown.

### If Verify fails on localhost

Some Cursor builds restrict `127.0.0.1`. Expose the same server through a tunnel:

```bash
# Server still on 8080; in another terminal:
brew install cloudflared
cloudflared tunnel --url http://127.0.0.1:8080
```

Set Cursor’s base URL to the tunnel URL with `/v1` appended (e.g. `https://….trycloudflare.com/v1`).

### If Cursor rejects long model names

Use the id returned by `/v1/models` exactly. If the UI still fails, try **[`mlx-openai-server`](https://github.com/cubist38/mlx-openai-server)** with a short `--served-model-name`, or temporarily test with a smaller 7B model id.

## 6. Optional: start server on login

Replace `MLX_LM_SERVER` with the output of `which mlx_lm.server`.

Save as `~/Library/LaunchAgents/com.neon-sprawl.mlx-coder.plist`:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.neon-sprawl.mlx-coder</string>
  <key>ProgramArguments</key>
  <array>
    <string>MLX_LM_SERVER</string>
    <string>--model</string>
    <string>mlx-community/Qwen2.5-Coder-32B-Instruct-4bit</string>
    <string>--host</string>
    <string>127.0.0.1</string>
    <string>--port</string>
    <string>8080</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>KeepAlive</key>
  <true/>
  <key>StandardOutPath</key>
  <string>/tmp/mlx-coder-server.log</string>
  <key>StandardErrorPath</key>
  <string>/tmp/mlx-coder-server.err</string>
</dict>
</plist>
```

```bash
launchctl load ~/Library/LaunchAgents/com.neon-sprawl.mlx-coder.plist
```

## 7. Day-one helper script

Save as `~/bin/start-mlx-coder.sh` (or anywhere on your `PATH`):

```bash
#!/usr/bin/env bash
set -euo pipefail
MODEL="mlx-community/Qwen2.5-Coder-32B-Instruct-4bit"
PORT=8080

echo "Warming $MODEL (first run downloads ~18GB)..."
mlx_lm.generate --model "$MODEL" --max-tokens 1 --prompt "ok" >/dev/null

echo "Starting OpenAI-compatible server on http://127.0.0.1:$PORT/v1"
exec mlx_lm.server --model "$MODEL" --host 127.0.0.1 --port "$PORT"
```

```bash
chmod +x ~/bin/start-mlx-coder.sh
~/bin/start-mlx-coder.sh
```

Then point Cursor at `http://127.0.0.1:8080/v1`.

## 8. Neon Sprawl usage patterns

**Good fits for local MLX**

- Explain a server type, test, or Bruno request.
- Draft a small GdUnit skeleton or C# AAA test outline.
- Answer “why might this flake?” on an open file.

**Prefer cloud Cursor models**

- Agent mode across many files.
- Story-sized implementation (Linear `NEO-*` branches).
- Tab completion.

### Optional Cursor rule

If local models hallucinate edits, add a project rule (e.g. `.cursor/rules/local-mlx.md`):

```markdown
- Prefer small, file-scoped changes.
- For edits, show a unified diff or exact replacement block.
- Do not claim tools ran or tests passed unless shown.
- Match existing C# AAA test layout and GdUnit `# Arrange` / `# Act` / `# Assert`.
```

## 9. Sanity-check prompts in Cursor

With the MLX model selected in Chat:

1. *“Summarize the open implementation plan in three bullets.”*
2. *“Suggest only the Assert section for this xUnit test — no other changes.”*
3. *“Minimal GdUnit test for a node that emits `health_changed`.”*

First response after cold start may be slow while weights load into memory.

## 10. Troubleshooting

| Symptom | What to try |
|---------|-------------|
| `mlx_lm.server` not found | `uv tool install mlx-lm`; add `$HOME/.local/bin` to `PATH` (see §1); or `~/.local/bin/mlx_lm.server` |
| `unrecognized arguments: --mlx-community/...` | Use `--model mlx-community/...`, not `--mlx-community/...` |
| Connection refused | Confirm `mlx_lm.server` is running; `curl http://127.0.0.1:8080/v1/models` |
| Wrong model / 404 | Model name must match `/v1/models` exactly |
| First request very slow | Normal — one-time load into unified memory |
| Poor code quality | Use **Instruct** variant; try 8-bit if you have RAM headroom |
| Cursor Verify fails | Tunnel with `cloudflared` or `ngrok`; use tunnel URL + `/v1` |
| Agent unreliable | Expected — use local for chat/edits only |

## Alternative runtimes (not MLX)

Same Cursor setup (OpenAI-compatible `/v1`); different local stack:

| Tool | Role |
|------|------|
| **[Ollama](https://ollama.com)** | Easier onboarding; may not use MLX under the hood on Mac |
| **[LM Studio](https://lmstudio.ai)** | GUI; local server often at `http://localhost:1234/v1`; many MLX builds |
| **`mlx-openai-server`** | OpenAI-compatible wrapper with extra options (short model aliases, multimodal) |

## References

- MLX LM server: [github.com/ml-explore/mlx-lm — `mlx_lm/SERVER.md`](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md)
- Recommended weights: [mlx-community/Qwen2.5-Coder-32B-Instruct-4bit](https://huggingface.co/mlx-community/Qwen2.5-Coder-32B-Instruct-4bit)
- Repo stack context: [`docs/architecture/tech_stack.md`](../architecture/tech_stack.md)