273 lines
8.8 KiB
Markdown
273 lines
8.8 KiB
Markdown
# Local MLX models with Cursor (macOS)
|
||
|
||
**Status:** Developer guide (not product architecture).
|
||
**Audience:** Neon Sprawl contributors on Apple Silicon who want **private, offline-capable** chat and inline edits in Cursor, backed by a local model.
|
||
|
||
Cursor does not talk to MLX directly. It needs an **OpenAI-compatible HTTP API** (`/v1/chat/completions`). This guide uses Apple’s **[MLX](https://github.com/ml-explore/mlx)** runtime via **[`mlx-lm`](https://github.com/ml-explore/mlx-lm)** and its built-in server.
|
||
|
||
## What works in Cursor
|
||
|
||
| Feature | Local MLX |
|
||
|---------|-----------|
|
||
| **Chat** (Cmd+L) | Yes |
|
||
| **Inline edit** (Cmd+K) | Usually yes |
|
||
| **Composer / Agent** | Limited — local models are weaker at multi-step tool use; prefer cloud models for large refactors |
|
||
| **Tab autocomplete** | No — Cursor tab completion uses separate proprietary models |
|
||
|
||
Use local MLX for **privacy, offline use, and cost**. Keep **cloud models** for Agent mode, tab completion, and multi-file story work.
|
||
|
||
## Architecture
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph cursor [Cursor IDE]
|
||
Chat[Chat Cmd+L]
|
||
Edit[Inline edit Cmd+K]
|
||
end
|
||
|
||
subgraph mac [Mac Apple Silicon]
|
||
API["mlx_lm.server :8080/v1"]
|
||
MLX[MLX runtime]
|
||
Model["Qwen2.5-Coder-32B-Instruct-4bit"]
|
||
end
|
||
|
||
Chat --> API
|
||
Edit --> API
|
||
API --> MLX --> Model
|
||
```
|
||
|
||
## Prerequisites
|
||
|
||
- **macOS** with Apple Silicon (MLX is not for Intel Macs).
|
||
- **Python 3.12+** (Homebrew or [uv](https://github.com/astral-sh/uv)).
|
||
- Enough disk for model weights (~18 GB for the recommended 32B 4-bit coder).
|
||
- **Cursor** with permission to override the OpenAI base URL (Settings → Models).
|
||
|
||
Rough RAM for the recommended model: **~18–22 GB** active during inference. A Mac with **48 GB+** unified memory is comfortable; **128 GB** can also run 8-bit variants for higher quality.
|
||
|
||
## 1. Install `mlx-lm`
|
||
|
||
**Option A — uv (isolated tool install):**
|
||
|
||
```bash
|
||
brew install uv
|
||
uv tool install mlx-lm
|
||
```
|
||
|
||
**Option B — venv + pip:**
|
||
|
||
```bash
|
||
brew install python@3.12
|
||
python3.12 -m venv ~/.local/venvs/mlx-lm
|
||
source ~/.local/venvs/mlx-lm/bin/activate
|
||
pip install mlx-lm
|
||
```
|
||
|
||
Verify:
|
||
|
||
```bash
|
||
mlx_lm.server --help
|
||
which mlx_lm.server # note path for LaunchAgent below
|
||
```
|
||
|
||
## 2. Choose a model
|
||
|
||
| Role | Hugging Face repo id | Approx. RAM |
|
||
|------|----------------------|-------------|
|
||
| **Primary coder (recommended)** | `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit` | ~18–22 GB |
|
||
| **Higher quality (optional)** | `mlx-community/Qwen2.5-Coder-32B-Instruct-8bit` | ~32–35 GB |
|
||
| **Fast / small (optional)** | `mlx-community/Qwen2.5-Coder-7B-Instruct-4bit` | ~5 GB |
|
||
|
||
For Neon Sprawl (C# server tests, GDScript client, Bruno API collections), prefer the **Instruct** coder variant — not the non-instruct base weights.
|
||
|
||
Models download on first use into the Hugging Face cache.
|
||
|
||
## 3. Download and smoke test
|
||
|
||
```bash
|
||
mlx_lm.generate \
|
||
--model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
|
||
--max-tokens 64 \
|
||
--prompt "Write a one-line C# xUnit test skeleton."
|
||
```
|
||
|
||
## 4. Start the OpenAI-compatible server
|
||
|
||
```bash
|
||
mlx_lm.server \
|
||
--model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
|
||
--host 127.0.0.1 \
|
||
--port 8080
|
||
```
|
||
|
||
Leave this process running while using Cursor.
|
||
|
||
Verify the API:
|
||
|
||
```bash
|
||
curl -s http://127.0.0.1:8080/v1/models | python3 -m json.tool
|
||
|
||
curl -s http://127.0.0.1:8080/v1/chat/completions \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"model": "mlx-community/Qwen2.5-Coder-32B-Instruct-4bit",
|
||
"messages": [{"role": "user", "content": "Say hello in one sentence."}],
|
||
"max_tokens": 64
|
||
}' | python3 -m json.tool
|
||
```
|
||
|
||
Use the exact `"id"` from `/v1/models` as the model name in Cursor.
|
||
|
||
Official server docs: [`mlx_lm/SERVER.md`](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md).
|
||
|
||
## 5. Configure Cursor
|
||
|
||
1. Open **Cursor Settings** (Cmd+,).
|
||
2. Go to **Models**.
|
||
3. Enable **Override OpenAI Base URL** (wording may vary: “use own API key”, custom OpenAI endpoint).
|
||
4. Set:
|
||
- **Base URL:** `http://127.0.0.1:8080/v1` (the `/v1` suffix is required)
|
||
- **API key:** any non-empty string (e.g. `local`)
|
||
5. **Add model** — paste the exact id from `/v1/models`, e.g. `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit`.
|
||
6. Click **Verify** if available.
|
||
7. In **Chat** (Cmd+L), select that model from the dropdown.
|
||
|
||
### If Verify fails on localhost
|
||
|
||
Some Cursor builds restrict `127.0.0.1`. Expose the same server through a tunnel:
|
||
|
||
```bash
|
||
# Server still on 8080; in another terminal:
|
||
brew install cloudflared
|
||
cloudflared tunnel --url http://127.0.0.1:8080
|
||
```
|
||
|
||
Set Cursor’s base URL to the tunnel URL with `/v1` appended (e.g. `https://….trycloudflare.com/v1`).
|
||
|
||
### If Cursor rejects long model names
|
||
|
||
Use the id returned by `/v1/models` exactly. If the UI still fails, try **[`mlx-openai-server`](https://github.com/cubist38/mlx-openai-server)** with a short `--served-model-name`, or temporarily test with a smaller 7B model id.
|
||
|
||
## 6. Optional: start server on login
|
||
|
||
Replace `MLX_LM_SERVER` with the output of `which mlx_lm.server`.
|
||
|
||
Save as `~/Library/LaunchAgents/com.neon-sprawl.mlx-coder.plist`:
|
||
|
||
```xml
|
||
<?xml version="1.0" encoding="UTF-8"?>
|
||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||
<plist version="1.0">
|
||
<dict>
|
||
<key>Label</key>
|
||
<string>com.neon-sprawl.mlx-coder</string>
|
||
<key>ProgramArguments</key>
|
||
<array>
|
||
<string>MLX_LM_SERVER</string>
|
||
<string>--model</string>
|
||
<string>mlx-community/Qwen2.5-Coder-32B-Instruct-4bit</string>
|
||
<string>--host</string>
|
||
<string>127.0.0.1</string>
|
||
<string>--port</string>
|
||
<string>8080</string>
|
||
</array>
|
||
<key>RunAtLoad</key>
|
||
<true/>
|
||
<key>KeepAlive</key>
|
||
<true/>
|
||
<key>StandardOutPath</key>
|
||
<string>/tmp/mlx-coder-server.log</string>
|
||
<key>StandardErrorPath</key>
|
||
<string>/tmp/mlx-coder-server.err</string>
|
||
</dict>
|
||
</plist>
|
||
```
|
||
|
||
```bash
|
||
launchctl load ~/Library/LaunchAgents/com.neon-sprawl.mlx-coder.plist
|
||
```
|
||
|
||
## 7. Day-one helper script
|
||
|
||
Save as `~/bin/start-mlx-coder.sh` (or anywhere on your `PATH`):
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
set -euo pipefail
|
||
MODEL="mlx-community/Qwen2.5-Coder-32B-Instruct-4bit"
|
||
PORT=8080
|
||
|
||
echo "Warming $MODEL (first run downloads ~18GB)..."
|
||
mlx_lm.generate --model "$MODEL" --max-tokens 1 --prompt "ok" >/dev/null
|
||
|
||
echo "Starting OpenAI-compatible server on http://127.0.0.1:$PORT/v1"
|
||
exec mlx_lm.server --model "$MODEL" --host 127.0.0.1 --port "$PORT"
|
||
```
|
||
|
||
```bash
|
||
chmod +x ~/bin/start-mlx-coder.sh
|
||
~/bin/start-mlx-coder.sh
|
||
```
|
||
|
||
Then point Cursor at `http://127.0.0.1:8080/v1`.
|
||
|
||
## 8. Neon Sprawl usage patterns
|
||
|
||
**Good fits for local MLX**
|
||
|
||
- Explain a server type, test, or Bruno request.
|
||
- Draft a small GdUnit skeleton or C# AAA test outline.
|
||
- Answer “why might this flake?” on an open file.
|
||
|
||
**Prefer cloud Cursor models**
|
||
|
||
- Agent mode across many files.
|
||
- Story-sized implementation (Linear `NEO-*` branches).
|
||
- Tab completion.
|
||
|
||
### Optional Cursor rule
|
||
|
||
If local models hallucinate edits, add a project rule (e.g. `.cursor/rules/local-mlx.md`):
|
||
|
||
```markdown
|
||
- Prefer small, file-scoped changes.
|
||
- For edits, show a unified diff or exact replacement block.
|
||
- Do not claim tools ran or tests passed unless shown.
|
||
- Match existing C# AAA test layout and GdUnit `# Arrange` / `# Act` / `# Assert`.
|
||
```
|
||
|
||
## 9. Sanity-check prompts in Cursor
|
||
|
||
With the MLX model selected in Chat:
|
||
|
||
1. *“Summarize the open implementation plan in three bullets.”*
|
||
2. *“Suggest only the Assert section for this xUnit test — no other changes.”*
|
||
3. *“Minimal GdUnit test for a node that emits `health_changed`.”*
|
||
|
||
First response after cold start may be slow while weights load into memory.
|
||
|
||
## 10. Troubleshooting
|
||
|
||
| Symptom | What to try |
|
||
|---------|-------------|
|
||
| Connection refused | Confirm `mlx_lm.server` is running; `curl http://127.0.0.1:8080/v1/models` |
|
||
| Wrong model / 404 | Model name must match `/v1/models` exactly |
|
||
| First request very slow | Normal — one-time load into unified memory |
|
||
| Poor code quality | Use **Instruct** variant; try 8-bit if you have RAM headroom |
|
||
| Cursor Verify fails | Tunnel with `cloudflared` or `ngrok`; use tunnel URL + `/v1` |
|
||
| Agent unreliable | Expected — use local for chat/edits only |
|
||
|
||
## Alternatives
|
||
|
||
| Tool | Role |
|
||
|------|------|
|
||
| **[Ollama](https://ollama.com)** | Easier onboarding; may not use MLX under the hood on Mac |
|
||
| **[LM Studio](https://lmstudio.ai)** | GUI; local server often at `http://localhost:1234/v1` |
|
||
| **`mlx-openai-server`** | OpenAI-compatible wrapper with extra options (short model aliases, multimodal) |
|
||
|
||
## References
|
||
|
||
- MLX LM server: [github.com/ml-explore/mlx-lm — `mlx_lm/SERVER.md`](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md)
|
||
- Recommended weights: [mlx-community/Qwen2.5-Coder-32B-Instruct-4bit](https://huggingface.co/mlx-community/Qwen2.5-Coder-32B-Instruct-4bit)
|
||
- Repo stack context: [`docs/architecture/tech_stack.md`](../architecture/tech_stack.md)
|