don
/
neon-sprawl
mirror of https://github.com/ViPro-Technologies/neon-sprawl
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into NEO-92

pull/130/head
VinPropane 2026-05-27 21:47:00 -04:00 committed by GitHub
parent 7682fac46a 56efafb12a
commit 83c6a4f1b6
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 274 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -12,6 +12,8 @@
Full rationale and constraints: [`docs/architecture/tech_stack.md`](docs/architecture/tech_stack.md).
Optional **local LLM in Cursor** (MLX on Apple Silicon): [`docs/dev/local-mlx-cursor.md`](docs/dev/local-mlx-cursor.md).
## Decomposition
Epic-level breakdown: [`docs/decomposition/README.md`](docs/decomposition/README.md).

272
docs/dev/local-mlx-cursor.md 100644
View File

@ -0,0 +1,272 @@
# Local MLX models with Cursor (macOS)
**Status:** Developer guide (not product architecture).
**Audience:** Neon Sprawl contributors on Apple Silicon who want **private, offline-capable** chat and inline edits in Cursor, backed by a local model.
Cursor does not talk to MLX directly. It needs an **OpenAI-compatible HTTP API** (`/v1/chat/completions`). This guide uses Apple’s **[MLX](https://github.com/ml-explore/mlx)** runtime via **[`mlx-lm`](https://github.com/ml-explore/mlx-lm)** and its built-in server.
## What works in Cursor
| Feature | Local MLX |
|---------|-----------|
| **Chat** (Cmd+L) | Yes |
| **Inline edit** (Cmd+K) | Usually yes |
| **Composer / Agent** | Limited — local models are weaker at multi-step tool use; prefer cloud models for large refactors |
| **Tab autocomplete** | No — Cursor tab completion uses separate proprietary models |
Use local MLX for **privacy, offline use, and cost**. Keep **cloud models** for Agent mode, tab completion, and multi-file story work.
## Architecture
```mermaid
flowchart LR
subgraph cursor [Cursor IDE]
Chat[Chat Cmd+L]
Edit[Inline edit Cmd+K]
end
subgraph mac [Mac Apple Silicon]
API["mlx_lm.server :8080/v1"]
MLX[MLX runtime]
Model["Qwen2.5-Coder-32B-Instruct-4bit"]
end
Chat --> API
Edit --> API
API --> MLX --> Model
```
## Prerequisites
- **macOS** with Apple Silicon (MLX is not for Intel Macs).
- **Python 3.12+** (Homebrew or [uv](https://github.com/astral-sh/uv)).
- Enough disk for model weights (~18 GB for the recommended 32B 4-bit coder).
- **Cursor** with permission to override the OpenAI base URL (Settings → Models).
Rough RAM for the recommended model: **~18–22 GB** active during inference. A Mac with **48 GB+** unified memory is comfortable; **128 GB** can also run 8-bit variants for higher quality.
## 1. Install `mlx-lm`
**Option A — uv (isolated tool install):**
```bash
brew install uv
uv tool install mlx-lm
```
**Option B — venv + pip:**
```bash
brew install python@3.12
python3.12 -m venv ~/.local/venvs/mlx-lm
source ~/.local/venvs/mlx-lm/bin/activate
pip install mlx-lm
```
Verify:
```bash
mlx_lm.server --help
which mlx_lm.server # note path for LaunchAgent below
```
## 2. Choose a model
| Role | Hugging Face repo id | Approx. RAM |
|------|----------------------|-------------|
| **Primary coder (recommended)** | `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit` | ~18–22 GB |
| **Higher quality (optional)** | `mlx-community/Qwen2.5-Coder-32B-Instruct-8bit` | ~32–35 GB |
| **Fast / small (optional)** | `mlx-community/Qwen2.5-Coder-7B-Instruct-4bit` | ~5 GB |
For Neon Sprawl (C# server tests, GDScript client, Bruno API collections), prefer the **Instruct** coder variant — not the non-instruct base weights.
Models download on first use into the Hugging Face cache.
## 3. Download and smoke test
```bash
mlx_lm.generate \
--model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
--max-tokens 64 \
--prompt "Write a one-line C# xUnit test skeleton."
```
## 4. Start the OpenAI-compatible server
```bash
mlx_lm.server \
--model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
--host 127.0.0.1 \
--port 8080
```
Leave this process running while using Cursor.
Verify the API:
```bash
curl -s http://127.0.0.1:8080/v1/models | python3 -m json.tool
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/Qwen2.5-Coder-32B-Instruct-4bit",
"messages": [{"role": "user", "content": "Say hello in one sentence."}],
"max_tokens": 64
}' | python3 -m json.tool
```
Use the exact `"id"` from `/v1/models` as the model name in Cursor.
Official server docs: [`mlx_lm/SERVER.md`](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md).
## 5. Configure Cursor
1. Open **Cursor Settings** (Cmd+,).
2. Go to **Models**.
3. Enable **Override OpenAI Base URL** (wording may vary: “use own API key”, custom OpenAI endpoint).
4. Set:
- **Base URL:** `http://127.0.0.1:8080/v1` (the `/v1` suffix is required)
- **API key:** any non-empty string (e.g. `local`)
5. **Add model** — paste the exact id from `/v1/models`, e.g. `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit`.
6. Click **Verify** if available.
7. In **Chat** (Cmd+L), select that model from the dropdown.
### If Verify fails on localhost
Some Cursor builds restrict `127.0.0.1`. Expose the same server through a tunnel:
```bash
# Server still on 8080; in another terminal:
brew install cloudflared
cloudflared tunnel --url http://127.0.0.1:8080
```
Set Cursor’s base URL to the tunnel URL with `/v1` appended (e.g. `https://….trycloudflare.com/v1`).
### If Cursor rejects long model names
Use the id returned by `/v1/models` exactly. If the UI still fails, try **[`mlx-openai-server`](https://github.com/cubist38/mlx-openai-server)** with a short `--served-model-name`, or temporarily test with a smaller 7B model id.
## 6. Optional: start server on login
Replace `MLX_LM_SERVER` with the output of `which mlx_lm.server`.
Save as `~/Library/LaunchAgents/com.neon-sprawl.mlx-coder.plist`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.neon-sprawl.mlx-coder</string>
<key>ProgramArguments</key>
<array>
<string>MLX_LM_SERVER</string>
<string>--model</string>
<string>mlx-community/Qwen2.5-Coder-32B-Instruct-4bit</string>
<string>--host</string>
<string>127.0.0.1</string>
<string>--port</string>
<string>8080</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
<key>StandardOutPath</key>
<string>/tmp/mlx-coder-server.log</string>
<key>StandardErrorPath</key>
<string>/tmp/mlx-coder-server.err</string>
</dict>
</plist>
```
```bash
launchctl load ~/Library/LaunchAgents/com.neon-sprawl.mlx-coder.plist
```
## 7. Day-one helper script
Save as `~/bin/start-mlx-coder.sh` (or anywhere on your `PATH`):
```bash
#!/usr/bin/env bash
set -euo pipefail
MODEL="mlx-community/Qwen2.5-Coder-32B-Instruct-4bit"
PORT=8080
echo "Warming $MODEL (first run downloads ~18GB)..."
mlx_lm.generate --model "$MODEL" --max-tokens 1 --prompt "ok" >/dev/null
echo "Starting OpenAI-compatible server on http://127.0.0.1:$PORT/v1"
exec mlx_lm.server --model "$MODEL" --host 127.0.0.1 --port "$PORT"
```
```bash
chmod +x ~/bin/start-mlx-coder.sh
~/bin/start-mlx-coder.sh
```
Then point Cursor at `http://127.0.0.1:8080/v1`.
## 8. Neon Sprawl usage patterns
**Good fits for local MLX**
- Explain a server type, test, or Bruno request.
- Draft a small GdUnit skeleton or C# AAA test outline.
- Answer “why might this flake?” on an open file.
**Prefer cloud Cursor models**
- Agent mode across many files.
- Story-sized implementation (Linear `NEO-*` branches).
- Tab completion.
### Optional Cursor rule
If local models hallucinate edits, add a project rule (e.g. `.cursor/rules/local-mlx.md`):
```markdown
- Prefer small, file-scoped changes.
- For edits, show a unified diff or exact replacement block.
- Do not claim tools ran or tests passed unless shown.
- Match existing C# AAA test layout and GdUnit `# Arrange` / `# Act` / `# Assert`.
```
## 9. Sanity-check prompts in Cursor
With the MLX model selected in Chat:
1. *“Summarize the open implementation plan in three bullets.”*
2. *“Suggest only the Assert section for this xUnit test — no other changes.”*
3. *“Minimal GdUnit test for a node that emits `health_changed`.”*
First response after cold start may be slow while weights load into memory.
## 10. Troubleshooting
| Symptom | What to try |
|---------|-------------|
| Connection refused | Confirm `mlx_lm.server` is running; `curl http://127.0.0.1:8080/v1/models` |
| Wrong model / 404 | Model name must match `/v1/models` exactly |
| First request very slow | Normal — one-time load into unified memory |
| Poor code quality | Use **Instruct** variant; try 8-bit if you have RAM headroom |
| Cursor Verify fails | Tunnel with `cloudflared` or `ngrok`; use tunnel URL + `/v1` |
| Agent unreliable | Expected — use local for chat/edits only |
## Alternatives
| Tool | Role |
|------|------|
| **[Ollama](https://ollama.com)** | Easier onboarding; may not use MLX under the hood on Mac |
| **[LM Studio](https://lmstudio.ai)** | GUI; local server often at `http://localhost:1234/v1` |
| **`mlx-openai-server`** | OpenAI-compatible wrapper with extra options (short model aliases, multimodal) |
## References
- MLX LM server: [github.com/ml-explore/mlx-lm — `mlx_lm/SERVER.md`](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md)
- Recommended weights: [mlx-community/Qwen2.5-Coder-32B-Instruct-4bit](https://huggingface.co/mlx-community/Qwen2.5-Coder-32B-Instruct-4bit)
- Repo stack context: [`docs/architecture/tech_stack.md`](../architecture/tech_stack.md)