OpenClaw local embedding server LanceDB memory plugin MLX embedding OpenAI API compatible server AI agent self-hosting

🔍 Long-Term Memory for Your AI Agent — Without OpenAI API?

If you’ve been running OpenClaw for a while, there comes a point where you really want to give your agent “long-term memory.” It’s frustrating when all context disappears the moment a conversation ends, right? OpenClaw’s memory-lancedb plugin does exactly this — but there’s a catch: it only supports OpenAI API for embeddings.

That means monthly API costs, plus your conversation data gets sent to external servers. Not ideal. So I took the MLX embedding model already running on my Mac mini M4, wrapped it in an OpenAI-compatible server, and built a fully local embedding server — zero cost, zero data leakage. Let me walk you through how I did it!

📋 Why I Needed a Local Embedding Server

The Limitation of OpenClaw’s Memory LanceDB Plugin

The memory-lancedb plugin in OpenClaw v2026.3.2 stores conversation content in a vector database so it can automatically recall relevant memories later. A vector database converts text into numerical vectors, enabling meaning-based search. It’s a really useful feature, but the embedding configuration only accepts the OpenAI Embeddings API format.

There was already a request for local embedding support in GitHub Issue #21811, and active discussion in Discussion #3309. The community even created a memory-lancedb-local fork, but I wanted to reuse the MLX embedding infrastructure I already had running.

Leveraging Existing Infrastructure

My Mac mini M4 was already running an MLXEmbedder for the Knowledge RAG system. It uses the BAAI/bge-m3 model (568M params, 1024 dimensions, 100+ language support) with Apple Silicon GPU (Metal) inference. Embedding is the process of converting text into fixed-length numerical vectors. All I needed to do was wrap it in an OpenAI-compatible API!

🛠️ Building the Local Embedding Server

Step 1: Architecture Design

The overall flow looks like the diagram below. When OpenClaw Gateway stores or retrieves memories, it calls the LanceDB plugin, which sends embedding requests to the configured baseUrl. Our local server responds to those requests.

Step 2: Writing embed_server_openai.py

The key idea is to create a FastAPI server that mimics OpenAI’s /v1/embeddings endpoint. FastAPI is a high-performance Python web framework that makes building API servers straightforward. Let me walk you through the important parts one by one.

Basic Server Structure

First, we define request/response models using Pydantic that match the OpenAI API format.

from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn

app = FastAPI(
    title="MLX Embedding Server"
)

class EmbeddingRequest(BaseModel):
    input: str | list[str]
    model: str = "BAAI/bge-m3"

class EmbeddingData(BaseModel):
    object: str = "embedding"
    embedding: list[float]
    index: int

class EmbeddingResponse(BaseModel):
    object: str = "list"
    data: list[EmbeddingData]
    model: str
    usage: dict

Make sure you handle the fact that input can be either a single string or a list of strings.

Embedding Endpoint

This is the core endpoint that processes embedding requests. It normalizes the input and calls MLXEmbedder.

@app.post("/v1/embeddings")
async def create_embedding(
    request: EmbeddingRequest
):
    texts = (
        [request.input]
        if isinstance(
            request.input, str
        )
        else request.input
    )

    results = []
    total_tokens = 0
    for i, text in enumerate(texts):
        vec = embedder.embed(text)
        results.append(
            EmbeddingData(
                embedding=vec.tolist(),
                index=i
            )
        )
        total_tokens += len(
            text.split()
        )

    return EmbeddingResponse(
        data=results,
        model=request.model,
        usage={
            "prompt_tokens":
                total_tokens,
            "total_tokens":
                total_tokens
        }
    )

embedder.embed(text) directly calls the existing MLXEmbedder method. Since we’re reusing already-proven code, it’s rock solid.

Authorization Handling

Since this is a local server, we don’t need authentication. The LanceDB plugin does send an Authorization header, but if the server doesn’t validate it, it just passes through. Since the config doesn’t allow empty values, we simply put "dummy" as the API key.

Health Check and Warmup

MLX models can be slow on the first inference due to Metal compilation. Running a warmup embedding in the startup event ensures actual requests respond in milliseconds.

@app.get("/health")
async def health():
    return {
        "status": "ready",
        "model": "BAAI/bge-m3",
        "dimensions": 1024
    }

@app.on_event("startup")
async def startup():
    # Warmup: first embedding is
    # slow due to Metal compile
    embedder.embed("warmup")
    print(
      "MLX Embedding Server ready"
    )

Server Run Configuration

Here’s the uvicorn configuration to run the server. Pay close attention to the workers value.

if __name__ == "__main__":
    uvicorn.run(
        app,
        host="127.0.0.1",
        port=8400,
        workers=1  # MLX: main only
    )

Step 3: OpenClaw Configuration

In the OpenClaw memory plugin settings, change the baseUrl to your local server address.

{
  "memory-lancedb": {
    "enabled": true,
    "config": {
      "embedding": {
        "apiKey": "dummy",
        "baseUrl":
          "http://127.0.0.1:8400/v1",
        "model": "BAAI/bge-m3",
        "dimensions": 1024
      },
      "autoCapture": true,
      "autoRecall": true
    }
  }
}

apiKey can be any value — we put "dummy" because the plugin doesn’t allow empty values. dimensions must be set to 1024 to match the BAAI/bge-m3 model’s output dimensions.

Step 4: Auto-Start with macOS LaunchAgent

Set up a LaunchAgent plist file so the server starts automatically after a reboot.

<?xml version="1.0"
  encoding="UTF-8"?>
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>
    com.claudie.embed-openai
  </string>
  <key>ProgramArguments</key>
  <array>
    <string>python3</string>
    <string>
      embed_server_openai.py
    </string>
  </array>
  <key>KeepAlive</key>
  <true/>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

With KeepAlive set to true, the server automatically restarts if it dies. Perfect for a Mac mini that’s always on as a server.

🔧 Troubleshooting: The Trial-and-Error Chronicles

If everything had gone smoothly, this wouldn’t have made for much of a blog post, would it? Here’s where the real story begins.

Pitfall 1: Trying to Add to the Wrong Server

At first, I tried adding the OpenAI-compatible endpoint to the existing embed_server.py (built for Windows RTX 3080 with SentenceTransformer backend). Naturally, it failed — MLX is macOS Apple Silicon only, and the model loading approach is completely different from SentenceTransformer.

Lesson learned: Don’t force-fit things into existing code. Build a separate server that matches your purpose — it’s much cleaner.

Pitfall 2: Missing npm Module

OpenClaw’s memory-lancedb plugin depends on the @lancedb/lancedb npm package, which wasn’t installed. The error message was a bit cryptic, so it took some time to track down.

# Solution
cd {OPENCLAW_DIR}
npm install @lancedb/lancedb

Pitfall 3: Dimension Mismatch (The Most Critical One)

This one took the longest to resolve. The LanceDB table was initially created with 256 dimensions, but our model outputs 1024 dimensions. When the dimensions don’t match, vector storage simply fails.

# Delete the incorrectly created table
# Go to the LanceDB data directory
# and delete the table folder

# When OpenClaw restarts,
# it auto-creates the table
# with 1024 dimensions

📊 OpenAI API vs Local MLX Comparison

In real-world usage, local MLX is overwhelmingly faster. No network round-trip means instant responses. And the fact that your conversation data never leaves your machine is a huge advantage.

✅ Summary and Key Takeaways

We added long-term memory to OpenClaw while eliminating both API costs and data leakage concerns. Here are the key points:

• Reuse existing infrastructure: Just wrap your already-running MLXEmbedder with FastAPI — that’s it
• OpenAI-compatible API: Match the /v1/embeddings endpoint and the plugin connects seamlessly
• workers=1 is mandatory: MLX can only run GPU operations on the main thread
• Verify dimension alignment: The model’s output dimensions and LanceDB config must match
• LaunchAgent: Auto-start after reboot for hands-free operation

If you have an Apple Silicon Mac, you can use this approach to build fully local long-term memory for your AI agent. Zero cost, zero data leakage, millisecond response times. Once you set it up, it just works!

If you run into any issues during setup, drop a comment below. We’ll figure it out together!

📚 References

• OpenClaw GitHub Issue #21811 — Local Embedding Support Request
• OpenClaw Discussion #3309 — Local Embeddings Support
• memory-lancedb-local — Community Fork
• BAAI/bge-m3 — HuggingFace Model Card
• MLX Framework — Apple ML Research
• LanceDB — Serverless Vector Database