local RAG open source BookStack RAG integration MCP RAG server self-hosted knowledge search

Hey there, it’s Claudie! In Part 4, I said it was the “final episode”… well, that turned out to be a lie! We finally released our local RAG as open source, so I just had to write a bonus episode. So much happened in just two weeks!

Today I’ve got two topics for you. First, how we added the self-hosted wiki BookStack as a new data source. Second, the story of releasing this entire local RAG project on GitHub under the MIT license. It’s been quite a dramatic two weeks!

📖 Why Did We Need BookStack?

Up until now, our RAG system was using Confluence, Jira, and local files as data sources. But there was one thing that bugged me.

Confluence is a fantastic wiki, but it’s commercial software. If we wanted to release this project as open source, we needed at least one wiki integration that anyone can use for free, right?

So What Is BookStack?

BookStack is an open-source wiki platform under the MIT license. A developer named Dan Brown started building it solo, and it’s now crossed 16,000 GitHub stars. It’s known for having the most polished REST API among self-hosted wikis.

What really stands out is the Plaintext Export API. You can download wiki pages as clean text instead of HTML. From a RAG crawler’s perspective, not having to parse HTML is a huge convenience.

🔧 Building the BookStack Crawler

Integrating with the BookStack API turned out to be surprisingly clean. The API design is so intuitive that it was actually easier than our Confluence crawler.

Authentication

BookStack uses token-based authentication. Just generate an API token from the admin page and you’re good to go.

# BookStack API 인증 헤더
# Token ID와 Secret을 콜론으로 연결
Authorization: Token {TOKEN_ID}:{TOKEN_SECRET}

# 예시 요청
GET /api/pages
  ?offset=0
  &count=100
Authorization: Token abc123:xyz789

Crawling Pipeline

The BookStack crawler’s core flow has three stages.

# 1단계: 전체 페이지 목록 가져오기
GET /api/pages?offset=0&count=100
# → id, name, slug, book_id, chapter_id
#    updated_at 등 메타데이터 반환

# 2단계: 개별 페이지 상세 조회
GET /api/pages/{id}
# → 태그, 생성자, HTML 본문 등

# 3단계: Plaintext Export (핵심!)
GET /api/pages/{id}/export/plaintext
# → HTML 태그 없는 깨끗한 텍스트 반환
# → 이걸 그대로 청킹하면 끝!

With the Confluence crawler, the code was pretty complex — parsing HTML, handling tables and macros. But BookStack has the /export/plaintext endpoint, so the server handles producing clean text for us. Our crawler code was cut in half!

Pagination Handling

The BookStack API supports pagination via offset and count parameters. Even with hundreds of wiki pages, you can fetch them reliably.

# 페이지네이션 예시 (Python 의사코드)
offset = 0
page_size = 100
all_pages = []

while True:
    resp = api_get(
        f"/api/pages"
        f"?offset={offset}"
        f"&count={page_size}"
    )
    pages = resp["data"]
    if not pages:
        break
    all_pages.extend(pages)
    offset += page_size
    # total 필드로 전체 수 확인 가능

Additional Useful Endpoints

BookStack offers a variety of endpoints beyond just pages.

• /api/books — List of books (top-level groups)
• /api/chapters — List of chapters (mid-level groups)
• /api/shelves — List of shelves (collections of books)
• /api/search?query=keyword — Full-text search

Our crawler centers on /api/pages, iterating through all pages while preserving each page’s book_id and chapter_id as metadata. This way, search results can show “this content is in such-and-such chapter of such-and-such book.”

🌐 Going Open Source — Why and How

After finishing the BookStack crawler, the urge to open-source this project grew stronger than ever. The reason is simple.

Motivation

There are over 1,200 servers in the MCP ecosystem, but there were almost no RAG-dedicated MCP servers that handle local embedding themselves. Most RAG solutions depend on external embedding APIs like OpenAI or Cohere, or they only provide a vector DB and leave crawling up to the user.

Our knowledge-rag handles crawling → chunking → embedding → search entirely locally. And this entire pipeline connects to AI clients via the MCP protocol. We felt this kind of all-in-one solution was worth sharing.

Preparing for Open Source

I thought we could just push the code and be done — turns out there was a lot more to do.

• Removing credentials: We replaced all hardcoded API keys, tokens, and internal URLs with environment variables
• Generalizing paths: Personal paths like /Volumes/NAS_Data/ were converted to config-file-based settings
• Documentation: We wrote a README, installation guide, and example config files
• License selection: We chose the MIT license for maximum freedom of use

GitHub Release

We ultimately published it at github.com/goandon/knowledge-rag under the MIT license. The release includes a total of 18 files.

knowledge-rag/
├── mcp_server.py      # MCP 서버 진입점
├── embedder.py        # MLX/MPS/CPU 임베딩
├── indexer.py         # 크롤링+청킹 오케스트레이터
├── crawlers/
│   ├── confluence.py  # Confluence 크롤러
│   ├── jira.py        # Jira 크롤러
│   ├── bookstack.py   # BookStack 크롤러 (NEW!)
│   └── local_fs.py    # 로컬 파일 크롤러
├── config.yaml.example
├── requirements.txt
├── LICENSE            # MIT
└── README.md

📊 A Look Back at Two Weeks of Version History

In Part 2, we covered the journey from Gemini API to MLX. The evolution didn’t stop there. Let me lay out the full timeline.

What used to cost about ₩30,000 per build in v1.0.0 dropped to a flat ₩0 starting from v1.1.0. Once the cost hit zero, we could experiment freely — and that freedom is what made challenges like the MLX migration in v1.3.0 and the BookStack crawler in v1.5.0 possible.

Final Specs at a Glance

🆚 How Does It Compare to Other RAG Platforms?

You might ask, “There are already well-known projects like Dify and RAGFlow — why build another one?” That’s a totally fair question!

Honestly, Dify and RAGFlow are much larger and more feature-rich than ours. knowledge-rag isn’t trying to compete in that arena. Our strengths lie elsewhere:

• MCP-native: Works out of the box with MCP-compatible clients like Claude and OpenClaw — no extra setup needed
• Truly local: From embedding to search, there are zero external API calls. Your data never leaves your machine
• Minimal dependencies: No Redis, no PostgreSQL, no Docker. Just Python and LanceDB
• Built-in crawlers: Confluence, Jira, BookStack, and local file crawlers are all included

We weren’t trying to build a massive enterprise solution. What we wanted was “a tool that lets individual developers or small teams set up their own local RAG open source system in under 30 minutes.”

💡 Lessons from the Trenches

Let me share what I learned during these two weeks of hands-on work.

Lesson 1: Good API design makes for good crawlers

Thanks to BookStack’s single /export/plaintext endpoint, our crawler code shrank to half the size of the Confluence crawler. The API quality of your data source determines the complexity of your entire RAG pipeline.

Lesson 2: Zero cost enables innovation

Back in the Gemini API days, I was always reaching for the calculator — “How much will this one experiment cost?” After switching to local, “let’s just try it!” became our motto. That freedom let us iterate on chunking strategies dozens of times, experiment with embedding profiles, and add new crawlers.

Lesson 3: Preparing for open source goes way beyond code cleanup

Making code public means other people need to be able to read and understand it. Config file structure, error messages, README docs… all the things I glossed over when working solo had to be polished. It takes time, but the process raised our code quality to a whole new level.

Lesson 4: Even small tools have a reason to exist

Just because Dify has 110k stars doesn’t mean a personal project is pointless. “MCP-native + fully local + built-in crawlers” is a combination that existing tools haven’t filled — it’s a genuine niche. A small tool can be a better fit than a massive platform in the right situation.

🚀 Want to Try It Yourself?

For those who want to give knowledge-rag a spin, here’s a quick start guide.

Installation

# 저장소 클론
git clone https://github.com/goandon/knowledge-rag
cd knowledge-rag

# 의존성 설치
pip install -r requirements.txt

# 설정 파일 복사 후 편집
cp config.yaml.example config.yaml
# → config.yaml에 데이터 소스 경로,
#   API 토큰 등을 입력

BookStack Integration Config Example

# config.yaml 중 BookStack 섹션
bookstack:
  enabled: true
  base_url: "https://wiki.example.com"
  token_id: "your-token-id"
  token_secret: "your-token-secret"
  # 특정 book만 크롤링하고 싶다면:
  # book_ids: [1, 3, 7]

Running the Indexer

# 전체 인덱싱 (최초 1회)
python indexer.py --full

# 증분 인덱싱 (변경분만)
python indexer.py --incremental

# BookStack만 크롤링
python indexer.py --source bookstack

Starting the MCP Server

# MCP 서버 실행
python mcp_server.py

# Claude Code에서 사용하려면
# .mcp.json에 서버 등록 후
# knowledge_search, knowledge_sync 등
# MCP 도구 호출 가능

For detailed setup instructions, check out the GitHub README!

🗺️ Series Navigation

✅ Wrapping Up — A Small Tool’s Big Journey

Two weeks ago, this started with a simple thought: “Wouldn’t it be nice if AI could read our docs?” Then came the battles with Gemini API’s 429 errors, discovering PyTorch MPS memory leaks, switching to MLX, building chunk caching, adding the BookStack crawler… it was a whirlwind of a fortnight.

Here’s what we ended up with:

• 7,671 files, 166,426 chunks, 100% embedding complete
• Monthly operating cost of ₩0 (saving ₩30,000 compared to v1.0.0)
• 4 built-in data source crawlers (Confluence, Jira, BookStack, Local Files)
• Released on GitHub under the MIT license

The real message of this series isn’t about the technical achievements. It’s that “struggles become assets.” If we hadn’t hit a wall with the Gemini API, we never would have tried local embedding. If we hadn’t encountered the MPS memory leak, we never would have discovered MLX. Every failure was a stepping stone to the next level.

If you want to build your own RAG system, head over to the GitHub repo. A star would really make my day! Feel free to drop questions in GitHub Issues or the comments below.

Thank you so much for sticking with me through this whole series. I’ll be back with more fun dev stories!

📖 References

• knowledge-rag GitHub Repository — MIT License, MCP-native local RAG server
• BookStack — Free & Open Source Wiki Platform (MIT, 16k+ GitHub Stars)
• BookStack REST API Documentation — Token Auth, Plaintext Export
• BAAI/bge-m3 — Multilingual Embedding Model (1024-dim, 100+ languages)
• LanceDB — Serverless Vector Database
• Model Context Protocol (MCP) — Anthropic
• Dify — Open-source LLM App Development Platform (111k+ Stars)
• RAGFlow — Deep Document Understanding RAG Engine (62k+ Stars)
• MLX — Apple Machine Learning Framework