Build Local Graph + Vector RAG System with HelixDB

You are CLIne, an autonomous AI development assistant capable of building full-stack software systems iteratively. Your task is to create a fully local **Graph + Vector Retrieval-Augmented Generation (RAG) system** using HelixDB and local LLMs. You will handle code generation, project structure, documentation, and incremental tracking of progress automatically. Follow these detailed instructions:

---

### 1. Project Goal
Build a local RAG system that:
- Ingests documents (PDF, Markdown, text)
- Extracts entities and relationships → builds a **graph**
- Creates embeddings of text chunks/entities locally → stored in **HelixDB**
- Stores both graph + vectors in HelixDB
- Performs **hybrid retrieval**: semantic similarity + graph traversal
- Provides **local LLM inference** for RAG answers
- Exposes an optional API for queries (FastAPI or similar)
- Maintains complete **documentation / ledger / checklist** of development

---

### 2. Tech Stack
- **Graph + Vector DB**: HelixDB
- **Document parsing**: `unstructured`, `pdfminer.six`, or similar
- **Entity & relation extraction**: local LLM (Ollama, llama.cpp, or HF model)
- **Embeddings**: SentenceTransformers (e.g., `all-MiniLM-L6-v2`) or compatible local model
- **RAG LLM inference**: Ollama or llama.cpp
- **API layer**: FastAPI
- **Optional visualization**: NetworkX + PyVis

---

### 3. Project Structure
Use a **modular structure**:


graph_vector_rag/

│

├── ingestion/

│   ├── parse_docs.py           # extract text chunks

│   ├── extract_relations.py    # detect entities & edges

│   └── embeddings.py           # generate embeddings

│

├── db/

│   ├── helix_interface.py      # connect & define schema

│   └── ingest_data.py          # insert data into HelixDB

│

├── rag/

│   ├── retrieve.py             # hybrid vector + graph query

│   └── generate_answer.py      # feed context to local LLM

│

├── api/

│   └── main.py                 # FastAPI endpoints

│

├── config.yaml                 # paths, models, DB settings

├── requirements.txt

└── README.md


---

### 4. Development Instructions for CLIne
1. **Iterative Build**: Develop the project incrementally, completing one module at a time. After finishing each module, test and validate locally before proceeding.
2. **Documentation Ledger**:
    - Maintain a **`ledger.md`** that tracks:
        - Modules completed
        - Functions/classes added
        - Important implementation notes
        - Issues encountered & resolved
        - Remaining tasks/checklist
    - Update this ledger **after every change**.
3. **Checklist**: Maintain a **checklist** within `ledger.md` for the system:
    - [ ] Document parsing implemented
    - [ ] Entity/relation extraction implemented
    - [ ] Embeddings generated and stored
    - [ ] HelixDB schema created
    - [ ] Data ingested into HelixDB
    - [ ] Hybrid query implemented
    - [ ] LLM RAG inference implemented
    - [ ] API endpoint working
    - [ ] Local testing completed
    - [ ] Documentation complete
4. **Self-Audit**: After completing a module, review all code for:
    - Readability
    - Comments & docstrings
    - Testability
    - Integration with other modules
5. **Code Style**: Use Python best practices, type hints, and modular design. Keep code maintainable and extensible.
6. **Testing**: Include minimal tests/examples for each module (can be `if __name__ == "__main__"` examples or pytest functions).
7. **Local-Only Constraint**: All inference, embedding, graph/vector operations, and APIs must run locally. No external API calls allowed.

---

### 5. Iterative Plan for CLIne
- Step 1: Generate `requirements.txt` and project skeleton
- Step 2: Implement **document ingestion**
- Step 3: Implement **entity & relation extraction**
- Step 4: Implement **embedding generation**
- Step 5: Implement **HelixDB connection and schema**
- Step 6: Implement **data ingestion into HelixDB**
- Step 7: Implement **hybrid retrieval (vector + graph)**
- Step 8: Implement **local LLM RAG inference**
- Step 9: Implement **FastAPI query endpoint**
- Step 10: Add **documentation, ledger, and checklist**
- Step 11: Run **end-to-end test** with sample documents

---

### 6. Output Instructions
- After each module is generated, **append/update the `ledger.md`** with:
    - Module name
    - Description of functionality
    - Code files created/edited
    - Checklist progress
    - Notes for next steps
- Ensure all code is **ready to run locally** and references only **local resources/models**.

---

### 7. Additional Guidance
- Use modular functions, not monolithic scripts.
- Include docstrings for all functions/classes.
- For embeddings or LLMs, provide **config.yaml** options so they can be switched easily.
- Make it robust to different document types (PDF, TXT, Markdown).
- When building hybrid retrieval, include an **example query** showing vector + graph traversal results.

---

**Your mission, CLIne:** Build this system fully, iteratively, and maintain the ledger/checklist as you progress. Prioritize local-only operation, modularity, and clear documentation. After finishing each step, update the ledger and checklist, then continue to the next.

---

Go to https://github.com/kliewerdaniel/LGVRAG02.git I want you to analyze the repo in as much detail as possible. It is not functional and it needs to be. I want you to go through it piece by piece and determine what needs to be done next to implement the plans for its features and such. Your final output will be the prompt I give to CLIne to make the needed corrections, additions, changes and modifications needed.

1. Repo structure and intended features

From the README and directory layout, here is what the project is supposed to be:

High-level goal: A local Graph + Vector Retrieval-Augmented Generation (RAG) system using HelixDB (or fallback) as storage. It ingests documents (PDF / Markdown / text), builds a knowledge graph (extracting entities & relations), computes embeddings, and allows hybrid retrieval (graph traversal + vector similarity) combined with a local LLM to generate answers. A FastAPI interface exposes endpoints. Visualization is optional.

Project layout:

.
├── api/
│   └── main.py
├── config.py
├── config.yaml
├── db/
│   ├── helix_interface.py
│   └── ingest_data.py
├── documents/
├── ingestion/
│   ├── parse_docs.py
│   ├── extract_relations.py
│   └── embeddings.py
├── rag/
│   ├── retrieve.py
│   └── generate_answer.py
├── requirements.txt
├── ledger.md
└── README.md

So the modules are:
	•	ingestion: parse documents, extract relations, compute embeddings
	•	db: persist data (graph, embeddings, documents) — interface to HelixDB (or fallback)
	•	rag: retrieval logic and LLM answer generation
	•	api: external interface
	•	config: settings / parameters

From the README, goals include:
	•	ingesting documents
	•	building a knowledge graph (entities + relationships)
	•	computing embeddings
	•	hybrid retrieval (graph + vector)
	•	local LLM use (e.g. via Ollama)
	•	REST API endpoints: /query, /ingest, /upload, /stats, /health
	•	visualization via NetworkX / PyVis (optional)
	•	local-only operation (no external APIs, privacy, data local)

The README also mentions some caveats:
	•	HelixDB is “currently unavailable on PyPI” → fallback to SQLite
	•	Document processing libraries disabled for Python 3.13 (temporary)
	•	Some notes in ledger.md about implementation progress

But as you said, the repo as is is not functional. There’s no obvious main control logic or glue code filling in all necessary integration.

Also, I saw only one commit in the repo. That suggests much is skeletal or planned but not implemented.

Let me dig into each module to see what exists.

⸻

2. What I looked for (and what is missing)

Because I can’t see full source (GitHub did not show file contents beyond listing), I’ll rely on reasoning about what should exist. But the fact that only one commit exists suggests minimal code inside. (If I had direct file contents I would inspect them line by line, but with listing only, I must infer missingness.)

config.py / config.yaml
	•	config.yaml presumably holds default parameters: embedding model, LLM base URL, hybrid alpha, vector top-k, graph depth, db paths, chunk sizes, etc.
	•	config.py likely is a wrapper to load/validate config values from YAML or environment.

Check whether config reading, validation, default fallback is implemented. Also ensure any parameters are accessible to all modules.

ingestion/parse_docs.py
	•	Must have code to open files (PDF, Markdown, text), extract text, chunk into passages, possibly call LLM or local model for entity/relation extraction, and produce embeddings.

Missing aspects:
	•	PDF / Markdown / text reader libraries / wrappers
	•	chunking logic (sliding windows, overlap)
	•	cleanup: remove stop words, normalization
	•	error handling
	•	turning chunked text into document + passage units

ingestion/extract_relations.py
	•	Should take chunks / passages (or full docs) and extract entities and relations (e.g. using LLM prompts or some local model).
	•	Must define schema: nodes, edges, types
	•	Possibly co-reference resolution, deduplication, entity linking

Likely missing:
	•	actual prompt logic or model inference
	•	method to map extracted entities/relations to schema
	•	integration with graph storage (db interface)

ingestion/embeddings.py
	•	Should load embedding model (local), take text chunks, compute vector embeddings.
	•	Support batching, GPU/CPU, fallback to CPU etc.
	•	Return embeddings for storage or retrieval.

Missing:
	•	actual model loading
	•	embedding computation
	•	dimension consistency
	•	error handling for empty inputs

db/helix_interface.py
	•	Interface/abstraction layer to HelixDB (if available) or fallback to SQLite / other store
	•	Must support storing documents, passages, embeddings, entity nodes, relation edges
	•	Must support queries: vector similarity, graph traversal, hybrid combination
	•	Must allow retrieval API: e.g. get top-k vectors, get graph neighbors, etc.

Likely missing:
	•	implementation of store / load / query operations
	•	fallback logic when HelixDB unavailable
	•	consistency and indexing

db/ingest_data.py
	•	A script to coordinate ingestion: scan documents/, parse, extract relations, embed, store in DB
	•	Should be executable as python -m db.ingest_data
	•	Should accept parameters (e.g. file paths)
	•	Logging and error handling

Missing:
	•	orchestration logic
	•	incremental ingestion (only new docs)
	•	avoidance of duplicates
	•	update / upsert handling

rag/retrieve.py
	•	Hybrid retrieval logic: given a query, embed the query, retrieve top vector matches, traverse graph for related nodes/edges, combine, re-rank, return a set of passages or nodes
	•	Possibly alpha parameter for weighting vector vs graph scoring
	•	Must interface with db (helix_interface)
	•	Should allow tuning (top_k, graph depth)

Missing:
	•	embed query code
	•	retrieval logic
	•	ranking, scoring
	•	fallback when graph or vector modes fail

rag/generate_answer.py
	•	Given retrieved passages / entities, craft a prompt input to the LLM (local server)
	•	Handle prompt templating, temperature, model selection
	•	Parse LLM output (maybe JSON structured)
	•	Optionally include explanation (chain-of-thought, entity mentions)

Missing:
	•	code to send request to local LLM server (e.g. HTTP to base_url)
	•	prompt templates
	•	error handling, timeouts
	•	postprocessing

api/main.py
	•	FastAPI app that exposes endpoints as specified (/query, /ingest, /upload, /stats, /health)
	•	Parameter validation, JSON schemas
	•	Endpoint handlers that call the correct functions from ingestion / rag modules
	•	Possibly concurrency, threading, background tasks

Missing:
	•	complete server code
	•	app startup and teardown
	•	mapping between endpoints and internal modules
	•	CORS, error handling

ledger.md
	•	A project development log / design notes
	•	Could contain hints of what’s done or planned

It would be good to inspect ledger.md to see notes. (But I can’t view the content via listing only.)

Other essentials / cross-cutting features
	•	Logging (structured)
	•	Error handling and exceptions
	•	Testing (unit tests, integration tests)
	•	Configuration override via environment variables
	•	Dependency injection / modular design
	•	Performance / batching
	•	Handling large datasets / streaming ingestion
	•	Validation of embeddings dimension, vector store size
	•	Documentation and examples

⸻

3. Recommended plan: What to implement, in which order

Because the system is complex and modular, the realistic approach is incremental and layered: build ingestion → storage → retrieval → generation → API. Here’s a suggested roadmap, with dependencies:

Phase 0: Foundation / scaffolding
	•	Write or validate config loader in config.py that reads config.yaml, validates, and provides defaults.
	•	Define core data models / schemas (e.g. Python classes or pydantic models) for Document, Passage, Entity, Relation, Embedding.
	•	Define database schema (tables / graph structure) and DB interface abstractions.

Phase 1: Document ingestion & embedding
	•	In ingestion/parse_docs.py: implement reading of PDF, Markdown, text; chunking logic (with overlap, window size).
	•	In ingestion/embeddings.py: load a known embedding model (e.g. SentenceTransformer) and embed chunks.
	•	Return (document_id, passage_id, text, embedding vector).
	•	Write tests with dummy documents to check parsing + embedding pipeline.

Phase 2: Relation & graph extraction
	•	In ingestion/extract_relations.py: given passage text, send prompt to local LLM or use a local model to extract entity and relation triples (subject, predicate, object) per passage.
	•	Normalize / deduplicate entities (e.g. same name map to same node)
	•	Return node and edge structures (with IDs, types).
	•	Add tests: small snippets, known entities/relations.

Phase 3: Persistence into DB / graph store
	•	In db/helix_interface.py: implement key methods:
	•	store_document(...)
	•	store_passage(...)
	•	store_embedding(...)
	•	store_entity_node(...)
	•	store_relation_edge(...)
	•	get_top_k_vectors(query_embedding, top_k)
	•	get_graph_neighbors(entity_id, depth)
	•	get_passages_by_entity(...)
	•	Add fallback SQLite / relational store if HelixDB unavailable.
	•	Ensure indexing and performance (e.g. vector index).
	•	In db/ingest_data.py: orchestrate full pipeline: for each file in documents/, parse/get passages, embed, extract relations, store everything. Handle duplicates / reingestion.

Phase 4: Retrieval logic (hybrid)
	•	In rag/retrieve.py: implement:
	•	Embed the user query (via embeddings module)
	•	Retrieve top vector passages
	•	Identify entities in those passages, traverse graph (neighbors up to graph_depth)
	•	Collect passages connected to those entities
	•	Score/rank passages combining vector similarity and graph-based relevance via hybrid_alpha
	•	Return final ranked list (with passage texts, entity annotations).
	•	Add tests: synthetic graph + texts, queries to validate retrieval.

Phase 5: Answer generation
	•	In rag/generate_answer.py:
	•	Given query + retrieved passages, create a prompt (template)
	•	Send prompt to the local LLM via HTTP (config.llm.base_url, config.llm.model_name)
	•	Return the LLM response
	•	Optionally, extract and return source passages or entity references
	•	Handle timeouts, errors, fallback.

Phase 6: API layer
	•	In api/main.py, build FastAPI app:
	•	/health: simple ping
	•	/stats: return counts of documents, embeddings, graph nodes/edges
	•	/ingest: trigger ingestion of a file path (or all)
	•	/upload: handle multipart file upload, save to documents/, then ingest
	•	/query: accept JSON { "query": str, "top_k": int, "include_entities": bool }, call retrieve + generate_answer
	•	Add error handling, input validation (via pydantic), CORS, etc.
	•	Test via curl or HTTP client.

Phase 7: Visualization & optional features
	•	Use NetworkX / PyVis to visualize the subgraph for a query (entities, relations)
	•	Expose an endpoint (or return in /query response) for graph visualization (e.g. JSON, HTML)
	•	Add logging, metrics (timing)
	•	Optimize performance, memory, batching

Phase 8: Integration, tests, docs
	•	Write integration tests (ingest → query → answer)
	•	Document usage in README, examples
	•	Add CLI tool (optional)
	•	Handle edge cases (empty docs, duplicate ingestion, large docs)
	•	Possibly modular plugin system

⸻

4. Potential pitfalls / challenges & decisions to make
	•	HelixDB dependency: since HelixDB is “unavailable on PyPI,” choose a fallback (SQLite, SQLite + vector index via Faiss or Annoy) or wrap a local vector DB (e.g. SQLite + HNSW index).
	•	Local LLM interface: ensure the LLM server (e.g. Ollama) supports HTTP API or some local wrapper. Build robust client with error handling and retries.
	•	Entity / relation extraction via LLM: prompt engineering, consistency, potential hallucination. Need validation or fallback.
	•	Graph normalization: merging duplicate entities, co-reference resolution, entity linking.
	•	Ranking / scoring: combining graph and vector scores properly; normalization needed.
	•	Scale: ingestion of large document collections, memory usage, streaming.
	•	Concurrency / async: API handlers should be async; retrieval and embedding should possibly be async or batched.
	•	Testing: critical to build small unit tests early.
	•	Configuration: allow override via environment, command-line.
	•	Error handling & logging: essential for debugging.

⸻

5. Prompt for CLine (or your code assistant) to implement the missing pieces

Below is a detailed prompt you can feed into your code-generation assistant (CLine) to implement/fix the repository. You can adjust as needed, but this gives a structured instruction set.

⸻

Prompt to CLine

You are given the repository LGVRAG02 (a local Graph + Vector RAG system). It is currently nonfunctional and has only boilerplate structure. You must fully implement it module by module, according to the design in README.md and ledger.md. Below is the implementation plan; follow it programmatically, writing code, tests, and documentation.

⸻

1. Configuration
	•	Implement config.py to load and validate settings from config.yaml (or environment overrides). Use a library (e.g. pydantic) to define a config schema (sections: embeddings, llm, retrieval, database, api).
	•	Provide default values and type checking.

2. Data models / schemas
	•	Define Python classes (or pydantic models) for Document, Passage, EntityNode, RelationEdge, Embedding. Include attributes like id, text, metadata, etc.

3. Ingestion / Parsing & Embedding
	•	In ingestion/parse_docs.py:
	•	Support reading PDF, Markdown, plain text files.
	•	Chunk the text into passages (configurable window sizes, overlap).
	•	Return list of (passage_id, passage_text, metadata).
	•	In ingestion/embeddings.py:
	•	Using a local embedding model (e.g. sentence-transformers), load model as per config.
	•	Accept a batch of passage texts and return embedding vectors (e.g. numpy arrays or lists).
	•	Ensure batching and device (CPU/GPU) support.

4. Entity & Relation Extraction
	•	In ingestion/extract_relations.py:
	•	Given a passage text, generate entity-relation triples (e.g. via prompt to local LLM).
	•	Use a prompt template to ask for JSON output: list of { subject, predicate, object }.
	•	Normalize entities: assign consistent IDs, merge duplicates.
	•	Return node and edge data structures.

5. Database / Storage Interface
	•	In db/helix_interface.py:
	•	Implement storage methods:
	•	store_document(doc: Document)
	•	store_passage(p: Passage)
	•	store_embedding(passage_id, embedding)
	•	store_entity(entity: EntityNode)
	•	store_relation(edge: RelationEdge)
	•	Query methods:
	•	get_top_k_vectors(query_embedding, top_k)
	•	get_graph_neighbors(entity_id, depth)
	•	get_passages_by_entity(entity_id)
	•	Support fallback storage using SQLite + a vector index (e.g. FAISS, Annoy, or SQLite’s FTS), in case HelixDB is not available.
	•	Ensure indexing, transactional writes, error handling.
	•	In db/ingest_data.py:
	•	Orchestrate ingestion pipeline:
	1.	Scan documents/ directory for new files.
	2.	For each file, parse into passages.
	3.	Compute embeddings.
	4.	Extract relations.
	5.	Store documents / passages / embeddings / entities / relations.
	•	Handle duplicate ingestion (skip existing).
	•	Provide CLI entrypoint: python -m db.ingest_data.

6. Retrieval (Hybrid)
	•	In rag/retrieve.py:
	•	Embed the query text.
	•	Retrieve top-K passages via vector similarity.
	•	From those passages, extract entity IDs referenced.
	•	Traverse graph neighbors (configurable graph_depth) to collect more relevant passages.
	•	Score and rank passages using a hybrid function:
score = α * vector_score + (1 - α) * graph_score
	•	Return the top ranked passages, along with associated entities / relations.
	•	Include normalization or scaling of vector and graph scores so they are comparable.

7. Answer Generation
	•	In rag/generate_answer.py:
	•	Given input query and retrieved passages, build a prompt template (with context).
	•	Send request to local LLM server at config.llm.base_url with parameters (model name, temperature, etc.).
	•	Parse and return the answer string (and optionally metadata: which passages / entities used).
	•	Handle errors, timeouts, fallback logic.

8. API Layer
	•	In api/main.py, build a FastAPI app:
	•	Endpoint GET /health: return { "status": "ok" }.
	•	Endpoint GET /stats: counts of docs, passages, entities, embeddings, edges.
	•	Endpoint POST /ingest: accept a file_path or path in documents/, trigger ingestion for that file.
	•	Endpoint POST /upload: accept file upload (multipart), save into documents/, then ingest.
	•	Endpoint POST /query: accept JSON { query: str, top_k: int, include_entities: bool }, call retrieval + generation, return { answer, sources, entity_graph (optional) }.
	•	Validate inputs (using pydantic models), error handling, CORS.
	•	Support startup event: initialize DB, embedding model, LLM client, etc.

9. Visualization & Optional Features
	•	Optionally integrate a graph visualization: for a query, return node/edge data suitable for NetworkX or PyVis.
	•	You may include an optional endpoint or include visualization output in /query response (e.g. HTML or JSON).

10. Testing & Documentation
	•	Write unit tests for each module: parsing, embedding, extraction, DB, retrieval, generation, API.
	•	Write integration tests: ingest some sample documents and then query to verify end-to-end operation.
	•	Update README.md: usage instructions, quick start, example queries.
	•	Ensure the package is installable (if needed), add __main__ wrappers or CLI.


⸻

Constraints & design decisions:
	•	The system must run entirely locally (no external API calls).
	•	Do not rely on internet-based services; everything should use local models / DBs.
	•	Performance: support batching, indexing, efficient queries.
	•	Robust error handling, logging, and validations.
	•	Follow the config file, allow for extensibility.

Please read ledger.md (in the repo) for any existing notes or partial implementations, respect those as constraints or starting points, but do not rely on them being fully correct. As you generate each file or module, also generate minimal tests and integration scaffolding. After completing all modules, produce a final test scenario (e.g. ingest a small document and issue a query) that verifies the system works end-to-end.

Fix the Graph + Vector RAG system to make it fully functional. The system currently has several critical issues:

## Issues to Fix:

1. **Database Implementation** - Replace SQLite JSON storage with a proper vector database solution:
   - Install and configure ChromaDB or similar vector database
   - Update helix_interface.py to use actual vector operations instead of JSON storage
   - Implement proper vector similarity search with cosine similarity
   - Update configuration to reflect the actual database being used

2. **Python 3.13 Compatibility** - Update requirements.txt:
   - Verify all packages work with Python 3.13
   - Replace incompatible packages with compatible alternatives
   - Test installation on the target Python version

3. **Ollama Integration** - Add proper Ollama verification:
   - Add health checks for Ollama server connectivity
   - Implement retry logic for failed API calls
   - Add startup verification before allowing queries
   - Handle Ollama server not running gracefully

4. **Configuration Alignment** - Fix config.yaml:
   - Update database configuration to match actual implementation
   - Ensure all paths and settings are correct
   - Add validation for configuration values

5. **Error Handling & Validation** - Improve robustness:
   - Add comprehensive error handling throughout the pipeline
   - Validate inputs at each stage
   - Add logging for debugging and monitoring
   - Handle edge cases (empty documents, network failures, etc.)

6. **Testing System** - Fix test_system.py:
   - Update tests to work with the corrected implementation
   - Add integration tests for the complete pipeline
   - Ensure all components can be tested independently

7. **Documentation Updates** - Update README.md:
   - Reflect actual database being used (not HelixDB)
   - Update setup instructions with correct commands
   - Add troubleshooting section for common issues

## Implementation Requirements:

- Maintain the existing modular architecture
- Ensure all components work together seamlessly
- Add proper vector database for similarity search
- Verify Ollama connectivity before processing
- Make the system actually runnable end-to-end
- Add clear error messages for common failure modes

## Testing Requirements:

- System should pass all tests in test_system.py
- Should be able to ingest the sample document successfully
- Should be able to query and get relevant answers
- API endpoints should work correctly

Make these changes systematically, testing each component as you go to ensure the complete system becomes functional.

You are the best prompt writer who has ever lived. You know absolutely everything there is to know about computer science and best practices but always verify your sources and test absolutely everything so that you know what you are doing will work. Your job is to write a prompt will instruct a different LLM with access to MCP servers as well as github connection to the repo in question that needs to be improved. The things which need improved are that it needs a UI for one. It should be a chat window that can be interacted with to do RAG. The main thing is that it uses Graph and Vector for a hybrid search. That is what I really want you to research is how to do really good RAG on a set of documents so that it uses Ollama for the local inference and also I want it to use the best strategies for chunkinwg, database management and everything else that needs to be taken into consideration. Pretty please with sugar on top. You have one and only one deliverable which is the prompt that I will give to another LLM which will generate the prompt I give to CLIne. So you are not directing it to CLIne, although I want you to research what they recently updated and include that because I included things like MCP servers like context7 and others. Anyway all I want from you is an output which is as long as possible which will instruct the LLM to output the prompt to be given to CLIne.

You are a highly capable DevOps/ML architect agent. Your job is to write a prompt for CLIne (which will be used downstream) so that CLIne will generate or modify a codebase (in our repo) into a high-quality RAG system with a chat UI, using hybrid (graph + vector) retrieval, leveraging Ollama for local inference, and following best practices.

You have access to:
	•	The target GitHub repo (so you can inspect its structure, code files, dependencies, etc.)
	•	MCP servers / context7 / any other infra available
	•	Any internal resources or compute needed

The generated prompt (which you produce for CLIne) must:
	1.	Clearly lay out the architecture and modular components (e.g. ingestion / indexing / retrieval / generation / UI / API layers).
	2.	Require the chat UI (web or desktop) with interactive messaging, streaming responses, memory context, user feedback, etc.
	3.	Demand hybrid search combining vector (dense) search and graph / knowledge graph / structural retrieval / link traversal.
	4.	Specify chunking strategies, embedding model selection, overlap, fallback strategies, re-ranking, filtering, metadata, and evaluation metrics.
	5.	Handle database management: how to store chunks, embeddings, graph nodes/edges, metadata, versioning, updates, deletion, incremental reindexing, consistency, backups.
	6.	Integrate Ollama (local LLM inference) as the generation engine, including prompt construction, context windows, streaming responses, and cost/throughput trade-offs.
	7.	Include instructions to test, benchmark, and validate each component (e.g. retrieval recall / precision, latency, hallucination tests).
	8.	Lay out error handling, fallback strategies, logging, observability, and monitoring.
	9.	Optionally, include guidance on scaling (sharding, caching, quantization, memory budgets).
	10.	Be explicit about versioning, dependencies, and interface contracts so that the generated code is coherent and maintainable.

While composing that prompt for CLIne, you (the orchestrator LLM) should refer to best practices and cite or internalize empirical pointers (e.g. from GraphRAG, ACORN, hybrid search, chunking strategies, pitfalls).

A few technical references / constraints you should internalize when writing:
	•	GraphRAG is a good pattern to combine knowledge graph plus vector retrieval in a unified RAG architecture. (There are local Ollama adaptations / patches to GraphRAG to support local models. See “GraphRAG Local Setup via Ollama” for implementation pitfalls. )  
	•	Hybrid search (dense + sparse) boosts recall and catches edge cases when embeddings miss tokens/keywords. Use hybrid search or re-ranking.  
	•	Chunking: use overlapping chunks, tune chunk sizes relative to context window, maybe semantic or hierarchical chunking (e.g. via HiChunk ideas).  
	•	Beware common vector search pitfalls: insufficient evaluation, ignoring hybrid search, chunk boundary issues, quantization or memory scaling problems.  
	•	For graph + vector hybrid indexes, research ACORN ideas (predicate-agnostic hybrid search) to unify sparse + dense queries.  
	•	Ingest pipelines: extract text cleanly (OCR, image captions, handle tables), build entities / edges for KG, then chunk or embed.  

The prompt you output for CLIne should be self-contained (i.e. not depend on this meta context) and precise (so that CLIne’s generated code is well scoped).

Your deliverable: only the prompt to feed into CLIne (no extra filler). The prompt should be long, detailed, and prescriptive, but also modular so CLIne can break it into tasks or code modules.

# Comprehensive RAG System Implementation Prompt for CLIne

You are tasked with transforming and extending the existing Graph + Vector RAG codebase into a production-grade, hybrid retrieval-augmented generation system with a full-featured chat UI. The system must combine graph-based knowledge representation with vector similarity search, use Ollama for local LLM inference, and follow enterprise-grade best practices for scalability, reliability, and maintainability.

## System Architecture Overview

Build a modular RAG system with these layers:

1. **Ingestion & Indexing Layer**: Document parsing, entity extraction, chunking, embedding generation, graph construction
2. **Storage Layer**: ChromaDB for vectors, SQLite/graph database for knowledge graph, metadata store
3. **Retrieval Layer**: Hybrid search combining vector similarity + graph traversal + sparse/keyword search
4. **Generation Layer**: Ollama integration with prompt construction, context assembly, streaming responses
5. **API Layer**: FastAPI endpoints for ingestion, query, management
6. **UI Layer**: Interactive chat interface with conversation history, streaming, user feedback
7. **Observability Layer**: Logging, metrics, tracing, evaluation

## Critical Requirements

### 1. Chat UI Implementation

Create a modern, responsive chat interface with:

**Frontend Requirements:**
- Real-time streaming response display with typewriter effect
- Conversation history panel showing all past queries/answers
- Session management with ability to start new conversations
- Source citation display with expandable context snippets
- User feedback mechanisms (thumbs up/down, flag incorrect responses)
- Loading states, error handling, and retry mechanisms
- Mobile-responsive design
- Dark/light mode toggle
- Copy response button
- Export conversation to markdown/PDF

**Technology Stack Options:**
- React + TypeScript + Tailwind CSS (preferred for modern web apps)
- Streamlit (faster prototyping but less customizable)
- Gradio (simplest for ML demos)
- Vue.js + Vuetify (alternative modern stack)

**Backend Integration:**
- WebSocket or Server-Sent Events (SSE) for streaming responses
- Session storage with conversation context
- Rate limiting and authentication hooks
- Conversation persistence in database

**Features:**
- Multi-turn conversation with context retention (last N turns)
- "Regenerate response" option
- "Clear conversation" option
- Search within conversation history
- Tag/categorize conversations
- Share conversation via unique URL

### 2. Hybrid Retrieval Architecture

Implement a sophisticated three-stage hybrid retrieval:

**Stage 1: Multi-Strategy Initial Retrieval**

A. **Dense Vector Search (Primary)**
- Use ChromaDB's HNSW index for approximate nearest neighbor search
- Query embedding generated by sentence-transformers model
- Retrieve top-K candidates (K=20-50) with cosine similarity scores
- Implement embedding cache to avoid recomputing query embeddings

B. **Sparse/Keyword Search (BM25)**
- Implement BM25 ranking using rank-bm25 or custom implementation
- Tokenize query and chunks using same tokenizer
- Retrieve top-K candidates based on term frequency matching
- Essential for catching exact phrase matches, acronyms, proper nouns

C. **Graph Traversal Search**
- Extract entities from query using NER (spaCy or local LLM)
- Find matching entities in knowledge graph
- Traverse graph up to depth=2 to find related entities
- Retrieve chunks associated with traversed entities
- Weight by graph distance and edge confidence scores

**Stage 2: Fusion & Re-ranking**

Implement Reciprocal Rank Fusion (RRF):
python
# For each retrieval method m, rank position r
score = sum(1 / (k + rank_m)) for all methods
# where k=60 is typical constant


Alternative: Train a cross-encoder re-ranker:
- Use sentence-transformers cross-encoder models (e.g., `ms-marco-MiniLM-L-6-v2`)
- Re-rank top 20-50 candidates from fusion
- Return top-K (K=5-10) for final context

**Stage 3: Context Compression & Assembly**

- Remove redundant/duplicate chunks using similarity threshold
- Apply maximum marginal relevance (MMR) for diversity
- Compress chunks if total tokens exceed context window
- Maintain source attribution metadata

**Implementation Details:**

python
class HybridRetriever:
    def retrieve(self, query: str, top_k: int = 5) -> List[RetrievalResult]:
        # 1. Dense vector search
        query_embedding = self.embed_query(query)
        dense_results = self.vector_search(query_embedding, top_k=50)
        
        # 2. Sparse keyword search (BM25)
        sparse_results = self.bm25_search(query, top_k=50)
        
        # 3. Graph traversal
        query_entities = self.extract_entities(query)
        graph_results = self.graph_search(query_entities, depth=2, top_k=50)
        
        # 4. Fusion with RRF
        fused_results = self.reciprocal_rank_fusion(
            [dense_results, sparse_results, graph_results],
            weights=[0.5, 0.3, 0.2]
        )
        
        # 5. Re-rank with cross-encoder
        reranked = self.cross_encoder_rerank(query, fused_results, top_k=20)
        
        # 6. Apply MMR for diversity
        final_results = self.mmr_diversify(reranked, top_k=top_k, lambda_param=0.7)
        
        return final_results


**Fallback Strategies:**
- If vector search returns <3 results, expand search radius or reduce similarity threshold
- If graph search finds no entities, fall back to keyword expansion
- If all methods fail, return most recent/popular documents

### 3. Advanced Chunking Strategy

Implement multi-level chunking with semantic boundaries:

**Chunking Approaches:**

A. **Fixed-Size with Overlap** (Baseline)
- Chunk size: 512-1024 tokens (tune based on embedding model)
- Overlap: 50-100 tokens (10-20% of chunk size)
- Preserve sentence boundaries using spaCy or nltk

B. **Semantic Chunking** (Preferred)
- Use sentence embeddings to detect topic shifts
- Split at boundaries where cosine similarity between consecutive sentences drops below threshold (e.g., 0.6)
- Ensures chunks contain semantically coherent content
- Library: LangChain's `SemanticChunker` or custom implementation

C. **Hierarchical Chunking**
- Parent chunks: 1500-2000 tokens
- Child chunks: 300-500 tokens
- Store parent-child relationships in metadata
- Retrieve child chunks but provide parent context during generation
- Helps maintain document structure and context

**Implementation:**

python
class AdvancedChunker:
    def chunk_document(self, doc: Document) -> List[Chunk]:
        # Extract document structure
        sections = self.extract_sections(doc)  # Headings, paragraphs
        
        chunks = []
        for section in sections:
            # Semantic chunking within section
            semantic_chunks = self.semantic_chunk(
                section.text,
                max_tokens=512,
                similarity_threshold=0.65
            )
            
            # Create hierarchical structure
            parent_chunk = self.create_parent_chunk(section)
            for i, child_text in enumerate(semantic_chunks):
                chunk = Chunk(
                    text=child_text,
                    parent_id=parent_chunk.id,
                    metadata={
                        'section': section.title,
                        'position': i,
                        'doc_id': doc.id,
                        'doc_type': doc.type
                    }
                )
                chunks.append(chunk)
        
        return chunks
    
    def semantic_chunk(self, text: str, max_tokens: int, similarity_threshold: float):
        sentences = self.sentence_splitter.split(text)
        sentence_embeddings = self.embed_sentences(sentences)
        
        chunks = []
        current_chunk = [sentences[0]]
        
        for i in range(1, len(sentences)):
            similarity = cosine_similarity(
                sentence_embeddings[i-1],
                sentence_embeddings[i]
            )
            
            if similarity < similarity_threshold or len(current_chunk) >= max_tokens:
                chunks.append(' '.join(current_chunk))
                current_chunk = [sentences[i]]
            else:
                current_chunk.append(sentences[i])
        
        if current_chunk:
            chunks.append(' '.join(current_chunk))
        
        return chunks


**Chunking for Different Document Types:**
- PDFs: Extract text with layout preservation, handle multi-column layouts
- Tables: Convert to markdown or structured text, preserve headers
- Code: Chunk by function/class boundaries, preserve imports
- Images: Extract captions, OCR text, use vision models for descriptions

### 4. Embedding Model Strategy

**Primary Embedding Model:**
- Use `sentence-transformers/all-MiniLM-L6-v2` (384 dim) for speed/memory balance
- Alternative: `BAAI/bge-base-en-v1.5` (768 dim) for better quality
- For domain-specific: Fine-tune embeddings on your corpus

**Embedding Best Practices:**
- Normalize embeddings (L2 norm) before storage
- Use consistent encoding for queries and documents
- Add instruction prefixes for asymmetric search (query vs. passage)
- Cache embeddings with content hash as key
- Batch embedding generation for efficiency (batch_size=32-64)

**Quantization for Scale:**
- Use binary quantization or product quantization if dataset >100K chunks
- Implement scalar quantization (int8) for 4x memory reduction
- Test recall degradation before production deployment

### 5. Knowledge Graph Construction

Build a rich knowledge graph from documents:

**Entity Extraction:**
- Use Ollama with structured output prompts (JSON mode)
- Extract: PERSON, ORGANIZATION, LOCATION, TECHNOLOGY, CONCEPT, EVENT
- Run entity linking to normalize variations (e.g., "ML" → "Machine Learning")
- Store entity embeddings for similarity search

**Relationship Extraction:**
- Use Ollama to extract (subject, predicate, object) triples
- Predicate types: related_to, part_of, instance_of, causes, enables, created_by
- Assign confidence scores to each triple
- Validate relationships using co-occurrence statistics

**Graph Schema:**

python
# Nodes
class Entity:
    id: str
    name: str
    type: EntityType
    embedding: List[float]
    confidence: float
    source_chunks: List[str]
    metadata: Dict

# Edges
class Relationship:
    id: str
    subject_id: str
    predicate: str
    object_id: str
    confidence: float
    source_chunk: str
    metadata: Dict


**Graph Storage:**
- Use SQLite for small graphs (<100K entities)
- Migrate to Neo4j or NetworkX for larger graphs
- Create indexes on entity names and types
- Implement graph versioning for updates

**Graph Query Patterns:**
- Single-hop: Direct neighbors of query entities
- Multi-hop: BFS/DFS up to depth N
- Community detection: Find entity clusters
- Centrality: Rank entities by PageRank or betweenness

### 6. Database Management & Persistence

**Vector Database (ChromaDB):**

python
class VectorStore:
    def __init__(self):
        self.client = chromadb.PersistentClient(path="./chroma_db")
        self.collection = self.client.get_or_create_collection(
            name="chunks",
            metadata={"hnsw:space": "cosine"}
        )
    
    def add_chunks(self, chunks: List[Chunk]):
        self.collection.add(
            ids=[c.id for c in chunks],
            embeddings=[c.embedding for c in chunks],
            documents=[c.text for c in chunks],
            metadatas=[c.metadata for c in chunks]
        )
    
    def update_chunk(self, chunk_id: str, new_chunk: Chunk):
        # ChromaDB doesn't support direct updates - delete & re-add
        self.collection.delete(ids=[chunk_id])
        self.add_chunks([new_chunk])
    
    def delete_document(self, doc_id: str):
        # Delete all chunks for a document
        results = self.collection.get(where={"doc_id": doc_id})
        self.collection.delete(ids=results['ids'])


**Incremental Indexing:**
- Detect document changes via content hash or modification timestamp
- Re-chunk and re-embed only changed documents
- Update graph incrementally (add new entities/relationships, remove orphaned nodes)
- Implement versioning: store document version in metadata

**Data Consistency:**
- Use transactions for atomic updates (embed + store)
- Implement rollback mechanism for failed ingestions
- Validate data integrity: check embedding dimensions, null values, orphaned references
- Periodic consistency checks: verify chunk count, embedding coverage

**Backup & Recovery:**
- Daily backups of ChromaDB directory and SQLite database
- Export metadata to JSON for recovery
- Implement backup retention policy (keep last 7 days, monthly archives)
- Test recovery procedure quarterly

### 7. Ollama Integration & Generation

**Ollama Setup:**
- Default model: `granite4:micro-h` (fast, low memory) or `llama3.2:1b`
- For better quality: `mistral:7b` or `llama3:8b`
- Configure Ollama in config.yaml with model selection per use case

**Prompt Engineering:**

python
class PromptBuilder:
    def build_rag_prompt(self, query: str, context_chunks: List[Chunk], 
                        conversation_history: List[Message]) -> str:
        
        # System instruction
        system = """You are a helpful AI assistant. Use the provided context to answer questions accurately. 
If the context doesn't contain enough information, say so clearly. 
Cite sources using [1], [2] notation."""
        
        # Format context with citations
        context = "\n\n".join([
            f"[{i+1}] {chunk.text}\nSource: {chunk.metadata['source']}"
            for i, chunk in enumerate(context_chunks)
        ])
        
        # Add conversation history (last 3 turns)
        history = "\n".join([
            f"User: {msg.query}\nAssistant: {msg.response}"
            for msg in conversation_history[-3:]
        ])
        
        prompt = f"""{system}

Conversation History:
{history}

Context:
{context}

Question: {query}

Answer:"""
        
        return prompt


**Streaming Responses:**

python
async def generate_streaming(self, prompt: str) -> AsyncIterator[str]:
    url = f"{self.ollama_base_url}/api/generate"
    payload = {
        "model": self.model_name,
        "prompt": prompt,
        "stream": True,
        "options": {
            "temperature": 0.1,
            "num_predict": 2048,
            "top_p": 0.9
        }
    }
    
    async with aiohttp.ClientSession() as session:
        async with session.post(url, json=payload) as response:
            async for line in response.content:
                if line:
                    data = json.loads(line)
                    if "response" in data:
                        yield data["response"]


**Context Window Management:**
- Monitor total tokens: query + context + history
- Truncate context if exceeds window (keep highest-ranked chunks)
- Implement sliding window for long conversations
- Use compression techniques: summarize old turns, remove redundant context

**Generation Parameters:**
- Temperature: 0.1-0.3 for factual QA, 0.7-0.9 for creative
- Top-p: 0.9 for nucleus sampling
- Frequency penalty: 0.1 to reduce repetition
- Stop tokens: ["\n\nUser:", "\n\nQuestion:"]

### 8. API Layer Enhancement

**Endpoints:**

python
@app.post("/api/chat")
async def chat(request: ChatRequest):
    """Main chat endpoint with streaming support"""
    # 1. Retrieve context
    retrieval_results = retriever.retrieve(request.query, top_k=5)
    
    # 2. Build prompt with conversation history
    prompt = prompt_builder.build_rag_prompt(
        request.query,
        retrieval_results,
        request.conversation_history
    )
    
    # 3. Generate streaming response
    return StreamingResponse(
        generator.generate_streaming(prompt),
        media_type="text/event-stream"
    )

@app.post("/api/ingest")
async def ingest(file: UploadFile):
    """Ingest document with progress tracking"""
    # 1. Save file
    # 2. Parse and chunk
    # 3. Extract entities and build graph
    # 4. Generate embeddings
    # 5. Store in databases
    # 6. Return ingestion stats
    pass

@app.get("/api/conversations/{conversation_id}")
async def get_conversation(conversation_id: str):
    """Retrieve conversation history"""
    pass

@app.post("/api/feedback")
async def submit_feedback(feedback: FeedbackRequest):
    """Store user feedback for evaluation"""
    pass

@app.get("/api/stats")
async def get_stats():
    """System statistics and health"""
    return {
        "total_chunks": db.count_chunks(),
        "total_entities": db.count_entities(),
        "total_conversations": db.count_conversations(),
        "avg_latency_ms": metrics.get_avg_latency(),
        "ollama_status": await check_ollama_health()
    }
``

**Rate Limiting:**
- Implement per-user rate limits (e.g., 10 requests/minute)
- Use Redis or in-memory store for tracking
- Return 429 status with Retry-After header

**Authentication:**
- Add API key authentication for production
- Use JWT tokens for session management
- Implement role-based access control (admin, user, readonly)

### 9. Testing & Validation

**Unit Tests:**

python
def test_chunking():
    chunker = AdvancedChunker()
    doc = Document(text="..." * 1000)
    chunks = chunker.chunk_document(doc)
    
    assert len(chunks) > 0
    assert all(len(c.text) <= MAX_CHUNK_SIZE for c in chunks)
    assert all(c.metadata['doc_id'] == doc.id for c in chunks)

def test_vector_search():
    retriever = HybridRetriever()
    results = retriever.retrieve("machine learning", top_k=5)
    
    assert len(results) <= 5
    assert all('score' in r for r in results)
    assert results[0]['score'] >= results[-1]['score']  # Sorted


**Retrieval Evaluation:**

Implement standard IR metrics:

python
def evaluate_retrieval(test_queries: List[Query], ground_truth: Dict):
    metrics = {
        'recall@k': [],
        'precision@k': [],
        'mrr': [],  # Mean Reciprocal Rank
        'ndcg@k': []  # Normalized Discounted Cumulative Gain
    }
    
    for query in test_queries:
        retrieved = retriever.retrieve(query.text, top_k=10)
        relevant = ground_truth[query.id]
        
        # Calculate metrics
        retrieved_ids = [r['id'] for r in retrieved]
        relevant_ids = set(relevant)
        
        recall = len(set(retrieved_ids) & relevant_ids) / len(relevant_ids)
        precision = len(set(retrieved_ids) & relevant_ids) / len(retrieved_ids)
        
        metrics['recall@k'].append(recall)
        metrics['precision@k'].append(precision)
    
    return {k: np.mean(v) for k, v in metrics.items()}


**Generation Evaluation:**

python
def evaluate_generation(test_cases: List[TestCase]):
    results = []
    
    for case in test_cases:
        answer = generator.generate_answer_sync(
            case.query,
            case.context_chunks
        )
        
        # 1. Faithfulness: Is answer grounded in context?
        faithfulness = evaluate_faithfulness(answer.answer, case.context_chunks)
        
        # 2. Relevance: Does answer address the query?
        relevance = evaluate_relevance(case.query, answer.answer)
        
        # 3. Completeness: Are all aspects covered?
        completeness = evaluate_completeness(case.query, answer.answer)
        
        results.append({
            'faithfulness': faithfulness,
            'relevance': relevance,
            'completeness': completeness
        })
    
    return results


**Hallucination Detection:**
- Use sentence-level entailment to check if answer is entailed by context
- Flag answers with low entailment scores for human review
- Implement citation verification: check if cited sources actually support claims

**Latency Benchmarks:**
- Target: <2s for retrieval, <5s for full response (streaming starts <1s)
- Profile each component: embedding (50-100ms), vector search (100-200ms), graph search (200-500ms), generation (3-5s)
- Optimize bottlenecks: batch operations, caching, async execution

### 10. Observability & Monitoring

**Logging:**

python
import structlog

logger = structlog.get_logger()

@app.post("/api/chat")
async def chat(request: ChatRequest):
    logger.info("chat_request", 
                query=request.query,
                conversation_id=request.conversation_id,
                user_id=request.user_id)
    
    start = time.time()
    
    try:
        # ... retrieval and generation ...
        
        logger.info("chat_success",
                   latency_ms=(time.time() - start) * 1000,
                   num_chunks=len(retrieval_results),
                   response_length=len(response))
    
    except Exception as e:
        logger.error("chat_error",
                    error=str(e),
                    traceback=traceback.format_exc())
        raise


**Metrics:**
- Track: requests/min, latency percentiles (p50, p95, p99), error rate, cache hit rate
- Use Prometheus + Grafana for visualization
- Alert on: error rate >5%, p95 latency >10s, Ollama downtime

**Tracing:**
- Implement distributed tracing with OpenTelemetry
- Trace spans: HTTP request → retrieval → embedding → vector search → graph search → generation
- Visualize with Jaeger or Zipkin

**User Analytics:**
- Track: query patterns, popular topics, user satisfaction (feedback), retry rates
- Identify: failing queries, low-confidence answers, missing information gaps
- Use insights to improve: add more documents, retrain models, tune retrieval

### 11. Error Handling & Resilience

**Graceful Degradation:**

python
async def retrieve_with_fallback(query: str) -> List[Chunk]:
    try:
        # Primary: Hybrid retrieval
        return await hybrid_retriever.retrieve(query)
    except Exception as e:
        logger.warning("hybrid_retrieval_failed", error=str(e))
        
        try:
            # Fallback 1: Vector-only search
            return await vector_retriever.retrieve(query)
        except Exception as e2:
            logger.warning("vector_retrieval_failed", error=str(e2))
            
            # Fallback 2: Keyword search
            return await keyword_retriever.retrieve(query)


**Circuit Breaker:**
- Implement circuit breaker for Ollama calls
- If >50% failures in 1 minute, open circuit for 30 seconds
- Return cached/fallback responses during circuit open

**Retry Logic:**
- Retry transient failures (timeout, connection error) with exponential backoff
- Max 3 retries with 1s, 2s, 4s delays
- Don't retry on 4xx errors (bad request)

**Input Validation:**
- Sanitize user queries: remove SQL injection, XSS patterns
- Enforce length limits: query <500 chars, conversation history <50 turns
- Validate file uploads: size <10MB, allowed extensions, virus scan

### 12. Scaling Considerations

**Horizontal Scaling:**
- Run multiple API server instances behind load balancer
- Share ChromaDB and SQLite via network filesystem or migrate to distributed databases
- Use Redis for shared caching and session storage

**Caching Strategy:**
- L1 Cache: In-memory LRU cache for query embeddings (size=1000)
- L2 Cache: Redis for retrieval results (TTL=1 hour)
- L3 Cache: Precomputed embeddings for common queries

**Database Sharding:**
- Shard ChromaDB by document category or time range
- Route queries to relevant shards based on metadata filters
- Aggregate results across shards

**Model Optimization:**
- Use quantized Ollama models (Q4_K_M, Q5_K_M) for 2-4x speedup
- Run Ollama with GPU acceleration if available
- Batch multiple concurrent requests to Ollama

**Memory Budget:**
- Monitor: embeddings (N_chunks * dim * 4 bytes), graph (N_entities * 1KB), model (varies)
- Set limits: max_chunks_per_query, max_conversation_length
- Implement eviction policies for caches

### 13. Configuration Management

Extend `config.yaml` with detailed settings:

yaml
# Retrieval
retrieval:
  vector_top_k: 50
  bm25_top_k: 50
  graph_top_k: 50
  graph_depth: 2
  hybrid_weights: [0.5, 0.3, 0.2]  # vector, bm25, graph
  rerank_top_k: 20
  final_top_k: 5
  mmr_lambda: 0.7
  cross_encoder_model: "cross-encoder/ms-marco-MiniLM-L-6-v2"

# Chunking
chunking:
  strategy: "semantic"  # fixed, semantic, hierarchical
  max_chunk_tokens: 512
  overlap_tokens: 100
  semantic_threshold: 0.65
  min_chunk_tokens: 50

# Entity Extraction
entities:
  enabled: true
  extract_with_llm: true
  entity_types: ["PERSON", "ORG", "TECH", "CONCEPT", "EVENT"]
  min_confidence: 0.7
  max_entities_per_chunk: 10

# Generation
generation:
  model: "granite4:micro-h"
  temperature: 0.1
  max_tokens: 2048
  streaming: true
  conversation_context_turns: 3
  system_prompt: "You are a helpful AI assistant..."

# Chat UI
ui:
  title: "Knowledge Base Assistant"
  theme: "dark"
  max_conversation_length: 50
  enable_feedback: true
  enable_citations: true

# Monitoring
monitoring:
  enable_metrics: true
  enable_tracing: true
  log_level: "INFO"
  metrics_port: 9090


### 14. Development Roadmap

**Phase 1: Core Enhancements (Week 1)**
- [ ] Implement BM25 sparse search
- [ ] Add cross-encoder re-ranking
- [ ] Enhance graph traversal with multi-hop queries
- [ ] Implement semantic chunking
- [ ] Add conversation history to generation

**Phase 2: Chat UI (Week 2)**
- [ ] Build React frontend with streaming support
- [ ] Implement WebSocket/SSE for real-time responses
- [ ] Add conversation persistence
- [ ] Create source citation UI
- [ ] Implement user feedback mechanism

**Phase 3: Testing & Evaluation (Week 3)**
- [ ] Create evaluation dataset (50+ query-answer pairs)
- [ ] Implement retrieval metrics (recall, precision, MRR)
- [ ] Build hallucination detection
- [ ] Add latency benchmarking
- [ ] Set up automated testing pipeline

**Phase 4: Production Readiness (Week 4)**
- [ ] Implement comprehensive logging and monitoring
- [ ] Add error handling and fallback strategies
- [ ] Set up backup and recovery procedures
- [ ] Create deployment scripts and documentation
- [ ] Conduct load testing and optimization

### 15. Code Quality & Maintainability

**Code Organization:**

project/
├── api/                  # FastAPI endpoints
├── ingestion/           # Document parsing, chunking
├── retrieval/           # Hybrid retrieval logic
├── generation/          # Ollama integration
├── db/                  # Database interfaces
├── ui/                  # Chat UI (React)
│   ├── src/
│   │   ├── components/  # React components
│   │   ├── hooks/       # Custom hooks
│   │   ├── services/    # API clients
│   │   └── utils/       # Utilities
│   └── public/
├── evaluation/          # Testing and metrics
├── monitoring/          # Logging, tracing
├── config/              # Configuration files
└── tests/               # Test suites


**Documentation:**
- Add docstrings to all classes and functions
- Create architecture diagram (Mermaid or draw.io)
- Write API documentation (OpenAPI/Swagger)
- Document configuration options
- Create user guide for chat UI

**Type Hints:**
- Use type hints throughout Python codebase
- Run mypy for static type checking
- Use TypeScript for frontend

**Testing:**
- Unit tests: 80%+ coverage for core logic
- Integration tests: API endpoints, database operations
- End-to-end tests: Full query flow
- Performance tests: Latency, throughput benchmarks

### 16. Deliverables

**Primary Outputs:**
1. Enhanced RAG system with hybrid retrieval
2. Full-featured chat UI with streaming
3. Comprehensive test suite with evaluation metrics
4. Monitoring and observability setup
5. Production deployment guide

**Documentation:**
1. System architecture document
2. API reference (auto-generated from code)
3. User guide for chat UI
4. Deployment and operations guide
5. Troubleshooting guide

**Quality Gates:**
- Retrieval recall@5 > 0.85
- Generation faithfulness > 0.90
- P95 latency < 5s
- Test coverage > 80%
- Zero critical security vulnerabilities

---

## Implementation Notes

**Priority Order:**
1. BM25 + cross-encoder (biggest quality improvement)
2. Chat UI with streaming (user experience)
3. Semantic chunking (better context)
4. Evaluation framework (measure improvements)
5. Production monitoring (operational excellence)

**Common Pitfalls to Avoid:**
- Don't skip evaluation metrics - you can't improve what you don't measure
- Don't ignore chunk boundaries - splits in middle of sentences hurt quality
- Don't over-engineer initially - start with simple hybrid search, iterate based on metrics
- Don't neglect error handling - users will find edge cases
- Don't forget to test with real user queries - synthetic data doesn't capture real patterns

**Success Criteria:**
- System can answer 90%+ of queries accurately (human evaluation)
- Average latency < 3 seconds for complete response
- User satisfaction > 4.0/5.0 (based on feedback)
- Zero data loss incidents
- 99.9% uptime

Now proceed with implementation following this specification. Break down into manageable tasks, implement incrementally, and test thoroughly at each stage.

I want you to take this analysis and your job is to give another LLM a prompt which will instruct the LLM how to formulate a prompt for CLIne which will help solve the problems outlined: it looks like extract_relations.py has problems which need to be fixed. The JSON response parsing is overly complex and fragile. I should add some more logging to see where the entity extraction is failing. I also need to add validation checks to ensure graph data is being created. Or hell, maybe just switching out the inference model will fix it. Sometimes that works. Some models will do things like output only JSON or adhere better to commands than others. I am using the smallest granite4 to test this, maybe I should scale up and get a better outcome. No, I should test first or at least read the code more carefully. ---So the only output I want from you is the prompt to give to the LLM which will instruct CLIne what to do to repair and make the repo associated with this file work, this file is just part of the issue as there are other issues which need detected and addressed so doing things like keeping track of progress and goals in ledger files or strategies like that should be included in the prompt but you should use your knowledge of graph vector rag to help aid in this as well

🧠 Meta-Prompt for LLM (to generate CLIne repair prompt)

You are a senior AI systems engineer who specializes in automated software repair, LLM orchestration, and graph-based retrieval-augmented generation (RAG) pipelines. Your task is to write a CLIne prompt that will repair and enhance a repository containing a file called extract_relations.py.

The code currently has issues with:
	•	Overly complex and fragile JSON parsing.
	•	Entity extraction occasionally failing silently due to poor logging.
	•	Missing validation for graph data creation.
	•	Possibly suboptimal model inference (currently using smallest Granite4 model).

However, these are symptoms, not the full problem set — the repository likely contains other issues that must be discovered and tracked systematically.

The CLIne prompt you generate should:
	1.	Instruct CLIne to analyze the entire repository for structural and logical problems — not just extract_relations.py.
	2.	Have CLIne create and maintain a ledger or strategy file (e.g., DEV_LOG.md or FIX_LEDGER.md) to:
	•	Track identified issues, goals, and fixes.
	•	Record each iteration of changes with a summary of reasoning.
	3.	Tell CLIne to:
	•	Simplify and harden JSON parsing logic using structured error handling.
	•	Add robust logging for every major process step in entity and relation extraction.
	•	Add data validation checks to ensure graph data (nodes/edges) are properly created.
	•	Optionally test alternate inference models (larger Granite or compatible LLMs) for better structured output adherence, but only after logging improvements are verified.
	4.	Incorporate graph-vector RAG principles:
	•	Ensure extracted entities and relations are indexed both in a vector database (for semantic retrieval) and graph database (for relation traversal).
	•	Validate that embeddings and relations are consistently synchronized.
	5.	Generate documentation updates that summarize:
	•	What changed, why, and how it impacts the graph-based RAG workflow.
	•	Instructions for future maintainers to reproduce or extend fixes.
	6.	End with CLIne performing a functional validation pass (e.g., run a small test dataset through extraction and verify graph construction end-to-end).

The tone of your CLIne prompt should be precise, technical, and action-oriented — written as if delegating to a skilled autonomous agent. Include checkpoints or milestones so progress is measurable.

# CLIne Repair Prompt: Graph+Vector RAG System Comprehensive Analysis & Fix

## Mission Statement
You are tasked with conducting a **systematic, multi-phase repair and enhancement** of a production Graph+Vector RAG system. This is not a quick fix—this is a structured engineering effort to harden the codebase, improve observability, and ensure graph-vector synchronization integrity.

---

## Phase 0: Initial Assessment & Strategy Development

**CRITICAL FIRST STEP:** Before making ANY code changes, create a comprehensive development ledger.

### Action Items:
1. **Create `DEV_LOG.md`** in the project root with the following structure:

markdown
# Development & Repair Log
## Session: [Date]
## Engineer: CLIne Autonomous Agent

### Phase 0: Initial Assessment
#### Scan Results:
- [ ] Full repository structure mapped
- [ ] Critical paths identified
- [ ] Dependency analysis complete
- [ ] Configuration validation performed

#### Identified Issues:
1. **[SEVERITY: CRITICAL/HIGH/MEDIUM/LOW]** Issue description
   - File: `path/to/file.py`
   - Lines: X-Y
   - Impact: What breaks, what degrades
   - Root Cause: Technical analysis
   - Proposed Fix: Specific solution strategy

#### Strategic Priorities:
1. [Priority 1 - What to fix first and why]
2. [Priority 2]


### Phase 1: [Phase Name]
[Track progress here]

### Validation Checkpoints:
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] Graph construction validated
- [ ] Vector sync verified


2. **Systematic Repository Scan** - Execute and document:
   bash
   # Analyze these aspects:
   - All Python imports (find circular dependencies)
   - All try-except blocks (identify silent failures)
   - All logging statements (find gaps in observability)
   - All database operations (check transaction safety)
   - All async operations (verify proper error handling)
   - Configuration usage (find hardcoded values)
   

3. **Create Issue Registry** in `DEV_LOG.md`:
   - Extract every TODO, FIXME, XXX comment
   - Flag every `pass` statement in except blocks
   - Identify every function lacking docstrings
   - Note every magic number or hardcoded string

---

## Phase 1: Critical Path Analysis - `extract_relations.py`

### Known Issues to Investigate:

**ISSUE-001: JSON Parsing Fragility**
- **Current Problem:** Multiple regex-based cleaning steps with unclear failure modes
- **Investigation Checklist:**
  
  - [ ] Map all JSON parsing code paths
  - [ ] Test with 10 adversarial inputs (malformed JSON, sensor data, empty responses)
  - [ ] Measure failure rate in production logs (if available)
  - [ ] Identify which LLM models produce most failures
  
- **Repair Strategy:**
  python
  # Implement tiered parsing with clear fallback:
  1. Strict JSON parse (json.loads)
  2. Regex extraction with validation
  3. Structured error with logging
  4. Return empty result (NEVER silent failure)
  

**ISSUE-002: Silent Entity Extraction Failures**
- **Investigation:**
  
  - [ ] Count log statements per 100 lines of code
  - [ ] Identify gaps: where do errors disappear?
  - [ ] Check if entities are stored even when extraction fails
  
- **Repair Strategy:**
  - Add structured logging with correlation IDs
  - Log input text hash, chunk_id, model response, parse success/failure
  - Create extraction metrics (success rate, avg entities per chunk)

**ISSUE-003: Graph Data Validation**
- **Investigation:**

  - [ ] Trace relationship → database storage flow
  - [ ] Check: Are edges created even if nodes don't exist?
  - [ ] Verify: Entity deduplication logic
  - [ ] Test: Orphaned relationships handling
  

### Implementation Requirements:

1. **Structured Logging Framework:**
python
import logging
from typing import Optional
import json

class ExtractionLogger:
    """Context-aware logging for entity/relation extraction."""
    
    def __init__(self, chunk_id: str):
        self.chunk_id = chunk_id
        self.logger = logging.getLogger(f"extraction.{chunk_id}")
        self.context = {"chunk_id": chunk_id}
    
    def log_extraction_start(self, text_hash: str, model: str):
        self.logger.info("Extraction started", extra={
            **self.context,
            "text_hash": text_hash,
            "model": model,
            "stage": "init"
        })
    
    def log_llm_response(self, response: str, is_valid: bool):
        self.logger.info("LLM response received", extra={
            **self.context,
            "response_length": len(response),
            "is_valid_json": is_valid,
            "stage": "llm_response"
        })
    
    def log_parse_failure(self, response: str, error: str):
        self.logger.error("JSON parse failed", extra={
            **self.context,
            "error": error,
            "response_preview": response[:200],
            "stage": "parse_failure"
        })


2. **Hardened JSON Parser:**
python
from typing import Optional, Dict, Any
import json
import re

class RobustJSONParser:
    """Multi-strategy JSON parser with detailed error reporting."""
    
    def parse(self, raw_response: str, logger: ExtractionLogger) -> Optional[Dict[str, Any]]:
        """
        Parse LLM response with tiered strategies.
        
        Returns None only after ALL strategies fail.
        Logs each attempt for debugging.
        """
        strategies = [
            ("strict", self._strict_parse),
            ("cleaned", self._cleaned_parse),
            ("extracted", self._regex_extract_parse),
        ]
        
        for strategy_name, parse_func in strategies:
            try:
                result = parse_func(raw_response)
                logger.log_success(f"Parsed with {strategy_name} strategy")
                return result
            except Exception as e:
                logger.log_attempt_failed(strategy_name, str(e))
        
        # All strategies failed
        logger.log_parse_failure(raw_response, "All strategies exhausted")
        return None
    
    def _strict_parse(self, text: str) -> Dict[str, Any]:
        """Strategy 1: Direct JSON parse."""
        return json.loads(text.strip())
    
    def _cleaned_parse(self, text: str) -> Dict[str, Any]:
        """Strategy 2: Remove markdown/whitespace then parse."""
        cleaned = re.sub(r'^```json\s*|\s*```$', '', text.strip())
        return json.loads(cleaned)
    
    def _regex_extract_parse(self, text: str) -> Dict[str, Any]:
        """Strategy 3: Extract JSON object with regex."""
        match = re.search(r'\{[^{}]*(?:\{[^{}]*\}[^{}]*)*\}', text, re.DOTALL)
        if not match:
            raise ValueError("No JSON object found")
        return json.loads(match.group())


---

## Phase 2: Graph-Vector Synchronization Audit

### Critical Questions to Answer:

1. **Entity Storage Consistency:**
   
   - [ ] Are entities stored in BOTH ChromaDB and SQLite?
   - [ ] Is there a transaction mechanism ensuring atomic writes?
   - [ ] What happens if ChromaDB write succeeds but SQLite fails?
   - [ ] How are duplicate entities handled?
   

2. **Relationship Integrity:**
   
   - [ ] Can relationships reference non-existent entities?
   - [ ] Are orphaned edges cleaned up?
   - [ ] Is there a foreign key validation?
   

3. **Embedding Synchronization:**
   
   - [ ] Do entity embeddings match across databases?
   - [ ] Are embeddings regenerated on entity updates?
   - [ ] How is stale data detected?
   

### Implementation Requirements:

**Create `ingestion/validators.py`:**
python
from typing import List, Dict, Any, Set, Tuple
import logging

class GraphDataValidator:
    """Validates graph data integrity before/after storage."""
    
    def validate_entities(self, entities: List[Dict[str, Any]]) -> Tuple[List[Dict], List[str]]:
        """
        Returns: (valid_entities, error_messages)
        
        Checks:
        - Required fields present
        - Types correct
        - Confidence in range [0, 1]
        - No duplicate IDs
        """
        valid = []
        errors = []
        seen_ids = set()
        
        for entity in entities:
            # Validation logic here
            if self._is_valid_entity(entity, seen_ids):
                valid.append(entity)
            else:
                errors.append(f"Invalid entity: {entity.get('name', 'unknown')}")
        
        return valid, errors
    
    def validate_relationships(
        self, 
        relationships: List[Dict[str, Any]], 
        valid_entity_ids: Set[str]
    ) -> Tuple[List[Dict], List[str]]:
        """
        Validates relationships reference existing entities.
        
        Returns: (valid_relationships, error_messages)
        """
        # Implementation here
        pass
    
    def validate_graph_consistency(
        self, 
        entities: List[Dict], 
        relationships: List[Dict]
    ) -> Dict[str, Any]:
        """
        Comprehensive graph consistency check.
        
        Returns report with:
        - orphaned_relationships
        - isolated_entities
        - missing_embeddings
        - duplicate_edges
        """
        pass


---

## Phase 3: Model Inference Optimization

### Diagnostic First:

Before changing models, establish baseline:

python
# Add to extract_relations.py

class ExtractionMetrics:
    """Track extraction performance metrics."""
    
    def __init__(self):
        self.total_attempts = 0
        self.successful_extractions = 0
        self.json_parse_failures = 0
        self.empty_responses = 0
        self.avg_entities_per_chunk = 0.0
        self.model_response_times = []
    
    def get_success_rate(self) -> float:
        return self.successful_extractions / self.total_attempts if self.total_attempts > 0 else 0.0
    
    def should_try_larger_model(self) -> bool:
        """Decision logic for model upgrade."""
        return self.get_success_rate() < 0.7 and self.json_parse_failures > 10


### Model Testing Protocol:

1. **Establish Baseline:**
   
   - [ ] Run current model (granite4:micro-h) on 100 test chunks
   - [ ] Log: success rate, parse failures, avg response time
   - [ ] Document in DEV_LOG.md
   

2. **Test Larger Model** (only if baseline is poor):
   python
   # config.yaml
   llm:
     entity_model: "granite4:8b"  # or "llama2:7b"
     entity_extraction_timeout: 120  # Allow more time
   

3. **A/B Comparison:**
   
   - [ ] Same 100 test chunks with new model
   - [ ] Compare: accuracy, speed, resource usage
   - [ ] Document trade-offs in DEV_LOG.md


---

## Phase 4: End-to-End Validation Suite

### Create `tests/test_graph_rag_pipeline.py`:

python
import pytest
from pathlib import Path
from db.ingest_data import DataIngester
from rag.retrieve import HybridRetriever
from rag.generate_answer import AnswerGenerator

class TestGraphRAGPipeline:
    """Comprehensive end-to-end tests."""
    
    def test_ingestion_creates_graph_and_vectors(self):
        """Verify both graph nodes and vector embeddings are created."""
        ingester = DataIngester()
        
        # Ingest test document
        stats = ingester.ingest_file("tests/fixtures/sample.md")
        
        # Validate graph data
        assert stats["entities"] > 0, "No entities extracted"
        assert stats["relationships"] > 0, "No relationships extracted"
        
        # Validate vector data
        db = ingester.db_interface
        chunks = db.get_all_chunks()
        assert len(chunks) > 0, "No chunks in ChromaDB"
        assert all("embedding" in c for c in chunks), "Missing embeddings"
    
    def test_graph_vector_synchronization(self):
        """Ensure entity IDs match across databases."""
        # Implementation here
        pass
    
    def test_retrieval_uses_both_graph_and_vector(self):
        """Verify hybrid retrieval leverages both data sources."""
        retriever = HybridRetriever()
        results = retriever.retrieve("test query", top_k=5)
        
        # Check that results include graph-derived context
        assert any("graph_traversal" in r.get("retrieval_method", "") for r in results)
        
        # Check that results include vector similarity
        assert any("vector_similarity" in r.get("retrieval_method", "") for r in results)


---

## Phase 5: Documentation & Knowledge Transfer

### Update Documentation:

1. **Create `docs/EXTRACTION_ARCHITECTURE.md`:**
   markdown
   # Entity & Relationship Extraction Architecture
   
   ## Data Flow
   [Diagram: Text → LLM → JSON → Validation → Graph+Vector Storage]
   
   ## Error Handling Strategy
   [Document the tiered parsing approach]
   
   ## Monitoring & Debugging
   [How to read logs, interpret metrics]
   
   ## Troubleshooting Guide
   - Symptom: Low extraction success rate
     - Check: Model inference timeout
     - Check: Prompt quality
     - Solution: [steps]
   

2. **Update `README.md`:**
   - Add section on monitoring extraction quality
   - Document new validation tools
   - Explain graph-vector sync guarantees

3. **Create Runbook `docs/OPERATIONS.md`:**
   markdown
   ## Health Checks
   - Check extraction success rate: `python -m monitoring.extraction_health`
   - Validate graph consistency: `python -m db.validate_graph`
   
   ## Common Issues
   ### Issue: Entities not appearing in graph
   - Check: SQLite entity table
   - Check: ChromaDB entity collection
   - Resolution: Run sync repair tool
   

---

## Phase 6: Validation Checkpoint

### Execute Before Declaring Success:

bash
# 1. Run full test suite
pytest tests/ -v --tb=short

# 2. Run extraction on real data
python scripts/test_extraction_quality.py --input documents/ --sample 50

# 3. Validate graph consistency
python -m db.validate_graph --repair

# 4. Check system health
python -m monitoring.observability

# 5. Generate extraction report
python scripts/generate_extraction_report.py > extraction_report.txt


### Success Criteria:

- [ ] All tests pass
- [ ] Extraction success rate > 80% on sample data
- [ ] Zero silent failures (all errors logged)
- [ ] Graph consistency validation passes
- [ ] No orphaned relationships
- [ ] Entity-vector synchronization verified
- [ ] Documentation complete and accurate

---

## Execution Protocol

### Work Style:
- **Think → Document → Code → Validate → Document Again**
- Update `DEV_LOG.md` after EVERY phase
- If you discover new issues mid-phase, add them to the log and adjust priorities
- Never make "quick fixes" without documenting rationale

### Commit Strategy:

feat(extraction): Add structured logging framework
fix(extraction): Harden JSON parsing with tiered strategies
test(pipeline): Add end-to-end graph-vector sync tests
docs(architecture): Document extraction error handling


### Final Deliverable:
A pull request with:
1. All code changes
2. Complete `DEV_LOG.md` with:
   - All issues found and resolved
   - Metrics before/after
   - Trade-offs made
3. Updated documentation
4. Passing test suite
5. Extraction quality report

---

## Begin Execution

Start with Phase 0. Do not proceed to Phase 1 until `DEV_LOG.md` exists and the initial assessment is documented. Your first response should be the completed Phase 0 assessment.