Multimodal Story Generation System

Transform visual inputs into structured narratives using cutting-edge AI technologies. This system combines computer vision and large language models to generate dynamic, multi-chapter stories from images.

Features

• 🖼️ Image Analysis - Extract narrative elements from images using LLaVA
• 📖 Adaptive Story Generation - Generate 5-chapter stories with Gemma2-27B
• 🧠 Context Awareness - Maintain narrative consistency with ChromaDB RAG
• 📊 Interactive Visualization - ReactFlow-powered story graph interface
• 🚀 Production Ready - Dockerized microservices architecture

Table of Contents

• Quick Start
• System Requirements
• Architecture
• Production Deployment
• Troubleshooting

Quick Start

Local Development Setup

1. Clone Repository

   bashCopy
   1git clone https://github.com/kliewerdaniel/ITB02
   2cd ITB02

2. Create Virtual Environment

   bashCopy
   1python -m venv venv
   2source venv/bin/activate  # Linux/Mac
   3venv\Scripts\activate     # Windows

3. Install Dependencies

   bashCopy
   1pip install -r requirements.txt
   2
   3# Apple Silicon Special Setup
   4pip install --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cpu
   5brew install libjpeg webp

4. Initialize AI Models

   bashCopy
   1ollama pull gemma2:27b
   2ollama pull llava

5. Start Services

   bashCopy
   1# Backend (FastAPI)
   2uvicorn backend.main:app --reload
   3
   4# Frontend (new terminal)
   5cd frontend
   6npm install && npm run dev

6. Verify Installation

   bashCopy
   1curl http://localhost:8000/health
   2# Expected response: {"status":"healthy"}

System Requirements

• Python 3.11+
• Node.js 18+
• Ollama runtime
• 16GB RAM (24GB+ recommended for GPU acceleration)
• 10GB+ Disk Space

Architecture

textCopy
1[Frontend] ←HTTP→ [FastAPI]  
2                 ↓     ↑  
3              [Ollama] ←→ [ChromaDB]  
4                 ↓  
5              [Redis]  
6                 ↓  
7            [Celery Workers]

Key Components

| Component | Technology Stack | Function | |---------------------|------------------------|------------------------------------| | Image Analysis | LLaVA, Pillow | Visual narrative extraction | | Story Engine | Gemma2-27B, LangChain | Context-aware chapter generation | | Knowledge Base | ChromaDB | Narrative consistency management | | API Layer | FastAPI | REST endpoint management | | Visualization | ReactFlow, Zustand | Interactive story mapping |

Production Deployment

Docker Setup

bashCopy
1# Build and launch all services
2docker-compose up --build
3
4# Initialize vector store
5docker exec -it backend python -c "from backend.core.rag_manager import NarrativeRAG; NarrativeRAG()"

Cluster Configuration

yamlCopy
1# docker-compose.yml excerpt
2services:
3  ollama:
4    deploy:
5      resources:
6        limits:
7          memory: 12G
8          cpus: '4'

Troubleshooting

Common Issues

1. Missing Vector Store

   bashCopy
   1rm -rf chroma_db && mkdir chroma_db

2. Out-of-Memory Errors

   bashCopy
   1export OLLAMA_MAX_LOADED_MODELS=2

3. CUDA Compatibility Issues

   bashCopy
   1pip uninstall torch
   2pip install torch --extra-index-url https://download.pytorch.org/whl/cu117

---

Daniel Kliewer
GitHub Profile
AI Systems Developer