AI Portfolio Assistant
Intelligent voice-enabled chatbot with RAG, speech recognition, and natural language understanding
Voice capture state with particle field
Call-to-action panel that opens the assistant
Overview
This AI chatbot serves as an interactive portfolio assistant, allowing visitors to ask questions about my skills, projects, and experience through text or voice. Built with modern AI technologies, it combines retrieval-augmented generation (RAG), speech recognition, and text-to-speech for natural conversations. Midway through the build I swapped the XTTS + whisper.cpp stack for Piper + Faster-Whisper and pinned the Docker dependencies (torch, sox, ALSA) so deployments stop breaking every time upstream images change.
Problem
Visitors needed an easy way to learn about my work without reading through multiple pages
Solution
AI-powered chatbot with voice interaction and contextual understanding of portfolio content
Impact
50% reduction in bounce rate, instant answers to common questions, memorable user experience
Technical Architecture
Backend (FastAPI + Python)
- ▸RAG Pipeline: FAISS vector database for semantic search across portfolio content, FAQs, and project descriptions
- ▸LLM Integration: Ollama running Qwen 2.5 3B Instruct (Q4 quantized) for fast local inference without API costs
- ▸Speech-to-Text: Faster-Whisper (base model) with voice activity detection for accurate transcription
- ▸Text-to-Speech: Piper TTS (low-latency mode) for sub-second audio generation
Frontend (Next.js + React)
- ▸Floating Widget: Persistent chat interface with smooth animations and accessibility features
- ▸Voice UI: WebRTC audio recording with visual feedback and automatic silence detection
- ▸Graceful Degradation: TTS availability check with text-only fallback
Infrastructure
- ▸Docker Compose: Multi-container setup (Ollama, FastAPI, Nginx reverse proxy)
- ▸DigitalOcean Droplet: 2GB RAM, 2 vCPUs, sufficient for CPU-only inference
- ▸SSL/TLS: Let's Encrypt certificates on api.ayushv.dev
Key Features
Voice Interaction
Click-to-talk interface with real-time transcription. Supports multiple languages and accents through Whisper's multilingual model.
- • Automatic silence detection
- • Visual recording indicator
- • Voice activity detection (VAD)
Contextual Understanding
RAG-powered responses retrieve relevant context from portfolio content before generating answers.
- • Semantic search via FAISS
- • Top-K retrieval (configurable)
- • Source attribution in logs
Low Latency
Optimized for speed with model caching, Piper TTS (sub-second generation), and local inference.
- • TTS pre-loading on startup
- • FAISS index caching
- • Async API endpoints
Admin Dashboard
Token-protected endpoints for managing content, viewing analytics, and reindexing knowledge base.
- • Query chat logs (SQLite)
- • Reindex FAISS data
- • Ingest new documents
Technical Challenges
Challenge: TTS Library Compatibility
Initial deployment failed with libtorchaudio.so errors due to missing system dependencies and XTTS requiring GPU drivers.
Solution: Added sox, libsox-dev, and alsa-utils to Docker image, swapped XTTS for Piper's ONNX runtime, and pinned torch/torchaudio to version 2.1.0 for compatibility. Implemented graceful fallback with TTS availability check in health endpoint.
Challenge: Response Quality
LLM responses contained markdown formatting (**, _, #) causing poor TTS pronunciation and cluttered UI.
Solution: Built text cleaning pipeline removing special characters, converting markdown links to plain text, and formatting paragraphs. Applied before both display and TTS synthesis.
Challenge: Docker Build Caching
Config module changes weren't picked up despite rebuilding, causing ModuleNotFoundError.
Solution: Fixed COPY path in Dockerfile (from COPY . /app/backend to COPY . /app) to prevent double nesting. Created --no-cache deployment script for critical updates.
Results & Impact
Average response time for text chat (including RAG retrieval + LLM inference)
TTS audio generation time using Piper low-latency mode
API costs - fully self-hosted with Ollama local inference
FAQ topics covered with semantic search accuracy
Technology Stack
Backend
Frontend
Lessons Learned
- ✓Start with low-latency models: Piper TTS proved much faster than XTTS v2 for real-time use. Voice quality trade-off was acceptable.
- ✓Docker caching gotchas: Always use
--no-cachefor critical dependency changes. Saved hours of debugging. - ✓Text cleaning is essential: LLMs output markdown by default. Clean before TTS to avoid pronunciation issues.
- ✓Health checks matter: Expose service availability flags (
tts_available) for graceful frontend degradation. - ✓Local LLMs are viable: Qwen 2.5 3B (Q4) provides good quality at ~500ms latency on 2 vCPUs with zero API costs.
Future Enhancements
Planned
- • Streaming TTS for faster perceived response
- • Multi-turn conversation memory
- • Voice activity detection visualization
- • Rate limiting on API endpoints
Ideas
- • Multiple voice options (male/female)
- • Background noise suppression
- • Conversation analytics dashboard
- • Export chat transcripts
Try the AI Assistant
Experience the chatbot live on the homepage - ask about my projects, skills, or technical approach!