Live
2023 – 2024Real-Time Financial News Sentiment Crawler
Asynchronous pipeline for ingesting finance headlines and sentiment analysis
Role
Creator / Backend Engineer
Period
2023 – 2024
Category
infra
Overview
A resilient async pipeline for collecting financial news from multiple RSS feeds (Reuters, CNBC), performing real-time sentiment analysis using Hugging Face transformers, and exporting clean JSON feeds for quant workflows. Features graceful fallbacks, robust XML/HTML parsing, persistent SQLAlchemy storage, and comprehensive test coverage with pytest.
Key Highlights
- Async RSS ingestion with graceful fallbacks and resilience
- Transformer-based sentiment analysis with lazy loading
- Robust parsers handling XML/HTML inconsistencies across sources
- Persistent SQLAlchemy ORM storage with clean PostgreSQL upgrade path
- JSON feed export for dashboard, n8n, and trading bot integration
Tech Stack
Python
aiohttp
SQLAlchemy
PostgreSQL
Hugging Face Transformers
Pydantic
pytest
Links
What I Learned
This project deepened my understanding of Python and aiohttp and reinforced best practices in system design and scalability. I gained valuable insights into production-grade development and performance optimization.