Ingesta
High-throughput data ingestion pipeline with configurable batch processing and dead-letter queuing.
Highlights
- Implemented configurable batch insert controls to improve reliability for high-volume ingestion.
- Added dead-letter queue with automatic retry and exponential backoff.
- Kafka consumer groups with manual offset commit for at-least-once delivery.
- Monitoring dashboard via Prometheus metrics and Grafana.