Overview

This project is a fault-tolerant, containerized trading engine that integrates real-time data ingestion, processing, and machine learning for alpha generation.

Key highlights:

• High-Throughput Streaming: Engineered the system using Kafka and ClickHouse to ingest 70K+ ticks/minute on AWS with sub-second latency.
• Automated MLOps Pipeline: Architected a pipeline for real-time feature engineering and continuous model retraining to adapt to current market conditions.
• Execution Engine: Executes trades based on machine learning, generating signals with integrated risk and portfolio management for alpha generation.
• Efficient Storage: Maintains a rolling 30-day window of high-frequency data while archiving older data to the cloud.

Additionally, I took away several lessons from building this:

• Designing for constant uptime: Long-running use surfaced unexpected failures, underscoring the need for proactive resource monitoring, alerting, and graceful restarts.
• Scaling from prototype to production: Early ad hoc decisions led to painful rework (such as migrating to Docker and consolidating services), so I now design with production in mind from the outset.
• Building for modular growth: A monolithic first pass was quick to ship but became a bottleneck as the system grew, reinforcing the value of clear module boundaries and interfaces early on.

Architecture

DigitalOcean — hosts the server on a $48/month droplet with 8GB RAM, 4 vCPUs, and 160GB SSD storage.

Docker — containerization platform for Kafka, ClickHouse, Grafana, Prometheus, and all project modules.

Grafana — powers all dashboards for real-time monitoring, visualization, and alerting.

Binance — source for cryptocurrency tick-level data, ingested in real-time via WebSocket through Finnhub.

Kafka — captures all WebSocket data via a producer/broker topology, accessed by various consumers downstream.

ClickHouse — connected to the data stream via Kafka, with batch appends and periodic monitoring snapshots.

AWS S3 — cold storage for data older than 7 days, converted to Parquet before archival.

• The machine learning module retrains five models at regular intervals using scikit-learn and TensorFlow.
• The execution module loads these ML models to generate real-time signals and simulate trades, supported by risk and portfolio monitoring engines.

Dashboards

Portfolio performance after running for ~30 days.

Trade executions with the risk engine setting trade limits.

ML models implemented in the project.

Time series of granular ETH price ticks (alongside BTC, SOL, ADA, XRP) via Kafka/ClickHouse.

Memory usage by Docker container.

Writing

Check out my Substack for more.