Data platform · Lakehouse
Status: LiveOpen Finance LakeHouse
51 financial series, batch and streaming, on an entirely open stack.
The foundation the rest of the portfolio is built on. Public Brazilian macro and market data lands as Delta on MinIO, is conformed by Spark into a star schema and served as DuckDB marts — every lane generated from one source registry. Adding a series is a YAML entry, not new code.
- 51 registered series across 10 source handlers (BACEN, IPEA, IBGE, Tesouro, ANBIMA, B3, Yahoo Finance) — medallion bronze → silver → gold on Delta Lake over MinIO, ending in 8 BI-ready marts.
- Airflow 3 Asset scheduling derived from the registry at parse time — 13 DAGs and 51 bronze assets, one task per series, so a failing API withholds exactly one asset and raises exactly one attributable alert.
- Spark
- Structured Streaming
- Delta Lake
- dbt
- Airflow 3
- MinIO
- Polars
- DuckDB
- Pandera
- OpenLineage
- OpenMetadata
- Kubernetes
Evidence
The Kubernetes/GitOps layer that deploys this stack lives in a separate repository that holds cluster credentials. It stays private permanently and is not linked from here.
