Увидел интересное репо, в котором автор собрал локальный опенсорсный стек:
Data Forge includes a complete modern data stack with industry-standard tools:
🗄️ Storage & Catalog
- MinIO → S3-compatible object storage for data lakes
- Hive Metastore → Centralized metadata catalog for tables and schemas
⚡ Compute Engines
- Trino → Interactive SQL query engine for federated analytics
- Apache Spark → Distributed processing for batch and streaming workloads
🌊 Streaming & CDC
- Apache Kafka → Event streaming platform
- Schema Registry → Schema evolution and compatibility
- Debezium → Change data capture from databases
🗃️ Databases
- PostgreSQL → Primary OLTP database (source system)
- ClickHouse → Columnar analytics database (sink)
🔄 Orchestration
- Apache Airflow 3 → Workflow orchestration
📊 Visualization & Exploration
- Apache Superset → Modern BI and data visualization
- JupyterLab → Interactive data science environment
Идеальный стек для отечественного (СНГ) дата инженера.
PS автору если интересно, может и вебинарчик провести для нас.
Forwarded fromИнжиниринг Данных
13October 3, 2025 1.5K 45
