Technical Architecture & Benchmarks
"Every business question creates a technical problem." How the Medallion Pipeline handles tens of millions of records and at what scale Spark becomes necessary.
Pipeline Strategy
Medallion
Bronze → Silver → Gold
Query Engine
DuckDB
In-process OLAP vectorization
Storage Format
Parquet
Columnar compression
Spark Crossover
50 GB+
When cluster scaling wins
Data Pipeline Architecture
Kaggle offline processing → Next.js static JSON delivery
Raw Data
NYC TLC Parquet
AWS S3 public bucket (~3-5 GB compressed)
→
Bronze
Ingestion & Schema
Drops corrupt rows & validates schema types
→
Silver
Cleaning & Enrichment
Filters outliers, joins zone maps, derives metrics
→
Gold
DuckDB Aggregation
Vectorized SQL exports pre-computed JSON
Benchmark Engine ("How Big Is Actually Big?")
Evaluating Pandas vs. DuckDB vs. PySpark across data volume tiers