Platform · Data & AI

Apache Spark

Apache Spark spreads data processing across a cluster so that transformations too big for one machine run in parallel, from batch ETL to streaming and machine learning, driven from Python, SQL, Scala, Java or R. Data engineers build pipelines with it, usually through Databricks or a cloud service. Most data is smaller than people assume, and for many jobs one machine with DuckDB or pandas is faster.

Maker's site: spark.apache.org (opens in a new tab) (no commission).