We keep coming back to SQL

In 2012 the answer to every data question was a Hadoop cluster. Didn’t matter what the question was. You had logs, you wanted a report, someone would say “we should put that in Hadoop”, and six months later there would be a rack of machines running HDFS and a team writing MapReduce jobs in Java to count things.

I don’t want to be too harsh about it. Hadoop did something real: it made it normal to keep all the data instead of sampling it, on cheap disks, and to bring the computation to the data. The “big data” phrase was mostly marketing, but the storage part was true. What hurt was everything on top. Hive let you write SQL that turned into MapReduce, so a query that should take a second took four minutes because it had to spin up a job first. Cloudera and Hortonworks sold you support for all of it, and there was a whole zoo of side projects with animal names. It worked, and it was miserable.

Spark, Kafka and the warehouses that fought back

Spark came out of Berkeley and became an Apache project in 2013, with 1.0 in 2014. It kept data in memory between steps and had a real API in Scala and Python, so the same code could do batch and, sort of, streaming. It became a common alternative to MapReduce. Databricks, the company the Spark people started, built around it.

Kafka was the other thing from that period that stuck. LinkedIn built it, open-sourced it, and it became the default answer for “how do events get from here to there”. It seems like every company has a Kafka cluster now, and about half of them use it as a slightly odd database.

Meanwhile the warehouse people had not gone away. Amazon Redshift became available in early 2013 and it was a proper columnar SQL warehouse you could rent by the hour. Google had BigQuery, which was even stranger: no cluster at all, just send SQL and pay per query. And then Snowflake came out of stealth in 2014 with storage and compute split apart. It sounds dull, but ten teams could each have their own compute cluster hitting the same tables and nobody stepped on anybody. By 2016 or so I think the verdict was in: for most companies, a warehouse with SQL beats a Hadoop cluster with everything.

The tooling years

Two tools from this period changed what a data team did all day.

Airflow, open-sourced by Airbnb in 2015, turned scheduling into code. Your pipeline is a Python file with a DAG in it, and the scheduler runs the tasks in order and retries the failed ones. It gave teams a shared alternative to their own scheduling scripts. People complain about it constantly. It is still everywhere.

dbt, from around 2016 and growing seriously from 2018, is the one I have more affection for. The idea is almost embarrassingly small: your transformations are SELECT statements in files, with Jinja templates for the repetitive bits, and dbt works out the order, runs them in the warehouse and tests the results. That’s it. But it turned SQL into something with version control and tests, and it created a job title, the analytics engineer, for the person who does that.

Put Fivetran (or Airbyte) for loading, Snowflake or BigQuery for storage, dbt for transformation, Airflow for orchestration and Looker or Metabase on top, and you have the “modern data stack”. Around 2020 and 2021 that phrase was everywhere. Snowflake had the biggest software IPO ever in 2020, and there were more data conferences than data problems. It was a bubble, and I liked working with the stack.

Lakehouse, then small again

The warehouse won on convenience, and then the bill arrived. Storing everything in a proprietary format and paying for compute on every query gets expensive when “everything” is petabytes. So the pendulum swung back toward files on object storage, only this time with table formats that make a pile of Parquet files behave like a database: transactions, schema changes, time travel, and files that don’t get corrupted when two jobs write at once. Databricks open-sourced Delta Lake in 2019 and started pushing the word “lakehouse” in 2020. Apache Iceberg came out of Netflix and became the neutral option that everybody could agree on, and by 2024 both Snowflake and Databricks were competing to be the best Iceberg catalog, which I did not expect. I’m less interested in declaring a winner than in being able to move the data later.

The other swing was toward small. DuckDB is an in-process analytical database, like SQLite for columns. It arrived around 2019 and hit 1.0 in 2024, and it made a lot of people notice that their “big data” fits on a laptop. I’d try the single-machine version before assuming a job needs a cluster. I like it a lot. Polars did something similar for dataframes.

Vectors and agents

2023 was the year of the vector database. Pinecone, Weaviate, Milvus, Chroma, and then vector indexes started appearing in the databases people already used. Data engineers suddenly had a new job: chunk the documents and embed them, then keep the index fresh. It’s an ingestion pipeline with an extra step. The same Airflow DAGs, basically.

And by 2025 the consumer of the data started to change. It’s still dashboards and it’s still analysts, but more and more it’s an agent, a model with tools, asking the warehouse a question on behalf of someone. That puts pressure on things the modern data stack never fully solved: the semantic layer (what does “revenue” mean, exactly) and metadata that a model can actually read. The Model Context Protocol from late 2024 gave agents a standard way to plug into databases and tools, and by 2026 the interesting data engineering work I see is about making a warehouse legible to a model. Good column names and tested definitions. The boring stuff, again.

Fourteen years, and the winning move at every step was to make it easier to write SQL. I don’t think that changes.