I spent a weekend going through 200 data engineer job postings on LinkedIn and Indeed. Senior roles, mid-level roles, startups, big tech, consulting firms. All of them.
I wanted to answer one question: what tools actually matter?
Not what Twitter argues about. Not what some influencer is hyping. What do companies actually put in their job descriptions when they are hiring and spending real money?
Here is what I found.
1. SQL
Every single posting mentioned SQL. Not some of them. All of them. If you are not comfortable writing complex queries — window functions, CTEs, subqueries, joins across multiple tables — nothing else on this list matters. SQL is the foundation. It is not optional.
2. Python
About 90 percent of postings mentioned Python. Not for building web apps. For writing data pipelines, automating tasks, working with APIs, and scripting. Libraries like Pandas, requests, and boto3 came up the most. You do not need to be a software engineer. You need to be dangerous enough to automate anything.
3. A Cloud Platform (AWS, GCP, or Azure)
Almost every posting required experience with at least one cloud platform. AWS was the most common, followed by GCP and Azure. The specific services that kept showing up: S3 or GCS for storage, BigQuery or Redshift for warehousing, Lambda or Cloud Functions for serverless, and IAM for access control. You do not need to know all three clouds. Pick one and go deep.
4. A Data Warehouse (Snowflake, BigQuery, or Redshift)
Roughly 80 percent of postings required hands-on experience with a modern cloud data warehouse. Snowflake was mentioned the most, followed by BigQuery. These are where your transformed, business-ready data lives. You need to know how to design schemas, optimize queries, manage costs, and handle partitioning.
5. dbt
This one surprised me a few years ago, but not anymore. dbt showed up in about 60 percent of postings and it is growing fast. Companies want engineers who can write modular, tested, version-controlled SQL transformations. If you do not know dbt yet, learn it now. It is becoming a requirement, not a nice-to-have.
6. Apache Airflow
About 55 percent of postings mentioned Airflow or a similar orchestration tool. Companies need someone who can schedule pipelines, manage dependencies between tasks, handle retries, and monitor job health. Airflow is the most requested, but Dagster and Prefect are gaining ground. Know at least one.
7. Kafka or a Streaming Tool
Around 40 percent of postings mentioned real-time data processing — and Kafka was by far the most common tool. This was more common in mid-to-senior roles and at companies with high-volume data. If you want to stand out from the crowd, learn the basics of event streaming.
———
Here is the takeaway. You do not need to learn 50 tools. You need to go deep on these 7. Every one of them showed up consistently across industries, company sizes, and experience levels. Master these, and you will match the requirements of the vast majority of data engineering roles out there.
Want to become a data engineer or level up your career? I run a coaching program where I help people break into data engineering with real projects, interview prep, and a clear roadmap. Book a free call with me, RemoteDataBlueprint.