Data, Analytics and AI
Data engineering and pipelines
Every dashboard and model depends on records arriving on time, complete and in the expected shape, and data engineering is the discipline that makes that happen while nobody is watching. Ingestion, transformation, warehouses and lakehouses, and orchestration are the subject here, along with how the AWS, Microsoft and Databricks exams divide the work between them, and why the best pipelines are the dull ones.
Why this topic exists: Ingestion, transformation, warehouses, lakehouses and orchestration (DMBOK Data Integration and Interoperability; DEA-C01; DP-700; Databricks; Spark, Airflow, dbt).
Data engineering is supply-chain work. Source systems produce records for their own purposes; the engineer moves those records into a store built for questions, reshapes them into tables analysts can trust, and keeps the flow running while sources add columns, change formats or fall silent without warning. SFIA's description of the skill runs from design through building and operation to security and monitoring, and that order is honest about where the time goes: a design takes a week, while monitoring never ends.
DMBOK places most of this under Data Integration and Interoperability, with neighbours in Data Storage and Operations as well as Data Warehousing. DP-900 introduces the vocabulary at fundamentals level in its analytics workload area, and the associate exams test it in depth on one platform each.
The four jobs inside a pipeline
- Ingestion. Pulling data from applications, files, APIs and event streams, either in scheduled batches or continuously. Change data capture, which reads a database's own log of changes, avoids copying whole tables every night.
- Transformation. Turning raw copies into modelled tables. Loading first and transforming inside the warehouse, known as ELT, has largely displaced transforming on the way in, because warehouse compute became cheap and SQL-based tools such as dbt made the logic reviewable.
- Storage. A warehouse holds structured, modelled tables for SQL; a data lake holds files of any kind cheaply; a lakehouse lays table formats and transactions over lake storage so that one copy serves analysts and data scientists alike. Spark does the heavy processing on most lake platforms.
- Orchestration. Running steps in the right order, retrying failures and backfilling history. Airflow is the best-known general orchestrator, and most platforms ship one of their own.
Exam coverage, and the failure it misses
AWS's DEA-C01 examines ingestion, transformation, storage and governance across its services. Microsoft's DP-700 does the same inside Fabric, lakehouse and warehouse included, and the Databricks associate concentrates on the lakehouse, incremental processing and the governance layer above them. All three prove you can find your way round a platform. None proves the habit that matters most, which is designing pipelines to fail loudly. The typical beginner's error is a job that succeeds while loading nothing: the source delivered an empty file, every step reported green, and the dashboard showed a quiet day. Row-count checks, freshness alerts and tests on key columns are tedious to build, and they are the difference between an engineer colleagues trust and one they double-check.
Where the work starts
Developers arrive with the habits pipelines lack: version control, automated tests, code review. Analysts arrive knowing what the tables are for. Both need SQL well beyond reporting level and a clear picture of how a warehouse bills for compute. Designing the operational database itself belongs to databases and SQL in the programming hub, and the storage services to cloud storage and databases.
Next to this topic
- Data analysis fundamentalsAsk, prepare, process, analyse, share: the analyst's workflow as Google's programme and DP-900's core data concepts teach it, before any tool (SFIA Data analytics).
- SQL for analysisQuerying, joining, aggregating and windowing data is the analyst's first real tool and the language of every warehouse (DP-900 relational concepts; DMBOK Data Storage and Operations; the programming field owns database design).
- Data visualisation and business intelligenceDashboards, reports and the semantic models behind them: PL-300's four skill areas and DMBOK's Data Warehousing and BI area (Tableau, Power BI, Looker).
- Statistics and experimentationProbability, inference, regression and A/B tests separate a chart from a claim (Google's Advanced Data Analytics; SFIA Data science).
- Python for data workpandas, NumPy, notebooks and plotting: the working environment of analysis and machine learning (SFIA Data science; the AI pillar's argument for Python).
- Data governance, quality and privacyOwnership, definitions, metadata, master data, quality rules and lawful use are DMBOK's centre and half its wheel; nobody hires for them until the numbers disagree.
- Machine learningSupervised and unsupervised learning, model evaluation, deep learning and the discipline of not fooling yourself (SFIA Machine learning; AIF-C01; Azure AI Fundamentals).
- Generative AI, LLMs and agentsFoundation models, prompting, retrieval, fine-tuning and agents: the newest layer of the field and the one AIF-C01 and Azure AI Fundamentals now test (the AI portal reads the concepts in order).
- Natural language processing and computer visionText and images are the two applied fields where classic machine learning met deep learning first, and where most production AI still lives (SFIA Machine learning; Azure AI Fundamentals AI-901 workloads).
- Responsible AI and AI governanceBias, transparency, privacy, safety and the frameworks that make them work (NIST AI RMF; AIF-C01's responsible-AI domain; SFIA 9's AI and data ethics skill; the EU AI Act).
- MLOps and AI engineeringGetting a model or an LLM application into production and keeping it there: deployment, monitoring, evaluation, cost (SFIA Machine learning and Data engineering; Applied Skills in Foundry; MLOps in the AI portal).
Concepts to know
Glossary entries with the reason each one matters here.
- Data Engineering
The topic's own concept.
- SQL
The transformation language of most pipelines.
- Object Storage
Where data lakes live.
- Containers
How pipelines are packaged and run.
Certifications that test it
Vendor exams and free certificates; facts, cost and the preparation path are on each page, and the certifications hub has them all.
Tools of the trade
- Apache Spark
Distributed processing for pipelines.
- Databricks
The lakehouse platform.
- Snowflake
The cloud warehouse.
- dbt
The transformation layer of the warehouse stack.
Frequently asked
- Is data engineering a good first job in data?
- More often it is a second one. Many data engineers were analysts tired of repairing their inputs, or developers who drifted towards data. Starting cold is possible with strong SQL and one cloud platform, but a year spent close to the data first makes pipeline design much easier to judge.
- Do I need to learn Spark?
- Only when data outgrows a single machine or the platform you work on is built around it, as Databricks and Fabric are. Plenty of companies' data fits comfortably in a cloud warehouse queried with SQL. Learn the ideas of partitioning and distributed joins early, and Spark itself when a job calls for it.
- What is the difference between ETL and ELT?
- The order of the last two letters. ETL transforms data before loading it into the target, which suited warehouses where compute was expensive. ELT loads raw copies first and transforms them inside the warehouse, which keeps the raw history available for reprocessing and puts the logic where analysts can read it.
Courses in the directory
268 courses are filed here; the top 6 by our ranking, details and the provider link on each course page.
Last reviewed 26 September 2026 · Getting Digital