Data engineers
Data engineers make data arrive: on time, in the shape it was promised, with a record of where it came from. Reporting, modelling and data-driven product features all inherit whatever this layer produces, including its mistakes. This guide covers what the discipline involves and what separates a candidate who has operated a platform from one who has built pipelines that were never wrong yet.
What does a data engineer do?
A data engineer builds and operates the systems that carry data from where it is created to where it is used. The work spans ingestion from applications, third-party services and event streams; transformation into models that analysts and downstream systems can depend on; scheduling and dependency management; the agreements about structure and meaning that hold between producing and consuming teams; automated quality checks; lineage that lets any figure be traced back to source; and the storage and compute decisions that determine what the platform costs to run each month. Success is not that data exists somewhere, but that it is correct, timely and explicable.
Failure in this layer is usually quiet. A job that crashes is close to a good outcome, because somebody is told. The expensive case is a job that completes normally having written the wrong rows: a column renamed upstream, a timezone assumption that changed, a file delivered twice, or a late record landing in the wrong partition. None of those raise an error, and whether they are caught at all depends on checks that were designed in advance rather than added after an executive noticed a figure looked odd.
Much of the difficulty sits at organisational boundaries. The teams producing data rarely know who consumes it, so a routine refactor in an application breaks a report three departments away. The durable answers are agreements rather than tooling: explicit contracts about structure and meaning, validation on the producing side, additive change as the default, and a deprecation path with notice, all of which require someone willing to have the conversation before the change ships.
Cost has become a design surface rather than an afterthought. Separating storage from compute made scale accessible to teams who could not previously afford it, and simultaneously made it possible to spend a great deal without noticing. Partitioning, clustering, file sizes, compaction, incremental against full rebuilds and the query patterns analysts actually run all move the bill by multiples, and none of them are visible from the output.
When teams need this capability
Data work is usually absorbed by application engineers and analysts until the seams start showing. These are the situations where that arrangement stops being economical.
Reports disagree with each other
Two dashboards give different answers for the same measure and the same month, because the definition was implemented separately in each tool by different people at different times. Every meeting then spends its first ten minutes arguing about whose number is right.
Pipelines break whenever an upstream service changes
A field is renamed or a type changes, nothing warned anybody, and the failure is discovered when a chart renders empty. Without agreements at the boundary this recurs indefinitely, and the fix is always retrospective.
Analysts spend most of their week preparing data
Skilled people are cleaning, joining and deduplicating by hand before they can begin the analysis they were hired to do — and each of them does it slightly differently, which is where the disagreeing reports come from.
The platform bill is growing faster than the business
Compute spend rises without a corresponding rise in what the platform delivers. Diagnosing it means looking at query patterns, table layout, materialisation choices and full rebuilds that could be incremental, rather than negotiating with the vendor.
Freshness requirements have moved from daily to minutes
An operational use case now needs data that a nightly batch cannot supply. Streaming introduces delivery semantics, out-of-order arrival, windowing and watermarks, and a set of operational burdens teams routinely underestimate before committing to them.
Nobody can say where a figure came from
An auditor, a regulator or a customer asks how a number was derived and the honest answer is a sequence of transformations nobody has documented. Lineage becomes urgent precisely when it is most difficult to reconstruct.
Core capabilities
Ingestion
Batch extraction, change data capture from operational databases, event consumption, and third-party connectors — together with the unglamorous parts: replays, backfills, rate limits and vendors whose exports are occasionally incomplete.
Transformation and modelling
Shaping raw arrivals into tables people can reason about: dimensional models where they help, deliberate denormalisation where they do not, incremental logic that is safe to rerun, and tests that live with the model rather than beside it.
Orchestration
Dependency graphs, retry policy, backfill semantics, partition awareness and freshness commitments — so that a delayed source degrades one dataset rather than cascading through everything scheduled after it.
Contracts and schema evolution
Agreeing what a producer guarantees, validating it where the data is produced, versioning rather than mutating, and giving consumers notice and a migration path before anything is withdrawn.
Quality checks
Freshness, row volume, uniqueness, referential integrity and distribution checks, positioned where they catch problems before consumers do, with alerting calibrated so that people still respond to it after a month.
Lineage and observability
Column-level provenance, impact analysis before a change is made rather than after it breaks something, and the ability to answer where a specific figure came from without reading every transformation by hand.
Storage architecture
Choosing between warehouse, lakehouse and operational stores, selecting an open table format, and getting partitioning, file sizing and compaction right — decisions that quietly determine both query speed and monthly spend.
Streaming
Windowing, watermarks, out-of-order and late arrivals, delivery guarantees, state management, and the judgement to recognise when a stream is not worth its operational weight for the decision it supports.
Cost engineering
Knowing which workloads dominate the bill, sizing compute to the job, choosing materialisation over recomputation where it pays, tiering storage, and measuring spend per dataset rather than as a single monthly figure.
Governance and access
Classifying personal data, masking and tokenising it, enforcing access at row and column level, applying retention, and ensuring that sensitive fields do not reappear in a derived table that nobody classified.
Technology ecosystem
Common technologies in data engineering are listed below. This describes the landscape of the discipline as it is practised, not a claim about any particular engineer’s toolkit. The products in this space are replaced far more often than the failure modes change, so reasoning about correctness, idempotency and cost transfers between stacks more reliably than any individual name on the list.
Languages
- Python
- SQL
- Scala
- Java
Ingestion and streaming
- Apache Kafka
- Debezium
- Apache Flink
- Airbyte
- Fivetran
- Amazon Kinesis
Processing and transformation
- Apache Spark
- dbt
- Apache Beam
- Polars
- DuckDB
Orchestration
- Apache Airflow
- Dagster
- Prefect
- Temporal
Warehouses and query engines
- Snowflake
- BigQuery
- Databricks
- Amazon Redshift
- ClickHouse
- PostgreSQL
Table formats and storage
- Apache Iceberg
- Delta Lake
- Apache Hudi
- Apache Parquet
- Amazon S3
Quality and cataloguing
- Great Expectations
- Soda
- dbt tests
- OpenMetadata
- DataHub
How this role works with your team
Engineers work inside your team, on your priorities, to your standards. You direct the work; Talent.ID carries the employment. The division below is the whole arrangement.
You keep
- Product
- Business priorities
- Roadmap
- Architecture
- Sprint priorities
- Engineering standards
- Day-to-day technical collaboration
Talent.ID handles
- Employment relationship
- Payroll
- Employee benefits
- Talent administration
- Ongoing employee relationship
What to look for when hiring
Candidates in this field are usually assessed on tool familiarity, which is the wrong axis. The products turn over every few years while the failure modes stay exactly the same, and the informative areas are the ones that only appear once a platform has been running long enough to have been wrong at least once.
Treatment of silent failure
Ask how they would learn that a successful job had written incorrect data. Engineers who have been through it design detection in advance and can name the specific checks; engineers who have not describe monitoring job status, which is monitoring the wrong thing.
- Distinguishes a failed job from a failed dataset without prompting
- Reconciles totals against the source system rather than trusting the load
- Can describe wrong figures reaching a report and how they were eventually caught
- Places checks where consumers are protected, not only at the end
Schema change and boundaries
Ask what happens when a producing team changes a field. The answer reveals whether they see this as an engineering problem to be handled downstream or as an agreement to be negotiated upstream, and only the second scales past a handful of sources.
- Has negotiated a contract with a team that produces data
- Treats additive change as the default and breaking change as an event
- Knows which consumers depend on which columns before altering them
- Gives notice and a migration window rather than announcing a fait accompli
Idempotency and recovery
Ask what happens when a job runs twice, or when yesterday has to be rebuilt after a source correction. This is the most reliable technical discriminator in the discipline and it is rarely asked directly.
- Designs so that rerunning a period produces the same result
- Can backfill a range without duplicating or losing rows
- Handles records that arrive after their partition has been processed
- Thinks in partitions and watermarks rather than in whole tables
Modelling judgement
Ask how they decide the shape of a table. Strong candidates model for the questions people actually ask and can justify a denormalisation; weaker ones apply a methodology uniformly and produce something elegant that nobody queries.
- Defines a business measure in exactly one place
- Can explain a deliberate departure from normal form
- Separates raw arrivals from curated output with a clear boundary
- Recognises when a full modelling layer is more than the problem needs
Honesty about freshness
Ask when they have argued against streaming. Real-time is frequently requested and less frequently required, and an engineer who asks which decision depends on the freshness will save a team a great deal of operational burden.
- Starts from the decision the data supports, not the technology
- Has recommended batch where continuous processing was expected
- Can explain delivery guarantees precisely rather than by slogan
- Accounts for what a streaming system costs to run at three in the morning
Cost awareness
Ask what the largest line in their last platform bill was and what they did about it. Engineers who have carried a budget answer specifically; engineers who have not treat spend as somebody else’s concern, which is how a platform quietly becomes the second-largest item in an engineering budget.
- Knows which workloads dominated spend and why
- Changed partitioning, file layout or materialisation on evidence
- Attributes cost to individual datasets rather than a single total
- Reasons about storage and compute independently
Working with consumers
The engineers who make the most difference talk to the analysts and product teams who use their output. Ask what they document about a dataset and who they wrote it for — the answer separates a service mindset from a delivery one.
- Documents what a column means, not only its type
- Knows the questions their most important datasets exist to answer
- Announces changes to consumers before making them
- Has retired a dataset deliberately rather than leaving it to rot
Interview questions worth asking
These are for your own interview process to use as you see fit. You assess each candidate and decide; the questions here are chosen because they are difficult to answer well without having operated a platform that somebody depended on.
A nightly job completes successfully and writes incorrect data. How would you have found out?
What a strong answer shows
The central problem of the discipline. Strong answers describe checks designed beforehand — reconciliation against source, distribution and volume monitoring, uniqueness constraints — rather than a plan to investigate once somebody complains.
An upstream team renames a field without telling anyone. What should have happened, and what do you do now?
What a strong answer shows
Whether they think in agreements or in fixes. The immediate remedy is straightforward; the interesting part is what they propose so that the next change is announced rather than discovered.
Explain how you make a pipeline safe to run twice.
What a strong answer shows
Practical idempotency. Look for deterministic partition keys, replace rather than append semantics, deduplication on a stable identifier, and awareness of which side effects cannot be undone by a rerun.
Two teams report different revenue for the same month. Take me through resolving it.
What a strong answer shows
Method under ambiguity. Expect them to compare definitions before comparing numbers, trace both figures to source, identify where the logic diverged, and then propose a single owned definition rather than reconciling once.
When have you argued against streaming, and what did you propose instead?
What a strong answer shows
Judgement about complexity. Strong candidates start from the latency the decision genuinely requires and are comfortable saying that an hourly batch met it at a fraction of the operational cost.
The warehouse bill has doubled in three months. Walk me through the investigation.
What a strong answer shows
Cost fluency. Look for query-level attribution, a new full rebuild replacing an incremental one, small-file proliferation, an unpartitioned scan, or a scheduled job somebody added and nobody reviewed.
How do you handle records that arrive late or out of order?
What a strong answer shows
Whether time is modelled explicitly. Expect a distinction between event time and processing time, a stated lateness allowance, and a decision about what happens to a record that arrives beyond it.
What do you write down about a dataset, and who reads it?
What a strong answer shows
Service orientation. The strongest answers cover meaning, grain, freshness commitment, known caveats and ownership — written for the analyst who will use it at short notice, not for the next engineer.
What this looks like in practice
A hypothetical scenario, written to show how the working model applies. It does not describe a Talent.ID client or a completed project.
- Challenge
- An organisation’s reporting has become unreliable. Definitions of the same measure differ between tools, pipelines fail whenever a producing service is refactored, analysts spend more time reconciling extracts than analysing them, and platform spend has risen without anyone being able to attribute it to a workload.
- Approach
- Additional engineering capacity works within the practices the team already has — their repository layout, their orchestration conventions, their review process and the platform direction they have chosen. Decisions about roadmap, architecture and the order of the backlog stay with the internal team; the added capacity takes on contract, quality and modelling work in the same sprint cadence.
- What this adds to the team
- The team expands what it can deliver without ceding control of its platform or its priorities. Which datasets matter, and what standard they must meet, remains a decision for the people who own the outcome.
Related disciplines
Frequently asked questions
- What is the difference between a data engineer and a data scientist?
- A data engineer is accountable for data arriving correctly, on time and with its meaning documented. A data scientist is accountable for what that data means: framing the question, designing the analysis, separating a genuine effect from a coincidence, and reporting the result with its uncertainty intact. The first builds the road; the second decides where to drive and whether the journey supported the conclusion. Teams that hire only the second often find they have bought expensive data cleaning.
- How does a data engineer differ from an analytics engineer?
- The analytics engineer works largely inside the warehouse, turning loaded data into well-modelled, tested and documented tables, and owning metric definitions with the analysts who use them. The data engineer owns the wider surface: ingestion, streaming, orchestration infrastructure, contracts with producing teams, storage architecture and cost. The two meet in the transformation layer, and in smaller organisations one person does both, which is fine until the ingestion side needs real attention.
- Do we need a data engineer or better tooling?
- Tooling genuinely helps with connectors, orchestration and quality checks, and there is no reason to build any of that by hand. It does not settle what a metric means, negotiate a contract with the team that keeps changing a schema, decide what to do about a late-arriving correction, or notice that a table nobody owns has been feeding a board report for a year. Those are the recurring problems, and they are organisational as much as technical.
- How much software engineering does this role require?
- More than is often assumed. Pipelines are production systems: they need version control, tests, code review, continuous integration, environment separation, dependency management and the ability to deploy a change safely. A team staffed with people who write capable SQL but have never worked in a disciplined engineering process will accumulate an unmaintainable estate faster than they expect.
- Do we need this capability before doing anything with machine learning?
- Almost always. Models inherit every defect in the data that trains them, and the most common cause of a modelling programme stalling is not the modelling but the absence of dependable, documented, well-understood inputs. A team can prototype without a platform, but it cannot operate reliably on top of extracts that a person assembles by hand each week.
- At what point does a team need a second data engineer?
- Usually once the platform has consumers who would notice an outage, which makes a single engineer both a bottleneck and an on-call risk carrying knowledge nobody else holds. It also becomes necessary when ingestion and transformation are under active development simultaneously, since those are different rhythms of work and one person alternating between them does neither well.
- How do data engineers work with an existing engineering team?
- Your team directs the work under a staff augmentation arrangement: your conventions, your review process, your roadmap and your ordering of the backlog decide what gets built and when, and the technical conversation happens with your own team. Talent.ID carries employment, payroll and employee benefits, together with talent administration and the ongoing relationship.
Tell us what your team needs
Describe the gap — the work, the stack, the way your team runs — and we will tell you what we can support. If it is not something we can help with, we will say so.