Data scientists
Data scientists turn an unclear commercial question into one that evidence can settle, then settle it honestly — including when the honest finding is that the evidence does not support the conclusion somebody was hoping for. The value lies in the framing and the rigour rather than in any model that may emerge along the way. This guide describes the discipline and how to evaluate it.
What does a data scientist do?
A data scientist answers questions with data. The work starts by converting a vague business question into a specific and measurable one, then choosing an approach capable of supporting the conclusion being sought: descriptive analysis, a designed experiment, an observational study with its assumptions stated, or a predictive model used to produce an estimate. It includes quantifying how uncertain the result is, identifying the confounders and selection effects that would make it misleading, and presenting the finding to decision-makers in a form that supports a decision without overstating what was actually shown.
Framing produces most of the value and receives the least attention. “Why are customers leaving” cannot be answered as posed: it has no population, no period, no comparison and no definition of leaving. Turning it into something answerable — which cohort, measured over what window, compared against whom, and what result would change the plan — determines whether the subsequent work is worth doing at all. An analyst who begins querying immediately will produce something; whether it bears on the decision is a separate matter.
The recurring technical question is whether an observed relationship supports the action being contemplated. Some questions can be settled by randomising and waiting; many cannot, because the change affects everyone at once, or randomising would be unethical, or the event already happened. For those, the practitioner has to choose a design — differences over time between comparable groups, an instrument, a discontinuity in eligibility, matching on observed characteristics — and be explicit about the assumptions on which the conclusion rests, because those assumptions are the finding’s weakest point and someone should be told where it is.
Communication is a technical skill in this role rather than a soft one. A confidence interval flattened into a single figure on a slide has lost precisely the information the decision was sensitive to. Knowing which uncertainty matters for the choice at hand, and conveying it so that a non-specialist can weigh it without being either misled or paralysed, is part of the analysis rather than a presentational afterthought.
When teams need this capability
Reporting tells an organisation what happened. These are the pressures that arise when it needs to know why, or what would happen if it did something differently.
Decisions rest on dashboards nobody interrogates
A chart moves and a plan changes, with no one asking whether the movement is larger than the noise, whether the population changed, or whether the definition behind the measure still means what it did when it was written.
Experiments run but nobody trusts the results
Tests are stopped when they look good, several measures are checked until one reaches significance, samples are too small to detect the effect anyone cares about, and assignment leaks between groups. The programme produces conclusions at a healthy rate and the organisation has quietly stopped believing them.
A measure has become a target
A number entered a compensation plan or a board pack and behaviour reorganised itself around raising it. Someone has to state clearly what it does and does not capture, and what is being sacrificed to move it.
An observational claim is about to drive spend
A channel, a feature or a campaign is described as driving an outcome on the basis of a correlation in a dashboard. Before a budget follows, someone needs to establish whether the effect is real or whether the people exposed were always going to convert.
The question is why, not what
Instrumentation reports events accurately and explains nothing. Understanding cause requires either an experiment designed in advance or an observational design chosen deliberately, and neither happens by inspecting charts more carefully.
Forecasts are needed and false precision is dangerous
Demand, capacity or headcount planning requires a projection. A single number carries an implied confidence nobody checked, and planning against it without a range means the contingency is set by optimism rather than by evidence.
Core capabilities
Problem framing
Converting an ambiguous request into a question that has a population, a period, a comparison and a decision attached to it — and establishing beforehand which possible results would change what anybody does.
Experiment design
Choosing the unit of randomisation, calculating the sample and duration needed to detect an effect worth acting on, selecting guardrail measures, and fixing the decision rule before the first observation arrives.
Causal inference from observational data
Applying designs suited to situations where randomisation is impossible — comparisons of trends between groups, instruments, eligibility thresholds, matching — and stating plainly the assumptions each one requires.
Statistical rigour
Sampling and its biases, multiple comparisons, survivorship and selection effects, the difference between a significant result and a meaningful one, and checking assumptions rather than inheriting them from a textbook example.
Critique of the data itself
Establishing how a dataset came to exist before drawing on it: what the instrumentation actually records, when a definition changed, what missingness indicates, and which records were silently excluded.
Modelling for understanding
Using models to estimate a quantity or to characterise a relationship, with attention to specification, collinearity, out-of-sample behaviour and interpretability, in settings where the output informs a person rather than serving live requests.
Measure definition
Constructing measures that reflect the outcome an organisation actually wants, resist gaming, distinguish leading from lagging indicators, and remain comparable when the product or the population changes.
Quantifying uncertainty
Attaching intervals and stated assumptions to every estimate, being explicit about what is unknown, and distinguishing sampling variation from the far larger uncertainty introduced by the choice of design.
Presentation to decision-makers
Choosing visual and narrative forms that make the finding and its limits legible in the time a decision-maker will give it, without either burying the caveats or hiding behind them.
Reproducible analysis
Version-controlled code, recorded query and data inputs, and a stated environment, so that a conclusion can be rerun, challenged and revisited after the situation has moved on.
Technology ecosystem
Common technologies in data science are listed below. This describes the landscape of the discipline as it is practised, not a claim about any particular engineer’s toolkit. Tooling is the least discriminating thing about a candidate here, since the reasoning that makes an analysis trustworthy is identical whichever language expresses it.
Languages
- Python
- R
- SQL
Analysis libraries
- pandas
- Polars
- NumPy
- SciPy
- statsmodels
- scikit-learn
Causal inference and Bayesian methods
- DoWhy
- EconML
- CausalImpact
- PyMC
- Stan
Notebooks and reporting
- Jupyter
- Quarto
- R Markdown
- Streamlit
Visualisation
- matplotlib
- seaborn
- plotly
- ggplot2
- Vega-Lite
Data access
- dbt
- BigQuery
- Snowflake
- DuckDB
- Databricks
Experimentation platforms
- GrowthBook
- Statsig
- Eppo
- Optimizely
How this role works with your team
Engineers work inside your team, on your priorities, to your standards. You direct the work; Talent.ID carries the employment. The division below is the whole arrangement.
You keep
- Product
- Business priorities
- Roadmap
- Architecture
- Sprint priorities
- Engineering standards
- Day-to-day technical collaboration
Talent.ID handles
- Employment relationship
- Payroll
- Employee benefits
- Talent administration
- Ongoing employee relationship
What to look for when hiring
Screening for this role tends to test what is easy to test — library fluency and textbook statistics — and to miss the two properties that decide whether the hire repays itself: whether they frame the right question, and whether they will say so when the evidence does not support the answer somebody wanted.
Framing before analysis
Give a candidate a genuinely vague question and see what they do with it. The response that matters is a set of clarifying questions about population, period, comparison and consequence. A candidate who moves straight to method is telling you how the first six months will go.
- Restates the question in a form that could be answered before answering it
- Asks which decision depends on the result
- Establishes what finding would change the plan, and what would not
- Has declined to run an analysis and can explain the reasoning
Causal reasoning
Present a correlation and a proposed action based on it. Strong candidates immediately separate what was randomised from what was merely observed, name the plausible confounders, and propose a design rather than a caveat.
- Names specific confounders rather than warning about confounding generally
- Can describe a quasi-experimental design and the assumption it rests on
- Treats a before-and-after comparison as weak evidence unless justified
- Distinguishes selection into a behaviour from the effect of that behaviour
Experiment discipline
Ask what they do before a test launches. The valuable habits are all preparatory: estimating the smallest effect worth detecting, sizing accordingly, and writing down the decision rule while nobody yet has a stake in the outcome.
- Calculates required sample and duration in advance
- Fixes the primary measure and the stopping rule before launch
- Can explain concretely what repeated checking does to a false positive rate
- Uses guardrail measures to catch harm outside the primary outcome
Scepticism about the data
Ask how they satisfy themselves that a dataset means what its column names suggest. Practitioners who have been embarrassed once do this reflexively; those who have not tend to trust the warehouse implicitly.
- Investigates how the data came to be recorded before analysing it
- Has found an instrumentation defect that invalidated a piece of work
- Treats missing values as evidence about a process, not as rows to drop
- Cross-checks an important figure against an independent source
Communicating uncertainty
Ask them to explain a result to a non-specialist. Look for someone who presents a range and what it implies for the choice, rather than either a bare point estimate or a wall of qualification that leaves the listener no better placed.
- States assumptions plainly, in the language of the business
- Presents the range and what it means for the decision at hand
- Says clearly when the evidence does not settle the matter
- Adapts depth to the audience without changing the substance
Independence under pressure
Ask about a finding that was unwelcome. The instructive part is what happened next: whether the analysis was reframed until it agreed with the preferred conclusion, and how the candidate handled being asked to look again.
- Has delivered a result that contradicted a senior expectation
- Separates the finding from the recommendation that follows it
- Will re-examine method willingly and change conclusions unwillingly
- Can describe how they preserved the working relationship regardless
Proportionality
Rigour should match the stake. A reversible decision needs a defensible estimate this week, not an unimpeachable study next quarter, and knowing which situation they are in is a large part of being useful inside a commercial team.
- Matches the depth of the work to the cost of being wrong
- Will give a rough answer with its limits stated when that is enough
- Recognises when a decision warrants a properly designed study
- Finishes analyses rather than refining them indefinitely
Interview questions worth asking
Suggested material for your own hiring conversations. You make the assessment and you make the decision; these questions are included because they reward reasoning and are hard to answer convincingly from reading alone.
A stakeholder asks why churn went up last month. What do you do first?
What a strong answer shows
Framing instinct. Strong candidates ask which customers, over what period, against what baseline, and what action is being considered — before opening anything. Weak ones start describing a cohort analysis immediately.
Customers who use a particular feature retain much better than those who do not. Is the feature working?
What a strong answer shows
Whether selection is recognised without prompting. The engaged users would likely have stayed anyway; the useful answer proposes how to find out, whether by experiment or by a design that addresses the selection directly.
An experiment shows a significant improvement on day three. Do you ship it?
What a strong answer shows
Discipline against a tempting result. Look for the pre-registered duration, the cost of repeated inspection, novelty effects, weekday composition, and whether the effect is large enough to matter commercially.
Tell me about a result nobody wanted. What happened afterwards?
What a strong answer shows
Independence and organisational skill together. The best answers describe holding the finding, engaging seriously with challenges to the method, and keeping the relationship intact without diluting the conclusion.
The change affects every customer at once, so it cannot be randomised. How do you evaluate it?
What a strong answer shows
Range of method. Expect comparison against a similar untreated population, a synthetic control, an interrupted time series, or a staged geographic rollout — accompanied by the assumptions each approach requires.
Describe a time the underlying data was wrong. How did you discover it?
What a strong answer shows
Whether they audit their inputs. Look for a discrepancy against another source, an implausible distribution, a step change matching a release date, or a definition that had quietly changed mid-period.
How do you present a genuinely uncertain result to someone who wants one number?
What a strong answer shows
Communication judgement. The strongest answers give the number, give the range, and translate the range into what it means for the decision — rather than refusing the question or pretending to a precision they do not have.
How would you choose the measure a team should be held to for the next quarter?
What a strong answer shows
Understanding of incentives. Look for alignment with the outcome the business actually wants, resistance to gaming, a sensible balance of leading and lagging indicators, and awareness of what the measure will distort.
What this looks like in practice
A hypothetical scenario, written to show how the working model applies. It does not describe a Talent.ID client or a completed project.
- Challenge
- A commercial team is deciding where to concentrate its spend. The available evidence is a set of correlations drawn from dashboards, a testing programme whose results are inconsistent and no longer trusted internally, and a forecast presented as a single figure with no range attached to it.
- Approach
- Additional analytical capacity works inside the team’s existing practices — their review conventions, their tooling, their reporting cadence and the analytical standards they have already established. The internal team retains ownership of the business priorities and of what gets investigated; the added capacity contributes to study design, analysis and reporting alongside them rather than as a separate function.
- What this adds to the team
- The team can pursue more questions properly without transferring judgement about which questions matter. What is investigated, and what evidence is considered sufficient to act on, remains with the people accountable for the decisions.
Related disciplines
Frequently asked questions
- What is the difference between a data scientist and a data analyst?
- An analyst is principally concerned with describing what happened: reporting, dashboards, definitions of measures, and answering questions whose form is already settled. A data scientist takes on the questions where the form is not settled — designing a study, distinguishing an effect from a coincidence, quantifying how confident anyone should be. The boundary moves between organisations, and job titles track it poorly, so it is worth describing the actual work in a brief rather than relying on the label.
- How does a data scientist differ from a machine learning engineer?
- The data scientist establishes what should be measured and whether a claimed effect is real; the machine learning engineer builds and maintains the system that trains and serves a model. A great deal of valuable data science produces no model at all, and a great deal of machine learning engineering concerns pipelines, serving and drift rather than the choice of question. Teams that expect one person to cover both usually find that one half is neglected, and which half depends on where the person’s interest lies.
- Do data scientists build production models?
- They frequently build the first version, and that version is generally intended to establish whether the approach has merit rather than to serve traffic. Once something has to run continuously, meet a latency budget, be monitored for decay and be rebuilt on a schedule, it becomes an engineering responsibility. Treating a prototype as a production system is a common and expensive mistake, and it is usually made by the organisation rather than by the person who wrote it.
- Do we need a data scientist or just better reporting?
- If the outstanding questions are about what happened and to whom, the answer is better instrumentation, cleaner definitions and analyst time — and hiring for research skills to service dashboard requests is a reliable way to lose the hire within a year. If the questions are why something happened, whether an intervention worked, or what would happen under a different plan, then this is the capability required, and reporting will never settle them however good it becomes.
- How much does domain knowledge matter?
- Considerably, because framing depends on it: knowing which comparison is fair, which seasonal pattern is ordinary, and which explanation is implausible on operational grounds. It is also the most acquirable of the requirements. A rigorous practitioner who spends their first weeks talking to the people who run the business will usually outperform someone with deep sector familiarity and loose method.
- Why do data science hires often disappoint?
- The cause is rarely technical. The recurring pattern is distance from the decisions: work is commissioned as requests rather than as questions, findings arrive after the choice has been made, the underlying data proves unreliable, and there is no established route from a conclusion to an action. A capable practitioner placed in that arrangement produces good analysis that changes nothing, and usually leaves.
- How do data scientists work with an existing engineering team?
- With staff augmentation, direction of the work stays with you: your analytical standards, your review practice, your priorities and your roadmap determine what is investigated and in what order, and the collaboration is with your own team. The employment side sits with Talent.ID — payroll and employee benefits, the administration that surrounds them, and the ongoing employment relationship.
Tell us what your team needs
Describe the gap — the work, the stack, the way your team runs — and we will tell you what we can support. If it is not something we can help with, we will say so.