Statistics and experimentation

Statistics is the discipline of deciding how much a set of observations can bear. A chart built during data analysis shows what happened in the data you hold; inference asks what that implies about data you do not hold: the customers who never responded, next month, the population a sample was drawn from. SFIA 9 builds its data science skill on mathematics, statistics and probability modelling. Google's Advanced Data Analytics certificate devotes a module each to statistics and regression and lists hypothesis testing and A/B testing among its skills, while the entry-level Google Data Analytics certificate barely touches the subject. That gap is the usual line between an analyst and a data scientist.

The core vocabulary

Five ideas carry most of the weight.

• Distributions and sampling. Any figure computed from a sample would come out differently from another sample; the sampling distribution says by how much.
• Confidence intervals. A range expressing that uncertainty, more honest than a single number and easier for non-specialists to read than a test result.
• Hypothesis tests and p-values. A p-value measures how surprising the data would be if nothing were going on. It is not the probability that a finding is true, the most frequent misreading in the whole field.
• Regression. Modelling one quantity as a function of others, where statistics and machine learning meet: linear and logistic regression appear in the textbooks of both.
• Experiments. Assigning people to variants at random so that the only systematic difference between groups is the thing under test. An A/B test is the web's version.

Where experiments go wrong

The beginner's mistake with A/B tests is peeking: checking results daily and stopping the moment a difference looks significant. Every look is another chance for noise to cross the line, so tests stopped this way crown winners that vanish on rollout. Fix the sample size in advance from the smallest effect worth detecting, run to it, and choose the primary metric before launch. Two further traps are testing many metrics and reporting whichever moved, and comparing groups that picked their own variant, which measures who chooses rather than what works. A third is ignoring novelty: users click on anything new for a while, and the early lift fades once the change is familiar.

Tooling matters less than reasoning. R was written by statisticians and still has the deepest library of methods; Python's statistical packages cover what most analysts need, and Python for data describes that kit. From here the path leads to machine learning, where the same ideas return as validation and overfitting, and back to data analysis fundamentals, where they make every chart's caption more careful. Interviews for data science roles probe this reasoning directly, often by asking a candidate to design an experiment or explain a surprising result, so practise saying the ideas aloud as well as computing them. The data and AI hub places both topics in the wider field.