Hypothesis Testing Without the Jargon: Alpha, Power and p-Values

By Dr. Sakshi Bharti, COO Aomics GmbH

I usually teach this topic by starting with the decision a clinical team is trying to make. Alpha controls the chance of a false positive under the null; power describes detection probability under a specified alternative; a p-value is not a probability that treatment works.

Hypothesis Testing Without the Jargon: Alpha, Power and p-Values: practical analytical flow.

The idea in plain language

Alpha controls the chance of a false positive under the null; power describes detection probability under a specified alternative; a p-value is not a probability that treatment works.

The practical question is always the same: what quantity or decision are we trying to support, and what assumptions connect the observed data to that target? This is why a protocol, estimand, analysis model and output should never be treated as separate paperwork exercises. They are parts of one argument.

The statistical core

For a two-sided Z test, reject when |Z|>z_(1−α/2).

The equation matters, but it should never float free of its clinical meaning. In production work, every symbol needs a source: an endpoint definition, a population rule, a planning assumption, a dataset variable, or a frozen configuration value.

Why this matters in a real trial

A method becomes clinically useful when the estimate, uncertainty and decision threshold are interpreted together. Point estimates without uncertainty are incomplete, while thresholds without an effect scale encourage mechanical decision making.

A mature workflow separates design-time assumptions from analysis-time observations. Planning values retain their evidence source and approval; analysis values retain dataset, data cut, mapping, population and engine provenance.

A small worked example

A p-value of 0.03 does not imply a 97% probability of treatment benefit.

The numbers in a worked example are illustrative. In a real study, Adaptrails should require user-approved parameters or versioned study data and retain the input source beside the result.

Inputs and provenance to capture

For this topic, the analysis record should capture at least: study and endpoint identifiers; analysis population; source of each planning assumption or input dataset; parameter values explicitly accepted by the user; engine/version; random seed where simulation is used; and the review state. For data-based analyses, the source dataset, immutable data cut and variable mapping belong in the same evidence record.

How this maps to Adaptrails

Relevant modules: M05, M17, M19. The platform should guide the user from inputs to analysis to explanation without duplicating statistical logic across modules. A method should have one authoritative implementation, while SAP Studio, dashboards and AI explanations consume the stored result.

What a junior statistician should check

• Is the clinical question explicit before the calculation?
• Are all inputs sourced or user-approved?
• Does the analysis population match the estimand?
• Is uncertainty reported, not only a point estimate?
• Are thresholds and defaults visible rather than hidden?
• Can another statistician reproduce the result from the stored data cut and configuration?

Common mistakes

• Choosing a familiar statistical procedure before defining the target quantity.
• Treating a software default as if it were a study assumption.
• Reporting a technically correct number without explaining its clinical interpretation.
• Recalculating the same statistic in a document generator or UI instead of using the authoritative engine result.

Three questions to ask in a project meeting

1. What will change if the assumption is wrong?
Run sensitivity analysis rather than defending one planning value as truth.

2. Which data source will feed the analysis?
At design stage, record the evidence source for assumptions. At analysis stage, record dataset, data cut, mappings and population flags.

3. Who makes the decision?
The statistical system calculates and may recommend. A statistician, DMC, medical monitor, clinical team or other authorized body retains the decision appropriate to the context.

Where to go next

For broader clinical biometrics and data-science services, visit Aomics GmbH. To explore the adaptive-trial design and statistical workflow platform, visit Adaptrials.

The best workflow is not the one with the fanciest method; it is the one whose assumptions, calculations and decisions remain visible.

References and further reading

• ICH E9: Statistical Principles for Clinical Trials
• EMA: Guideline on multiplicity issues in clinical trials

Decision-support note: Adaptrails is designed to support qualified clinical and statistical professionals. Statistical results, simulations and AI-assisted explanations require appropriate human review before clinical, safety, operational or regulatory action.