How Many Samples? Sample Size Justification for Design Verification

“How many samples do we need?” is the most common statistical question in product development, and the most commonly answered with a number that has no basis: thirty.

Thirty is not a justification. It’s a habit. Reviewers increasingly say so.

A defensible sample size is the output of three inputs: the risk you are managing, the claim that risk demands, and the capability of your process. Get those three on the table and the number falls out. This article walks that path for the situations that cover most verification testing.

Where “n = 30” actually comes from

The number has a real origin — it just isn’t the one people invoke. Thirty is roughly where the t distribution becomes close enough to the normal distribution that the difference stops mattering for many estimates, and where the Central Limit Theorem is often adequate for the sampling distribution of a mean.

That is a statement about estimating a mean. It says nothing about demonstrating that 99% of your product meets a specification, which is what design verification usually needs to show. Using n = 30 for a reliability claim applies a rule of thumb from one problem to a different problem.

Risk sets the claim, and the claim sets n

Before any arithmetic, one decision governs everything downstream: how much evidence does this characteristic deserve?

That is a risk question, not a statistical one. The chain runs:

risk of the failure mode → the confidence/reliability claim → the sample size

A dimension whose failure could injure a patient warrants stronger evidence than one that affects only appearance. Most organizations encode this in a sampling policy tied to the severity — and sometimes occurrence — ratings in their ISO 14971 risk analysis. A common shape:

These particular numbers are a convention, not a regulation. Nothing in 21 CFR 820 or ISO 13485 names them. What makes an individual sample size defensible is that the threshold came from a written policy tied to your own risk classification — not from the study that needed it. A reviewer can accept 95/90 for a low-risk feature; what they cannot accept is 95/90 chosen because it produced a convenient number.

Reliability is the expensive dial

The risk decision is not a formality — it is the single biggest driver of how much testing you will do. Holding confidence at 95% and raising only the reliability claim:

Going from 90% to 99% reliability multiplies the attribute sample size roughly tenfold (29 → 299). Raising confidence from 95% to 99% costs far less (299 → 459). Reliability is about how much of the population you are covering; confidence is only about how sure you are of that coverage — and coverage is what costs parts.

This cuts both ways. Inflating the claim “to be safe” is expensive, and an unjustifiably high reliability target is as much a finding in an audit as an unjustifiably low one.

Case 1: Variables data — the tolerance interval

When you measure something on a continuous scale (force, pressure, dimension, dose), you can use the measurements themselves rather than just pass/fail. This is by far the most efficient option, and the one most often left on the table.

The claim is framed as confidence/reliability — “95% confidence that at least 99% of units meet the specification.” You demonstrate it with a one-sided tolerance interval: compute $\bar{x} - k s$ and show it falls above the lower specification limit (or $\bar{x} + k s$ below an upper limit).

The factor $k$ depends on the sample size, the confidence level, and the reliability claimed. It shrinks as $n$ grows:

Closing the loop: what n do you actually need?

The demonstration passes when the tolerance limit clears the specification:

$$\bar{x} - k s \ge \text{LSL}$$

Dividing through by $s$ puts it in a far more useful form:

$$\frac{\bar{x} - \text{LSL}}{s} \ge k(n, C, R)$$

Now notice what the left side is. Since $\hat{C}_{pl} = \dfrac{\bar{x} - \text{LSL}}{3s}$, the condition is simply:

$$3\,\hat{C}_{pk} \ge k(n, C, R)$$

That is the answer to “how many samples?” for variables data. The required $n$ is whichever sample size makes $k$ small enough for your process capability to clear it. There is no separate formula to memorize — sample size and capability are two views of the same margin.

So the procedure is:

1. Estimate the capability you expect — from a pilot run, historical data, or a similar product and process. (That estimate carries its own uncertainty; see how much data a capability index needs.)
2. Set the claim from risk, using your sampling policy.
3. Choose the smallest n where $k(n, C, R) \le 3 \times$ expected Cpk.

Which gives this table — the practical answer to the question the article opened with:

Two things jump out. A capable process needs remarkably few samples — at Cpk 1.33 you can support a 95/99 claim with 10 measurements, where the attribute equivalent needs 299.

And capability trades against sample size very steeply. Continuing the 95/99 column downward:

The runaway is not an artifact. As $n$ grows, $k$ converges to $z_R$ — the reliability z-score itself (2.326 for 99%). So the demonstration is only achievable when $3\,C_{pk} > z_R$, giving a hard floor of Cpk ≈ 0.78 for a 99% reliability claim. Below that, no sample size will ever pass, because you cannot demonstrate that 99% of output conforms from a process incapable of producing 99% conforming output.

The practical lesson: when a sample size comes back implausibly large, that is rarely a sampling problem. It is the arithmetic telling you the process isn’t capable enough for the claim. Reducing variation is frequently cheaper than testing more parts — and near the floor, it is the only option.

The minimum is not a plan

Here is the part that gets skipped, and it matters more than the table above.

Those numbers are the $n$ at which passing becomes possible — not likely. Your sample mean and standard deviation vary from study to study, so planning exactly at the boundary is planning for a coin flip.

Simulating a process whose true capability is Cpk = 1.33, demonstrating 95/99:

At the theoretical minimum, about half of such studies fail — not because the process is bad, but because the sample happened to look slightly worse than the process really is. A failed verification study costs far more than a few extra parts: rework, re-testing, schedule, and an awkward explanation.

So size for assurance, not for the minimum. Target a comfortable probability of passing — 80–90% is reasonable — or equivalently, size against a deliberately conservative capability estimate rather than your best one. If a pilot suggests Cpk 1.5, plan as though it were 1.2.

This is also why a pilot to estimate $s$ is usually a better investment than a large study built on a guess. Sample size for variables data is only as good as the variability estimate behind it.

Assumption to check: tolerance intervals assume normality, and they are sensitive to it in the tails — exactly where a reliability claim lives. This is another argument against single-digit sample sizes: with n = 5 you cannot meaningfully assess the distribution you are relying on. Verify normality, and use a distribution-free or transformed approach when it doesn’t hold.

Case 2: Attribute data — pass/fail

When the result is only conforming/nonconforming, you lose the information in the measurement and pay for it in sample size.

For zero observed failures, the relationship is compact:

$$n = \frac{\ln(1 - C)}{\ln(R)}$$

where $C$ is confidence and $R$ is reliability. So:

Two things stand out. First, the familiar “n = 29” is just 95/90 with zero failures — a legitimate result, but a modest claim; it is not evidence for 99% reliability. Second, the cost of a high reliability claim on attribute data is steep.

If a single unit fails, the required n rises sharply — roughly 473 for 95/99. Plan for that possibility rather than being forced to re-justify mid-study.

This is the argument for measuring instead of counting. Compare the two tables: at 95/99, a capable process needs 10 measurements or 299 pass/fail results. Recording the actual force instead of “pass/fail” is often the single largest sample-size reduction available to you — and it yields a capability estimate as a bonus.

Case 3: Comparing to a target or another group — power

When the question is “is the new design different?” or “does it meet a target mean?”, you’re in hypothesis-testing territory and the driver is power — the probability of detecting a real difference of a size you care about.

Sizing requires three inputs:

• Effect size ($\Delta$) you need to detect — the smallest practically meaningful difference
• Variability ($\sigma$, from prior data, a pilot run, or a conservative estimate)
• $\alpha$ and power, conventionally 0.05 and 80–90%

For a two-sample comparison of means, roughly:

$$n_{\text{per group}} \approx \frac{2\left(z_{\alpha/2} + z_{\beta}\right)^{2} \sigma^{2}}{\Delta^{2}}$$

Note the $\Delta^{2}$ in the denominator: halving the difference you want to detect quadruples the sample size. This is why “we’ll just detect any difference” is not a specification — it implies infinite $n$.

Risk enters here too, through the effect size. The smallest difference worth detecting should be the smallest difference that could matter to the patient or the process — a risk judgment, not a statistical one. And power is itself a risk decision: 80% power means accepting a one-in-five chance of missing a real effect, which may be unacceptable for a high-severity characteristic.

Writing a justification that survives review

A reviewer isn’t looking for a big number. They’re looking for a traceable argument. A sound justification states:

1. What is being demonstrated — the specific characteristic and its specification.
2. The risk basis — the failure mode, its severity and occurrence from your risk analysis, and the confidence/reliability (or power) your sampling policy assigns to that risk level. This is the load-bearing step.
3. The method and assumptions — the interval or test used, the distributional assumption, and how it was checked.
4. The calculation — inputs, the source of any capability or variability estimate, and the resulting n, including any margin added for assurance.
5. The acceptance criterion, stated before the data is collected.

The single most important sentence is the one connecting the statistical claim back to risk. A sample size traceable to a risk-based policy is defensible even if a reviewer would have chosen differently; a sample size with no such link is indefensible even when the number happens to be large.

Three traps worth naming

• Sizing after the fact. Choosing the analysis once you’ve seen the data invalidates the error rates. Write the plan first.
• Ignoring measurement error. Your observed variation includes gauge variation. If the measurement system is weak, you’re sizing against inflated $\sigma$ — testing more units to compensate for a fixable problem. Run an MSA first.
• Treating units as independent when they aren’t. Thirty parts from one lot, one cavity, or one operator is n = 1 for between-lot variation. Sample across the sources of variation you intend to claim coverage over.

---

Need a sample size justification a reviewer will accept? See our sample size and power services or book a call.