--- title: "Non-inferiority and Equivalence Trials" author: "Gosuke Homma" date: "`r Sys.Date()`" output: rmarkdown::html_vignette: toc: true toc_depth: 2 vignette: > %\VignetteIndexEntry{Non-inferiority and Equivalence Trials} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 4.5, fig.align = "center" ) ``` ```{r load} library(bbssr) ``` ## Hypotheses and margin Let group 1 receive the experimental treatment and group 2 the control. A non-inferiority trial with a margin $\delta > 0$ on the scale of the risk difference tests $$ H_{0}: p_{1} - p_{2} \le -\delta \quad \text{against} \quad H_{1}: p_{1} - p_{2} > -\delta , $$ which corresponds to `alternative = 'greater'` and `margin = delta`. With `alternative = 'less'` the null hypothesis is $p_{1} - p_{2} \ge \delta$, which suits an endpoint for which a lower response probability is better. A negative margin tests superiority by more than its absolute value. A margin other than 0 requires one of the two tests below and a one-sided alternative. The last section covers a margin on the scale of the risk ratio, `margin.scale = 'RR'`. ## The two tests Both tests refer $$ Z = \frac{\hat{p}_{1} - \hat{p}_{2} + \delta}{\widehat{\mathrm{SE}}} $$ to the standard normal distribution and reject $H_{0}$ when $Z > z_{1 - \alpha}$. The test of Blackwelder (1982), `Test = 'Blackwelder'`, uses the standard error at the observed proportions, $$ \widehat{\mathrm{SE}} = \sqrt{\frac{\hat{p}_{1}(1 - \hat{p}_{1})}{N_{1}} + \frac{\hat{p}_{2}(1 - \hat{p}_{2})}{N_{2}}} . $$ The test of Farrington and Manning (1990), `Test = 'Farrington-Manning'`, uses the same expression at the maximum likelihood estimates $\tilde{p}_{1}$ and $\tilde{p}_{2}$ under the restriction $p_{1} - p_{2} = -\delta$. With $\delta = 0$ the Farrington-Manning test is the chi-squared test and the Blackwelder test is the Wald test. The two tests are asymptotic, but the package computes their rejection regions, power and sample sizes exactly, as for the other tests. ```{r rr} rr.bw <- BinaryRR(N1 = 60, N2 = 60, alpha = 0.025, Test = 'Blackwelder', margin = 0.1) rr.fm <- BinaryRR(N1 = 60, N2 = 60, alpha = 0.025, Test = 'Farrington-Manning', margin = 0.1) c(Blackwelder = sum(rr.bw), Farrington.Manning = sum(rr.fm)) plot(rr.fm) ``` The size of a test of non-inferiority is its largest rejection probability on the boundary $p_{1} = p_{2} - \delta$ of the null hypothesis. The function below evaluates it over a fine grid of $p_{2}$. ```{r boundary-size} boundary_size <- function(RR, margin) { N1 <- attr(RR, 'N1') N2 <- attr(RR, 'N2') m <- matrix(as.vector(RR), N1 + 1L, N2 + 1L) p2 <- seq(margin + 0.001, 0.999, by = 0.001) size <- vapply(p2, function(q) { sum(outer(dbinom(0:N1, N1, q - margin), dbinom(0:N2, N2, q)) * m) }, numeric(1)) data.frame(p2 = p2[which.max(size)], size = max(size)) } sizes <- cbind(Test = c('Blackwelder', 'Farrington-Manning'), rbind(boundary_size(rr.bw, 0.1), boundary_size(rr.fm, 0.1))) sizes ``` The nominal level is 0.025. With `r attr(rr.fm, 'N1')` patients per group the size is `r signif(sizes$size[1], 3)` for the Blackwelder test, attained at $p_{2} =$ `r sizes$p2[1]`, and `r signif(sizes$size[2], 3)` for the Farrington-Manning test. The size of an asymptotic test at a given sample size can lie on either side of the nominal level, so it is worth computing for the sample sizes of a design. ## Sample size With a margin, `BinarySampleSize()` replaces the difference in the normal approximation by its distance $p_{1} - p_{2} + \delta$ from the boundary of the null hypothesis. `method = 'standard'` evaluates the variance in the term of the significance level at the large sample values of the restricted estimates, which gives formula (4) of Farrington and Manning (1990). `method = 'alternative.variance'` uses the variance under the alternative in both terms, which gives the formula of Blackwelder (1982). `method = 'exact'` steps from the formula of Farrington and Manning, one patient at a time, to the sample size at which the exact power of the test first attains the target. Below, both groups have a response probability of 0.7 and the margin is 0.15. ```{r sample-size} ni.ss <- expand.grid(method = c('exact', 'standard', 'alternative.variance'), Test = c('Blackwelder', 'Farrington-Manning'), stringsAsFactors = FALSE) ni.ss$N <- NA ni.ss$Power <- NA for (i in seq_len(nrow(ni.ss))) { ss <- BinarySampleSize(p1 = 0.7, p2 = 0.7, r = 1, alpha = 0.025, tar.power = 0.8, Test = ni.ss$Test[i], method = ni.ss$method[i], margin = 0.15) ni.ss$N[i] <- ss$N ni.ss$Power[i] <- round(ss$Power, 4) } ni.ss ``` `Power` is the exact power of the test at the sample size of each row, so the rows of the two formulas show how close the normal approximation comes to the target for each test. ## Blinded sample size re-estimation Friede, Mitchell and Mueller-Velten (2007) re-estimate the sample size of a non-inferiority trial from the blinded overall response rate, which is the pooled response probability of the package. The assumed difference is usually 0, so both recovered probabilities equal the blinded pooled response probability, and the sample size is re-estimated by the formula of Farrington and Manning (`ss.method = 'standard'`) or of Blackwelder (`ss.method = 'alternative.variance'`). The design below is planned for a pooled response probability of 0.7 and re-estimates the sample size after half of the patients. ```{r bssr} plan <- BinarySampleSize(p1 = 0.7, p2 = 0.7, r = 1, alpha = 0.025, tar.power = 0.8, Test = 'Farrington-Manning', method = 'standard', margin = 0.15) ni <- BinaryPowerBSSR( p = seq(0.4, 0.8, by = 0.1), Delta.A = 0, Delta.T = 0, N1 = plan$N1, N2 = plan$N2, omega = 0.5, r = 1, alpha = 0.025, tar.power = 0.8, Test = 'Farrington-Manning', ss.method = 'standard', margin = 0.15 ) ni plot(ni) summary(ni) ``` The planned total sample size is `r plan$N`. The variance of a binary endpoint is largest at one half, so the fixed-sample design loses power when the true pooled response probability lies closer to one half than the planning value, and the re-estimation enlarges the trial in that case. At a pooled response probability of `r ni$p[1]` the power is `r round(ni$power.BSSR[1], 3)` with re-estimation and `r round(ni$power.TRAD[1], 3)` without. ## Type I error rate on the boundary With a margin, `BinaryTypeIErrorBSSR()` evaluates the type I error rate on the boundary of the null hypothesis. The boundary is parametrized by the pooled response probability `theta`, and the values at which a response probability falls outside the unit interval are dropped. ```{r tie} ni.tie <- BinaryTypeIErrorBSSR( Delta.A = 0, N1 = plan$N1, N2 = plan$N2, omega = 0.5, r = 1, alpha = 0.025, tar.power = 0.8, Test = 'Farrington-Manning', ss.method = 'standard', margin = 0.15, theta = seq(0.1, 0.9, by = 0.01) ) ni.tie plot(ni.tie) ``` The largest type I error rate is certified over the interval spanned by `theta`, here from `r attr(ni.tie, 'interval')[1]` to `r attr(ni.tie, 'interval')[2]`. With the default `theta = seq(0, 1, by = 0.005)` the interval is the whole part of the boundary on which both response probabilities lie in the unit interval, which with an allocation ratio of 1 is `theta` from $\delta / 2$ to $1 - \delta / 2$. The same rejection probabilities follow from `BinaryPowerBSSR()` with the true difference on the boundary, `Delta.T = -margin`. ```{r boundary-check} on.boundary <- BinaryPowerBSSR( p = c(0.5, 0.7), Delta.A = 0, Delta.T = -0.15, N1 = plan$N1, N2 = plan$N2, omega = 0.5, r = 1, alpha = 0.025, tar.power = 0.8, Test = 'Farrington-Manning', ss.method = 'standard', margin = 0.15 ) rows <- vapply(c(0.5, 0.7), function(t) which.min(abs(ni.tie$theta - t)), integer(1)) max(abs(on.boundary$power.BSSR - ni.tie$TIE.BSSR[rows])) ``` `BinaryAlphaAdjBSSR()` accepts the margin in the same way and adjusts the nominal level on the boundary. ## Margin on the scale of the risk ratio With `margin.scale = 'RR'` the margin is a ratio $R_{0} > 0$, and `alternative = 'greater'` tests $$ H_{0}: p_{1} / p_{2} \le R_{0} \quad \text{against} \quad H_{1}: p_{1} / p_{2} > R_{0} . $$ With `alternative = 'less'` the null hypothesis is $p_{1} / p_{2} \ge R_{0}$, as in the second example of Farrington and Manning (1990), where the relative risk of pertussis among recipients of an acellular vaccine compared with those of a whole cell vaccine is to be shown to be below 1.5. A margin on this scale also requires one of the two tests and a one-sided alternative, and in the functions with re-estimation the assumed and the true effects are then risk ratios (`effect = 'RR'`). Both tests refer statistic (5) of Farrington and Manning (1990), $$ Z = \frac{\hat{p}_{1} - R_{0} \hat{p}_{2}}{\widehat{\mathrm{SE}}} , \qquad \widehat{\mathrm{SE}}^{2} = \frac{\tilde{p}_{1}(1 - \tilde{p}_{1})}{N_{1}} + R_{0}^{2} \frac{\tilde{p}_{2}(1 - \tilde{p}_{2})}{N_{2}} , $$ to the standard normal distribution. The Blackwelder test takes $\tilde{p}_{1} = \hat{p}_{1}$ and $\tilde{p}_{2} = \hat{p}_{2}$, which is Method 1 of the article. The Farrington-Manning test takes the maximum likelihood estimates under the restriction $p_{1} = R_{0} p_{2}$. With $\theta = N_{2} / N_{1}$, the estimate $\tilde{p}_{1}$ is the smaller root of $$ (1 + \theta) x^{2} - \{R_{0}(1 + \theta \hat{p}_{2}) + \theta + \hat{p}_{1}\} x + R_{0}(\hat{p}_{1} + \theta \hat{p}_{2}) = 0 , $$ formula (13) of the article, and $\tilde{p}_{2} = \tilde{p}_{1} / R_{0}$. With $R_{0} = 1$ the restricted estimates are the pooled proportion, so the Farrington-Manning test is the chi-squared test. ```{r ratio-rr} rr.ratio <- BinaryRR(N1 = 60, N2 = 60, alpha = 0.025, Test = 'Farrington-Manning', margin = 0.8, margin.scale = 'RR') rr.one <- BinaryRR(N1 = 60, N2 = 60, alpha = 0.025, Test = 'Farrington-Manning', margin = 1, margin.scale = 'RR') rr.chisq <- BinaryRR(N1 = 60, N2 = 60, alpha = 0.025, Test = 'Chisq') c(ratio.0.8 = sum(rr.ratio), ratio.1 = sum(rr.one), chisq = sum(rr.chisq)) identical(as.vector(rr.one), as.vector(rr.chisq)) ``` `BinarySampleSize()` replaces the difference in the normal approximation by the distance $p_{1} - R_{0} p_{2}$ from the boundary, or $R_{0} p_{2} - p_{1}$ for `alternative = 'less'`. With the allocation ratio $r = N_{1} / N_{2}$ the size of group 2 is $$ N_{2} = \frac{\left(z_{1 - \alpha} \sqrt{v_{0}} + z_{1 - \beta} \sqrt{v_{1}}\right)^{2}}{(p_{1} - R_{0} p_{2})^{2}} , \qquad v_{1} = \frac{p_{1}(1 - p_{1})}{r} + R_{0}^{2} p_{2}(1 - p_{2}) , $$ where $1 - \beta$ is the target power and $v_{0}$ is the same expression at the large sample values of the restricted estimates. `method = 'standard'` gives formula (8) of Farrington and Manning (1990), and `method = 'alternative.variance'`, which sets $v_{0} = v_{1}$, gives the formula of their Method 1. The table below repeats the comparison of the sample sizes for equal response probabilities of 0.7 and the margin 0.8. ```{r ratio-sample-size} rr.ss <- expand.grid(method = c('exact', 'standard', 'alternative.variance'), Test = c('Blackwelder', 'Farrington-Manning'), stringsAsFactors = FALSE) rr.ss$N <- NA rr.ss$Power <- NA for (i in seq_len(nrow(rr.ss))) { ss <- BinarySampleSize(p1 = 0.7, p2 = 0.7, r = 1, alpha = 0.025, tar.power = 0.8, Test = rr.ss$Test[i], method = rr.ss$method[i], margin = 0.8, margin.scale = 'RR') rr.ss$N[i] <- ss$N rr.ss$Power[i] <- round(ss$Power, 4) } rr.ss ``` The second example of Farrington and Manning (1990), with $p_{1} = p_{2} = 0.01$, $R_{0} = 1.5$, `alternative = 'less'`, the one-sided level 0.025 and the power 0.9, needs 12,890 patients per group by formula (8), and its true power is 90.4 per cent. The rejection region at this sample size has more than $1.6 \times 10^{8}$ outcomes, so `inst/reproduce/reproduce-published.R` sums the exact power over the outcomes that carry the probability, and reproduces both values together with Tables I and II of the article. The re-estimation of Friede, Mitchell and Mueller-Velten (2007) carries over with the assumed ratio `Delta.A = 1`, which recovers the pooled proportion for both groups. The design below is planned for a pooled response probability of 0.7, re-estimates the sample size by formula (8) after half of the patients, and allows at most twice the planned sample size. ```{r ratio-bssr} plan.rr <- BinarySampleSize(p1 = 0.7, p2 = 0.7, r = 1, alpha = 0.025, tar.power = 0.8, Test = 'Farrington-Manning', method = 'standard', margin = 0.8, margin.scale = 'RR') ni.rr <- BinaryPowerBSSR( p = seq(0.4, 0.8, by = 0.1), Delta.A = 1, Delta.T = 1, N1 = plan.rr$N1, N2 = plan.rr$N2, omega = 0.5, r = 1, alpha = 0.025, tar.power = 0.8, Test = 'Farrington-Manning', effect = 'RR', ss.method = 'standard', N.max = 2 * plan.rr$N, margin = 0.8, margin.scale = 'RR' ) ni.rr ``` At equal response probabilities $p$ the distance from the boundary is $(1 - R_{0}) p$, so it shrinks with $p$, and a lower pooled response probability calls for more patients. The planned total sample size is `r plan.rr$N`. At a pooled response probability of `r ni.rr$p[1]` the expected total sample size of the re-estimation design is `r sprintf('%.1f', ni.rr$E.N[1])`, close to the upper bound of `r 2 * plan.rr$N`, and the power is `r round(ni.rr$power.BSSR[1], 3)` with re-estimation and `r round(ni.rr$power.TRAD[1], 3)` without. On the boundary $p_{1} = R_{0} p_{2}$ the pooled response probability is $\theta = (1 + r R_{0}) p_{2} / (1 + r)$, and both response probabilities are linear in $\theta$, so `BinaryTypeIErrorBSSR()` certifies the largest type I error rate as on the scale of the risk difference. With the default `theta` the interval runs from 0 to $\min(1, 1 / R_{0}) (1 + r R_{0}) / (1 + r)$, at which the larger response probability reaches 1. ```{r ratio-tie} tie.rr <- BinaryTypeIErrorBSSR( Delta.A = 1, N1 = plan.rr$N1, N2 = plan.rr$N2, omega = 0.5, r = 1, alpha = 0.025, tar.power = 0.8, Test = 'Farrington-Manning', effect = 'RR', ss.method = 'standard', N.max = 2 * plan.rr$N, margin = 0.8, margin.scale = 'RR' ) attr(tie.rr, 'interval') attr(tie.rr, 'max') plot(tie.rr) ``` The largest type I error rate is `r sprintf('%.4f', attr(tie.rr, 'max')$TIE[1])` for the re-estimation design, at $\theta =$ `r round(attr(tie.rr, 'max')$theta[1], 3)`, and `r sprintf('%.4f', attr(tie.rr, 'max')$TIE[2])` for the fixed-sample design, at $\theta =$ `r round(attr(tie.rr, 'max')$theta[2], 3)`. Both exceed the nominal level of 0.025, and the fixed-sample design exceeds it by more, so the excess comes from the asymptotic test rather than from the re-estimation. `BinaryAlphaAdjBSSR()` accepts `margin.scale` in the same way and lowers the nominal level until the rate is controlled. ## Tests of equivalence With `alternative = 'equivalence'` the null hypothesis $p_{1} - p_{2} \le -\delta_{L}$ or $p_{1} - p_{2} \ge \delta_{U}$ is tested against $-\delta_{L} < p_{1} - p_{2} < \delta_{U}$ by two one-sided tests, the test against `'greater'` with the margin $\delta_{L}$ and the test against `'less'` with the margin $\delta_{U}$, each at the level $\alpha$. The margins are given as `margin = c(lower, upper)`, or as a single value for both. The null hypothesis is rejected when both tests reject, so the rejection region is the intersection of the two one-sided regions. ```{r equiv-rr} eq <- BinaryRR(N1 = 80, N2 = 80, alpha = 0.05, Test = 'Farrington-Manning', alternative = 'equivalence', margin = 0.2) lo <- BinaryRR(N1 = 80, N2 = 80, alpha = 0.05, Test = 'Farrington-Manning', alternative = 'greater', margin = 0.2) up <- BinaryRR(N1 = 80, N2 = 80, alpha = 0.05, Test = 'Farrington-Manning', alternative = 'less', margin = 0.2) identical(as.vector(eq), as.vector(lo) & as.vector(up)) ``` The normal approximation to the sample size adds the approximate powers of the two one-sided tests and subtracts 1, which gives a lower bound for the approximate probability that both reject, and solves for the size at which this bound reaches the target power. The exact search of `BinarySampleSize()` starts from that size. ```{r equiv-ss} plan.eq <- BinarySampleSize(p1 = 0.4, p2 = 0.4, r = 1, alpha = 0.05, tar.power = 0.8, Test = 'Farrington-Manning', alternative = 'equivalence', margin = 0.2) plan.eq ``` In a re-estimation design the assumed difference `Delta.A` must lie between the margins, and is usually 0. The type I error rate is evaluated on both boundaries of the null hypothesis, $p_{1} - p_{2} = -\delta_{L}$ and $p_{1} - p_{2} = \delta_{U}$, and its largest value is certified on each. For normally distributed outcomes, Glimm, Yau and Woehling (2021) found that blinded re-estimation can inflate the type I error rate of a test of equivalence more than that of a test of superiority, because on a boundary away from no difference the blinded estimate of the variance depends on the true difference. For binary outcomes the rate can be computed exactly. ```{r equiv-tie} tie.eq <- BinaryTypeIErrorBSSR( Delta.A = 0, N1 = plan.eq$N1, N2 = plan.eq$N2, omega = 0.3, r = 1, alpha = 0.05, tar.power = 0.8, Test = 'Farrington-Manning', alternative = 'equivalence', ss.method = 'standard', margin = 0.2 ) attr(tie.eq, 'max') plot(tie.eq) ``` The largest type I error rate is `r sprintf('%.4f', attr(tie.eq, 'max')$TIE[1])` for the re-estimation design and `r sprintf('%.4f', attr(tie.eq, 'max')$TIE[2])` for the fixed-sample design of `r plan.eq$N` patients. The re-estimation design `r if (attr(tie.eq, 'max')$TIE[1] > attr(tie.eq, 'max')$TIE[2]) 'exceeds' else 'does not exceed'` the fixed-sample design, and both rates are computed for the same test of Farrington and Manning, whose fixed-sample rate `r if (attr(tie.eq, 'max')$TIE[2] > 0.05) 'already exceeds' else 'does not exceed'` the level of 0.05. With equal margins and equal allocation the two boundaries are mirror images and give the same rates. `BinaryAlphaAdjBSSR()` lowers the level of both one-sided tests until the rate is controlled on both boundaries. ```{r equiv-adj} adj.eq <- BinaryAlphaAdjBSSR( Delta.A = 0, N1 = plan.eq$N1, N2 = plan.eq$N2, omega = 0.3, r = 1, alpha = 0.05, tar.power = 0.8, Test = 'Farrington-Manning', alternative = 'equivalence', ss.method = 'standard', margin = 0.2 ) adj.eq ``` The adjusted level of each one-sided test is `r format(adj.eq$alpha.adj[1])` for the re-estimation design and `r format(adj.eq$alpha.adj[2])` for the fixed-sample design. ## References Blackwelder, W. C. (1982). "Proving the null hypothesis" in clinical trials. *Controlled Clinical Trials*, 3, 345-353. Farrington, C. P. and Manning, G. (1990). Test statistics and sample size formulae for comparative binomial trials with null hypothesis of non-zero risk difference or non-unity relative risk. *Statistics in Medicine*, 9, 1447-1454. Friede, T., Mitchell, C. and Mueller-Velten, G. (2007). Blinded sample size reestimation in non-inferiority trials with binary endpoints. *Biometrical Journal*, 49, 903-916. Glimm, E., Yau, L. and Woehling, H. (2021). Type I error inflation of blinded sample size re-estimation in equivalence testing. *Statistics in Biopharmaceutical Research*, 13, 156-169.