There are some contentious and unsettled issues in the statistics of vaccine development. Here we discuss two of these that appear to be most prominent in vaccine debates. They are the small sample size issue and the so called “non-inferiority” tests.

The small sample size issue arises from conflict between (A) the safety issue in the administration of the tested vaccine and (B) the safety issue in the test itself.  Ideally we would like to have a vaccine with a high level of confidence in (A), the safety of the tested vaccine; but that would require putting a large number of people at risk in the test (B). This difficult balancing task typically leads to the use of smaller test sample sizes than what we would use if we didn’t care about the welfare of the humans we are testing it on. This is the so called “small sample size” problem. The small sample size problem remains a serious and unresolved issue in vaccine development not because vaccine developers are evil but because there is no easy answer to a difficult question.

The other unsettled statistical issue in vaccine development has to do with “non-inferiority” tests. This issue involves a tested and FDA approved vaccine where some modifications have been made that are considered minor, as for example a change in the formulation with equivalent material or a change in the manufacturing process. In such cases the vaccine must be re-tested. However, since the vaccine that was altered had already been proven safe and is an FDA approved vaccine, there is some question as to what the null hypothesis should be in the hypothesis test. Strictly speaking, the null hypothesis in hypothesis tests is always the negation of what you believe is true and in this case the null hypothesis must be the same as in the initial test and that is that the vaccine is not safe. However, since the vaccine has already been proven safe and since the only question is whether a minor change in the manufacturing process or change in materials supplier has made a significant change, the null hypothesis used is that there is no change. Thus only the rejection of this null hypothesis implies the possibility of harm and, in an anomalous way, the “fail to reject” condition actually proves the safety of the vaccine. This is a serious but unresolved anomaly in the statistics of vaccine development, but, as in issue-A, it has no easy answer and no simple solution. However, at the minimum, statistical tests where fail to reject is the criterion for safety, a high value of  α should be used as for example α=0.10 to ensure that any reasonable ability to discriminate is not overlooked. Here is an example in climate science [LINK] .

Conclusion: Vaccine development statistics issues are like no other medical statistics issue because of the involvement of human subjects in potentially very harmful tests and because of the need to continually modify manufacturing procedures to stay abreast of technological and materials availability issues in the area of vaccine manufacture and distribution.

See also:

https://us17.campaign-archive.com/?u=2c6057c528fdc6f73fa196d9d&id=905babc439&e=3624e5b785

Nature Briefing on vaccine development.

Excerpt: More than 90 vaccines for the coronavirus are at various stages of development, and at least six are being tested for safety in people. Now, developers, funders and other stakeholders are laying the groundwork for their biggest challenge yet: determining which vaccines actually work. The World Health Organization has proposed an adaptive trial design that allows vaccines to be added and dropped on an ongoing basis. The agency still has to work out which vaccines to test first, and how to convince drug developers to have their products pitted against each other. Large trials are usually necessary to determine safety and efficacy. An alternative is to administer vaccines that look safe in early-stage trials to high-risk groups — such as health-care workers — under ‘emergency use’ rules.

See also  https://www.cdc.gov/vaccines/basics/test-approve.html

Excerpt: Clinical development is a three-phase process. During Phase I, small groups of people receive the trial vaccine. In Phase II, the clinical study is expanded and vaccine is given to people who have characteristics (such as age and physical health) similar to those for whom the new vaccine is intended. In Phase III, the vaccine is given to thousands of people and tested for efficacy and safety. Many vaccines undergo Phase IV formal, ongoing studies after the vaccine is approved and licensed.

See also: https://www.ncbi.nlm.nih.gov/pubmed/8483112

Excerpt: We discuss in detail factors that influence sample size. Factors most influential are the incidence rate of HIV infection in the study population and the minimum efficacy at which a vaccine is still considered acceptable. The smaller either of these factors is, the larger the sample size will be.

A few other issues in noninferiority tests are mentioned in the literature exemplified by the Wang etal 2006 paper below:

Wang, W. W. B., et al. “Statistical considerations for noninferiority/equivalence trials in vaccine development.” Journal of biopharmaceutical statistics 16.4 (2006): 429-441.  Noninferiority/equivalence designs are often used in vaccine clinical trials. The goal of these designs is to demonstrate that a new vaccine, or new formulation or regimen of an existing vaccine, is similar in terms of effectiveness to the existing vaccine, while offering such advantages as easier manufacturing, easier administration, lower cost, or improved safety profile. These noninferiority/equivalence designs are particularly useful in four common types of immunogenicity trials: vaccine bridging trials, combination vaccine trials, vaccine concomitant use trials, and vaccine consistency lot trials. In this paper, we give an overview of the key statistical issues and recent developments for noninferiority/equivalence vaccine trials. Specifically, we cover the following topics: (i) selection of study endpoints; (ii) formulation of the null and alternative hypotheses; (iii) determination of the noninferiority/equivalence margin; (iv) selection of efficient statistical methods for the statistical analysis of noninferiority/equivalence vaccine trials, with particular emphasis on adjustment for stratification factors and missing pre- or post-vaccination data; and (v) the calculation of sample size and power.