Login
Every year, the Centers for Disease Control and Prevention (CDC) estimates how effective last year’s flu vaccine was at preventing flu-related hospitalizations and deaths.
But are the CDC’s estimates accurate? Can you trust them?
Dr. Eyal Shahar, a professor emeritus of epidemiology at the University of Arizona’s Mel & Enid Zuckerman College of Public Health, says no — because the CDC uses what he says is an inherently biased study design to arrive at those estimates.
In an interview with The Defender, Shahar said that for years, the CDC has insisted on using a “problematic” test-negative study design, even though better designs exist.
“There are many supporters of the test-negative design, and the CDC simply adopted that design long ago,” Shahar said. “There are CDC-associated networks, all of which exclusively use that design to estimate effectiveness. It is time for a change.”
Shahar has authored nearly 200 papers, a book chapter on biases in epidemiological research, and two books, including a textbook on epidemiology and statistics. He has taught courses in methodology and is the former associate editor of the American Journal of Epidemiology.
A test-negative study design method is relatively easy and cheap, but it is known to produce biased and incoherent results, Shahar said.
Test-negative studies on flu vaccine effectiveness enroll people who sought medical care for flu-like symptoms and were tested for the flu. The researchers compare the vaccination status between those who tested positive and those who tested negative.
But that means the study is limited to people who sought medical care and were tested. That’s a form of selection bias, because the study excluded everyone who didn’t do those things. Limiting the study in this way can also cause collider bias, which could skew the results, Shahar said.
Shahar said better study designs would likely yield more accurate results, but they require more effort and money.
The ideal study design is a randomized trial, but some people consider it unethical to use this design when testing a vaccine’s effectiveness, he said.
“The best alternative is a well-designed cohort with two endpoints: symptomatic flu and asymptomatic flu,” he said.
Participants in the cohort design should be matched based on when they received the vaccine, he added.
In a recent post on Medium, Shahar analyzed a CDC test-negative study for the flu vaccine in 2022-2023, when the vaccine was well-matched to the dominant strain.
He showed how the study suffered from several kinds of statistical bias and produced results that contradicted what the researchers would logically expect to find.
The U.S. Food and Drug Administration (FDA) has commonly relied on test-negative studies when deciding whether to approve both flu and COVID-19 vaccines, Shahar said. That needs to change, he said.
Shahar wrote in an earlier Medium post:
“It is time for the FDA to implement a major revision in the approval of the Covid vaccine and the flu vaccine. It is not enough to show effectiveness against symptomatic infection by the test-negative design or by rising antibody titers.
“They need to see estimates of the effectiveness against both symptomatic infection and asymptomatic infection from a cohort design to distinguish coherent results from incoherent results.”
NIH director calls out same problem with CDC studies on vaccine efficacy
Shahar is not alone in criticizing the CDC for using a test-negative design to measure vaccine efficacy.
In early April, National Institutes of Health Director Jay Bhattacharya, who at the time was serving as the CDC’s interim director, initially delayed publishing a test-negative study on COVID-19 vaccine efficacy in the CDC’s long-running Morbidity and Mortality Weekly Report (MMWR).
Later that month, the MMWR outright rejected the study.
The study uses a test negative design that will not produce an unbiased estimate of efficacy (and it’s impossible to say which way the bias will go). MMWR should not be a place where such studies are published.
The study belongs to the authors. They can release it as they see fit, just not in MMWR.
— Jay Bhattacharya (@DrJBhattacharya) April 23, 2026
After the paper’s rejection sparked backlash on X, Bhattacharya wrote on X:
“There is no obligation for the MMWR — a non-peer-reviewed journal that serves as the voice of CDC policy — to publish a paper based on the statistically flawed test-negative design.
“The authors can do whatever they want with the paper, but they do not have a right to publish in MMWR just because they like their own method. Given its statistical flaws, no one should view the paper as providing an accurate measure of vaccine efficacy.”
Shahar, who personally knows Bhattacharya, said he recently shared with Bhattacharya some of his work on the CDC’s flawed flu vaccine efficacy studies.
The CDC has epidemiologists who are aware of a cohort design, he said. Still, the agency continues to employ the bias-prone design.
Meanwhile, Shahar continues to produce papers that expose how problematic the test-negative design is. “Publishing critiques is not easy,” he said.
He shared with The Defender the preprint for a paper about a newly described bias in a test-negative design. “I can’t get it published,” he said.
The Defender is 100% reader-supported. No corporate sponsors. No paywalls. Our writers and editors rely on you to fund stories like this that mainstream media won’t write. 
This article was funded by critical thinkers like you.
Cleveland Clinic: 2024-2025 flu shot increased risk of getting the flu
According to a Cleveland Clinic study published last year that used a cohort design, people who received a flu vaccine formulated for the 2024-25 flu season had a 27% higher risk of getting the flu than those who didn’t get the vaccine.
The study authors noted in their report that a strength of the study was that it didn’t rely on a test-negative design.
They wrote that test-negative design studies “systematically overestimate true vaccine effectiveness.”
The Cleveland Clinic study wasn’t the first to find that flu vaccines could actually increase the risk of becoming ill.
A 2012 peer-reviewed study published in Clinical Infectious Diseases showed that children who received the flu vaccine were at significantly increased risk of contracting non-flu respiratory virus infections.
“The flu shot has never really prevented transmission of the flu and in some years (i.e., last year) the efficacy of preventing the flu is actually negative,” said Brian Hooker, Ph.D., chief scientific officer at Children’s Health Defense, in an Oct. 5 interview with The Defender.
“Flu would blow through a vaccinated population with nearly the same ease as it would an unvaccinated population,” Hooker added.
Related articles in The Defender
- Georgia’s Largest Healthcare System Will Fire Employees Who Refuse Flu Vaccine
- Breaking: Moderna’s mRNA Flu Vaccine Gets Unanimous Thumbs-Up Despite Risks, Low Efficacy
- ‘Deeply Concerning’: This Year’s Flu Shots Led to 27% Higher Risk of Flu
- HHS Tells CDC to Yank ‘Wild to Mild’ Flu Vaccine Ad Campaign, Shift Focus to ‘Informed Consent’
