In economic research, a tiny statistical difference can have outsized consequences. Usually denoted with a star (or asterisks), that symbol serves as a quality stamp on one’s research, and the absence or presence of that star can sometimes make or break a career by influencing publication prospects—or even determining who gets hired.
Lamis Kattan, an Assistant Professor of Economics at Georgetown University in Qatar, has spent the early years of her career scrutinizing the machinery behind statistical significance. Her findings suggest that the power of the star has become a structural flaw that distorts both who gets hired and what we accept as truth.
“I’ve been involved in attending research replication conferences since my Ph.D., which essentially looked at old data in a new light,” Dr. Kattan says. These collegial spaces, where younger and more seasoned economists come together to share ideas, have increasingly become places where alarm bells are sounding. “We have been having many conversations about data, transparency, AI, and the future of empirical work.”
Institute for Replication workshops in progress with scholars from around the world trying to replicate publicly available research
In a 2026 study published in the European Economic Review, “Job Market Stars: Statistical Significance and Academic Hiring in Economics,” Dr. Kattan and her colleagues tracked 200 young economists on the job market. They found that candidates whose research crossed the statistical significance threshold—even by the slimmest margin—were far more likely to land prestigious academic positions than those who fell just short.
“The difference between a p-value of 0.09 and 0.10 is statistically almost meaningless, but it appears to carry real weight in hiring committees’ decisions,” Dr. Kattan observes. “It suggests that intellectually honest researchers may be penalized.”
This incentive structure creates an ecosystem where “p-hacking”—the practice of trying different analytical approaches until a statistically significant result emerges—can become a survival mechanism. It is rarely a case of malicious fraud, notes Dr. Kattan. “Some of these choices are legitimate and even necessary. The problem is that, in an environment where significant results are rewarded, these choices can become shaped by the desire to find it.”
These incentives don’t just affect researchers’ careers—they also shape the scientific record itself.
In Nature’s “Reproducibility and Robustness of Economics and Political Science Research,” Dr. Kattan and her colleagues at the Institute of Replication re-examined 110 published studies to determine whether their findings actually held up. On one hand, they found that journals requiring mandatory replication packages and dedicated data editors fostered much higher rates of computational reproducibility.
Yet when the researchers reanalyzed the underlying data, the cracks in the foundation widened. Nearly one in six studies whose findings had previously been deemed statistically significant failed to hold up upon reanalysis. Eight papers even produced results pointing in the opposite direction, with the potential to alter policy conclusions and reshape the foundational knowledge passed on to students.
The findings underscore the importance of critical thinking now more than ever.
“I tell my international affairs students that statistical significance is not the same as truth,” Dr. Kattan says. “As future policymakers and leaders, I want them to read the data and ask whether the story is visible before sophisticated econometric methods are used.”
Here’s where the implications become even more concerning. AI systems powering chatbots, search engines, and research tools are trained on existing academic literature. If that literature is systematically skewed—filled with results that appear cleaner and more definitive than they actually are—then AI learns that distorted picture as though it were reality. It’s like training a student using a textbook full of subtle errors. The student doesn’t know the errors are there; they simply learn them as facts and repeat them confidently.
The good news, according to Dr. Kattan, is that while AI is making p-hacking easier, it is also making it easier to detect data or methodological manipulation. While it may be too early to tell if an equilibrium can be reached, her research indicates that the solution is analog: centering human-human relationships in scholarship.
“There’s more awareness of the need for transparency,” she says. Journals that require researchers to publicly share their raw data and methods consistently show much higher rates of honest, reproducible research. When other scientists can examine and verify your work, the incentives to manipulate results diminish.
Dr. Kattan’s message to the next generation of researchers is simple: stop chasing the star.
“A non-significant result can still be important if the question matters and the design is credible,” she says. “In the long run, careful and transparent work is more valuable than a result that looks clean only because the messy parts were hidden.”
Read more about Dr. Kattan’s career, and her advice for women considering careers in economics