AI and Research 

Artificial intelligence has changed academic research very quickly. ChatGPT, Gemini, and similar tools can now suggest statistical tests, generate SPSS or R code, help organize and clean data, interpret statistical output, prepare tables, and even draft complete methodology and results sections.

​

I am a strong believer in using technology to make research more efficient. AI can be extremely helpful, and its capabilities are improving rapidly. However, after more than 25 years working with statistics, research, dissertations, and academic studies, I see an important distinction that researchers should keep in mind:

Producing a statistical analysis is not the same as knowing that the analysis is correct.

​

This becomes particularly important when someone with limited statistical training asks AI to make most of the analytical decisions. The output can look excellent. The tables are professional, the terminology sounds correct, and the explanation may be written with great confidence. There may even be perfectly reasonable-looking p-values, confidence intervals, and effect sizes. The problem is that the analysis can still be wrong. And, in my experience, the most concerning errors are not always the obvious ones. They are the small, hidden mistakes that produce completely believable results.

​

The AI Only Knows What You Tell It

Selecting the correct statistical analysis requires much more than telling an AI which variables are continuous and which are categorical. Before I select an analysis, I need to understand the research questions, hypotheses, study design, sampling method, instruments, how the variables were measured, how participants were selected, whether observations are independent or related, whether measurements were repeated, how missing data occurred, and what conclusions the researcher is trying to make. AI does not automatically know any of this.

More importantly, a researcher who has limited statistical experience may not know which of these details are important enough to tell the AI. That can create a problem that is easy to miss. The AI may provide a perfectly reasonable answer to an incomplete or incorrectly formulated question.

 

Consider something as simple as comparing two means. The appropriate analysis can be very different depending on whether those two means come from two independent groups, the same participants measured before and after an intervention, matched participants, or observations clustered within hospitals, classrooms, or other settings. The variables may look almost identical in the dataset. Statistically, however, these are very different situations. If the AI is not given the correct structure of the study, it can recommend an analysis that looks completely legitimate but is inappropriate for the research.

 

Sometimes One Small Mistake Reveals a Much Larger Problem

One thing that concerns me about unsupervised AI analysis is how convincing an incorrect analysis can look.

Statistical software will usually perform the analysis it is instructed to perform. It does not stop and ask whether the analysis actually answers the research question. AI can add another layer by providing a polished explanation of that analysis. The result may therefore look much more trustworthy than it actually is.

​

After years of reviewing statistical work, an experienced eye can often spot something that does not fit.

Sometimes it is surprisingly simple: a sophisticated results section contains an elementary statistical error. The sample size changes from one paragraph to another without explanation. A paired study is analyzed as though the groups were independent. The text reports one statistical test while the table clearly contains the output from another. A regression suddenly contains variables that were never mentioned in the research questions. A questionnaire score has been calculated incorrectly. A categorical code such as 1, 2, and 3 has been treated as though the numerical distances themselves have substantive meaning.

 

Another common warning sign is an interpretation that sounds impressive but is not supported by the analysis. For example, the results may say that one group “improved significantly more” than another when the researcher only tested improvement separately within each group. Neither test, by itself, answers the question of whether the amount of improvement differed between the groups. These may appear to be small details, but sometimes one small mistake tells me that I need to examine the entire analysis more carefully.

​

I would not claim that anyone can reliably determine from a document alone whether AI was used. That is neither necessary nor the important issue. What an experienced statistician can often recognize are patterns suggesting that the analysis was assembled without a full understanding of the statistical decisions behind it.

Whether the mistake originated from AI, software, a researcher, or someone else is secondary. What matters is whether the analysis is correct.

​

The Most Difficult Statistical Mistakes Are Often Hidden

There are many ways an analysis can go wrong without producing an error message.

I have seen problems involving repeated observations being treated as independent cases, incorrect questionnaire scoring, failure to reverse-code items, inappropriate treatment of missing data, incorrect group definitions, inappropriate exclusion of outliers, confusion between dependent and independent variables, and statistical procedures that simply do not answer the stated research question.

There can also be more subtle issues: clustering may have been ignored, important confounding variables omitted, multiple comparisons conducted without considering their consequences, or conclusions extended far beyond what the study design allows. All of these analyses can still generate numbers. That is an important point. A p-value appearing in an output does not establish that the correct question was tested.

The calculation itself may be mathematically correct while the analysis is scientifically inappropriate.

​

Statistical Assumptions Are Not Just Boxes to Check

AI is very good at listing assumptions. Ask for the assumptions of multiple regression or analysis of variance and it can provide them almost instantly. Applying those assumptions correctly is more complicated.

For example, researchers are sometimes told that because their data are “not normally distributed,” they must automatically abandon a parametric analysis. That is an oversimplification. The importance of normality depends on the particular model, what component of the model the assumption concerns, the sample size, the research design, the severity of the departure, and the robustness of the statistical procedure. The same applies to heteroscedasticity, influential observations, linearity, independence, and other model assumptions.

An experienced statistician does not simply run a series of assumption tests and allow their p-values to decide which analysis to use. The diagnostics have to be interpreted in the context of the data and the research question.

​

Data Cleaning Requires Judgment Too

AI can be very useful for identifying missing values, unusual observations, duplicates, and possible coding errors. But identifying something unusual and deciding what should be done about it are different matters.

An extreme observation is not automatically a “bad” observation that should be deleted. It could be a data-entry error. It could also be a completely legitimate participant, an important clinical observation, evidence that the assumed statistical model does not fit the data well, or even one of the most interesting findings in the study. Missing data create similar problems. Deleting every incomplete case or automatically replacing missing observations with an average value can introduce problems of its own.

These decisions require an understanding of where the data came from and why they look the way they do.

​

Good Writing Can Make a Weak Analysis Look Strong

One of the greatest strengths of modern AI is also one of the reasons researchers need to be careful with it: AI writes extremely well. A paragraph can sound authoritative even when the statistical interpretation is too strong.

A statistically significant result does not necessarily mean that an effect is important. A nonsignificant result does not prove that there is no effect. Statistical significance and clinical significance are not the same thing. An association does not automatically demonstrate causation. The design of the study determines what conclusions can reasonably be drawn from the analysis. This is especially important in dissertations, doctoral projects, clinical studies, and manuscripts where reviewers are not simply looking for a p-value. They want to know whether the statistical evidence actually supports the researcher's conclusion.

​

References and Numerical Results Also Need Verification

AI can occasionally provide information that sounds completely genuine but is not. This is commonly described as hallucination. References are a particularly good example. An AI-generated citation can contain plausible authors, a convincing article title, a journal name, volume, pages, and even something resembling a DOI—and still be incorrect or nonexistent. Every important reference should therefore be checked against the original publication or a reliable bibliographic source. I apply the same principle to statistical results. If a Results section reports n = 84, a mean of 17.4, a regression coefficient of 0.38, p = .012, and a 95% confidence interval, I should be able to trace those numbers back to the actual analysis. A beautifully written paragraph is not evidence that those calculations were performed correctly.

​

Asking One AI to Check Another AI Is Helpful—but It Is Not Independent Statistical Review

Researchers increasingly use several AI platforms. They may conduct an analysis with one system and then paste the results into another and ask, “Is this correct?” That can certainly help. I use multiple forms of verification in my own work as well. But two AI systems agreeing with each other does not prove that an analysis is appropriate. If an important feature of the research design was omitted from the original description, neither AI system knows it. Both can therefore reach the same conclusion from the same incomplete information. The missing information may be precisely the detail that changes the analysis.

​

One Question I Would Ask Every Researcher Using AI

If you have relied substantially on AI for your statistical analysis, ask yourself one simple question:

If the AI made a subtle statistical mistake, would I recognize it?

If you are confident that you understand the analysis well enough to answer yes, AI can be an extremely powerful research tool.

If you are not sure, that does not mean your analysis is wrong. It means that an independent statistical review may be worthwhile before the work is submitted.

The purpose of that review is not to prove that AI made mistakes. In many cases, much of the AI-assisted work may be perfectly good.

The purpose is to determine which parts can be trusted.

​

How I Can Help If You Have Already Used AI for Your Research

I expect more students and researchers will use AI in their research, not fewer. For that reason, I offer statistical review and verification for research that has already been partially or substantially completed using ChatGPT, Gemini, or other AI tools. You do not necessarily need to start again. You can send me the work you already have. Depending on the stage of your project, this might include your proposal or methodology, research questions and hypotheses, original dataset, questionnaire or other instruments, SPSS/R/Stata/SAS output, tables, AI-generated analysis, and draft Results section. I can then review the project as an integrated piece of research rather than simply checking whether individual calculations look reasonable.

​

I Can Review Your Data Analysis Plan Before You Run the Analysis

If you are still at the proposal or methodology stage, I can review and complete your proposed data analysis plan.

I will examine your research questions and hypotheses, identify the variables involved, review how they are measured, and determine which statistical procedures are appropriate. This is an important stage to get right. It is much easier to establish a defensible analysis plan before conducting dozens of unnecessary or inappropriate tests than to repair the analysis afterward.

​

I Can Audit an Analysis That AI Has Already Completed

If you already have an analysis, I can independently check it. I can examine whether the tests selected by AI are appropriate for your research questions and design, whether your variables were coded correctly, whether questionnaire scores were calculated properly, whether relevant assumptions were considered, and whether important features such as paired observations, repeated measurements, clustering, missing data, or confounding variables were handled appropriately. I am not simply asking AI whether another AI's work looks correct. I am reviewing the underlying research logic.

​

I Can Reproduce the Results From Your Original Data

When necessary, I can work from the original dataset and independently reproduce the important analyses.

This provides a much stronger form of verification. I can check whether the sample sizes, means, standard deviations, test statistics, p-values, confidence intervals, regression coefficients, effect sizes, and other reported values can actually be reproduced. If my results do not agree with the existing analysis, I can investigate why.

Sometimes the difference comes from a simple coding or filtering decision. Sometimes it reveals a more substantial methodological problem.

​

I Can Compare the Results Section With the Actual Statistical Output

If AI has written your Results section, I can review it against the actual statistical output. I can check the numerical values, statistical terminology, interpretation, tables, sample sizes, degrees of freedom, significance tests, confidence intervals, and effect sizes. Just as importantly, I can check whether the written conclusions accurately represent what the analysis found. This is where subtle mistakes can easily enter an otherwise polished Results section.

​

I Can Check the Entire Chain From the Research Question to the Conclusion

For a complete review, I examine the relationship among:

Research Question → Study Design → Variables and Data → Statistical Analysis → Output → Results → Conclusion.

These components should tell one consistent statistical story. A test can be calculated correctly and still be wrong for the research question. A table can contain the correct values while the paragraph interpreting it is wrong. The Results section can be correct while the Discussion makes a causal claim that the design cannot support. Looking at the complete chain is therefore much more informative than checking isolated p-values.

​

If I Find Problems, I Can Help Correct Them

My service does not have to end with a list of mistakes. If the existing analysis needs correction, I can determine the appropriate statistical approach, rerun the analyses where necessary, correct tables and figures, and revise the statistical reporting. If only a small part of the analysis is problematic, there may be no reason to redo everything. The objective is to preserve the work that is correct and repair the parts that are not. This can be particularly useful for a student who has spent considerable time working with AI and has reached the point where the analysis looks complete but they are no longer certain which parts they can confidently defend before their supervisor or committee.

​

A Final Review Before Submission Can Prevent Much Larger Problems Later

A statistical review can be particularly valuable before submitting a dissertation or thesis chapter, completing a doctoral project, attending a dissertation defense, submitting a manuscript to a journal, or responding to a reviewer who has questioned the statistical analysis. It is much better to identify a seemingly small statistical error before submission than to have a committee member or peer reviewer identify it afterward.

Sometimes one small mistake raises questions about everything that follows it.

​

AI and the Statistician Should Work Together

I do not see AI as the enemy of statistical consulting. I see it as another powerful research tool.

I use modern technology in my own work, and I expect AI to become increasingly useful for researchers and statisticians. But AI does not replace the need to understand the study. AI can generate code in seconds. It can suggest ten different analyses. It can explain a statistical procedure beautifully. What still matters is knowing which analysis should be performed, why it should be performed, whether it was performed correctly, and what the results actually allow us to conclude. That is where statistical experience remains important. For researchers, the most productive approach is not necessarily AI or a statistician. It is often AI combined with an experienced statistician who can verify the work.

​

Have You Already Used ChatGPT or Another AI Tool for Your Research?

If you have already used AI to develop your methodology, select statistical tests, analyze your data, interpret your output, or write your Results section, you do not necessarily need to redo the entire project. You can send me what you have already completed. I can independently review the work, identify any hidden statistical or methodological problems, verify the important analyses against your original data, and tell you clearly what is correct and what needs to be changed. Where corrections are necessary, I can also help complete the analysis and prepare statistically accurate results that you can understand and defend.

​

Dr. Ron Fisher
Fisher Statistics
Professional Statistical Analysis,

Research Support, and Independent Statistical Consultat 
FisherStat.com

​

References

Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804

van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023). ChatGPT: Five priorities for research. Nature, 614, 224–226. https://doi.org/10.1038/d41586-023-00288-7

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108

Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019). Moving to a world beyond “p < 0.05.” The American Statistician, 73(sup1), 1–19. https://doi.org/10.1080/00031305.2019.1583913

Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Biles, C., Brown, S., Kenton, Z., Hawkins, W., Stepleton, T., Birhane, A., Hendricks, L. A., Rimell, L., Isaac, W., et al. (2022). Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 214–229). Association for Computing Machinery. https://doi.org/10.1145/3531146.3533088