By Alan R. Gaby, M.D.

During the past 49 years, I have reviewed and analyzed more than 50,000 papers from the biomedical literature, most of which were related to the field of nutritional medicine. Doing this work has given me some understanding of how to assess the reliability of a study. Over the past 10 to 15 years, an uncomfortably large and growing number of published papers related to my area of expertise have left me wondering whether the research was fabricated; that is, whether people were writing papers about research that had not actually been conducted. If the studies were not actually conducted, the publishing of this research is an affront to all who value integrity in science, and it has the potential to harm practitioners and patients who rely on its findings.

The studies that have raised concerns have come primarily from Iran and to a lesser extent from Egypt, China, India, Japan, and a few other countries. Characteristics of these concerning studies typically include one or more of the following:

1. The study comes from an investigator or research group that has published an enormous number of randomized clinical trials in a relatively short period of time.
2. The number of participants in the trial is unusually large, when considering the resources that appear to be available to the researchers.
3. The recruitment period for the trial is unusually short.
4. The paper is submitted to a journal unusually rapidly after the study is completed, or in some cases before it would have been possible to have completed the trial.
5. A randomized double-blind trial is conducted before there is any preliminary evidence of efficacy in humans (such as case reports or uncontrolled trials). Because double-blind trials are expensive to conduct, such trials are generally reserved for treatments for which there is some evidence of efficacy.
6. The magnitude of the reported improvement is much larger than is typically seen in trials using just one or two nutrients.
7. No funding source is listed or the study is listed as self-funded. This is of particular concern when the sample size or study design suggests that the study was expensive.
8. The design of the study raises ethical issues, such as participants not being permitted to use treatments that are known to be effective.
9. One or more baseline characteristics of the study group appear to be implausible.
10. The research was conducted by a student as part of a graduate school thesis, and the magnitude of the project seems to have been beyond the capabilities and resources of a student.

This tenth characteristic was included in part because of an article published in the journal Science in 2016. That article, which was titled, “In Iran, a shady market for papers flourishes,” revealed that numerous companies in Iran will write theses and scientific papers for a fee. According to one Iranian scientist, many politicians and diplomats obtain advanced degrees from universities without actually earning them. Having an advanced degree allows them to compete better for scarce jobs in the science industry, and the law mandates that individuals with these degrees earn a higher salary for the same job. One company offered to have a paper placed in a journal published by Elsevier or Springer, for a fee of $1,660. The Science article stated that many young scientists who are unable to find another job are involved in the paper-selling business.

A few of the many papers that have raised concerns are examined below. 

Prolific Iranian researcher

A researcher might be considered prolific if they were involved in, say, half-a-dozen or so randomized clinical trials over a period of a decade. In contrast, Iranian researcher Zatollah Asemi and coworkers have published (by my count) at least 191 randomized clinical trials, including 148 between January 1, 2016 and March 30, 2019. The implausibility of Asemi’s enormous productivity is compounded by the fact that nearly every study reported positive results (often with a large effect size), regardless of the nutrient being tested or the condition being treated.

In reviewing about half of the papers written by Asemi and coworkers, I identified 12 papers that were submitted to the journal before it would have been possible to have completed the study.[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] For example, in one study the recruitment period was June 2017 to August 2017 and the treatment being studied was administered for 12 weeks. If recruitment ended on August 1, 2017, the earliest the study could have been completed would have been October 24, 2017. However, the paper was received by the journal on September 27, 2017.²

In a letter to the Journal of Clinical Endocrinology and Metabolism[13] I called attention to 2 papers that Asemi and coworkers had submitted before it was possible to have completed the studies.^{1 9} In their response to my letter,[14] Asemi et al changed the recruitment dates that had been listed in those 2 papers. One of the changes was to shorten the recruitment period in a gestational diabetes study from 2 or 3 months originally, down to 3 weeks. That is, during a 3-week period, they recruited 60 women from a single clinic who were between 24 and 28 weeks’ gestation and had gestational diabetes. After reading their response, I found data on the birth rate in the region of Iran where the study took place. I also found data about the proportion of pregnant women in that region who develop gestational diabetes. Based on that information I calculated that, even if the study had enrolled every woman with gestational diabetes in the entire city where the study was conducted (population, approximately 500,000), no more than 36 women could have been enrolled in a 3-week period.

Three other papers from Iran, none of which Asemi was involved with, are discussed below.

N-acetylcysteine for ulcerative colitis

Iranian patients (n = 168) with severe acute ulcerative colitis who had gone into remission after treatment with prednisolone (1 mg/kg/day) and oral mesalamine for 4 weeks were randomly assigned to receive, in double-blind fashion, 400 mg of N-acetylcysteine (NAC) twice a day or placebo for 16 weeks, and were then followed for an additional 6 weeks. At the start of NAC or placebo treatment, prednisolone was gradually tapered and discontinued. During 22 weeks of follow-up, the relapse-free period was significantly longer (p = 0.007), and the proportion of patients who had an endoscopic relapse was significantly lower (p < 0.001) in the NAC group than in the placebo group.[15] There are numerous issues with this paper that raise questions about its credibility.

1. Issue related to the publication date: According to the Iranian Registry of Clinical Trials document associated with this study, patients were recruited from July 23, 2019 through April 20, 2020. The study included 4 weeks of high-dose prednisolone, followed by 16 weeks of NAC or placebo, followed by 6 weeks of observation. Thus, the earliest date the study could have been completed would have been October 19, 2020. Presumably, it would take at least a month or two for the data to be analyzed, the paper to be written, and the journal to accept and publish the paper. However, the paper appeared in print on October 14, 2020, even before it was possible to have completed the study.
2. The number of study subjects seems implausibly large: Patients were ineligible to participate if they did not have pancolitis (which excludes about 80% of ulcerative colitis patients) or if they had been treated in the past 6 months with a tumor necrosis factor inhibitor or azathioprine (both of which are common treatments for ulcerative colitis). In addition, patients were eligible only if they had been treated with a specific drug regimen (high-dose prednisolone plus mesalamine) and if they had gone into remission from that treatment. Moreover, eligible patients had to agree to visit the clinic every 2 weeks for 16 weeks to have their blood drawn and to donate a stool sample. It is likely that many patients would not be willing to enroll in such a tedious and time-consuming study. Considering these various obstacles to enrollment, the researchers would likely have had to treat many thousands of acute severe flare-ups at a single clinic over a 9-month period in order to enroll 168 patients.
3. Implausible diagnostic testing: The paper stated that pancolitis was verified endoscopically in all enrolled patients just before the start of the study. However, the paper also stated that all patients had been diagnosed previously, based on standard diagnostic criteria including endoscopy. It does not seem plausible that every patient who presented with an acute flare-up of a previously diagnosed disease would immediately be given another colonoscopy.
4. Implausible treatment regimen: The paper stated that every patient was started on prednisolone at a dosage of 1 mg/kg/day for 4 weeks. Prednisolone was then tapered by 5 mg per day every week until a dosage of 20 mg per day was reached, after which the dosage was tapered by 2.5 mg per day every week until it was discontinued. For a 70-kg person, it would take 10 weeks to taper to 20 mg per day, and an additional 4 weeks to taper to 10 mg per day (a total of 14 weeks to reach 10 mg per day). However, the paper also stated that the dosage was reduced to less than 10 mg per day in the third month (“in,” not “after” the third month). “In the third month” would be around weeks 9 to 13. It would be impossible to have decreased the dosage to less than 10 mg per day in weeks 9 to 13, if it took 14 weeks to taper to 10 mg per day.
5. Illogical diagnostic criterion: Clinical remission was defined in part as having at least 3 non-bloody stools per day. While cessation of bleeding is a logical criterion for remission, it is likely that some patients in remission would not have 3 or more bowel movements per day.
6. Funding issue: Because double-blind studies are expensive, it is unusual to conduct a double-blind study when there has been no prior evidence from case reports or uncontrolled trials that the treatment being tested is effective. This study was particularly expensive, because it included more than 300 colonoscopies and more than 6,000 laboratory tests. Conducting lab tests on every patient every 2 weeks for 16 weeks seems excessive and an unwise use of resources. One wonders why anyone would have funded this study.

Peppermint oil gel to prevent pressure sores

A student in Semnan, Iran, as part of a master’s of science thesis, published a double-blind trial that examined the effect of topical peppermint oil on the incidence of stage 1 pressure injury (pressure sores) in critically ill patients.[16] One hundred fifty patients admitted to a neurosurgical intensive care unit (ICU) because of head trauma, who required endotracheal intubation, were randomly assigned to have a gel containing peppermint oil or a placebo gel applied to at-risk areas of their skin 3 times per day. The incidence of stage 1 pressure injury was significantly lower in the peppermint group than in the placebo group (23% vs. 77%). There are a number of aspects of this study that raise concerns.

1. Discrepancy related to the number of subjects: The sample size was listed as 150 in the paper and 70 in the document submitted to the Iranian Registry of Clinical Trials (IRCT).
2. Unusually low mortality rate: In most studies of patients with severe head trauma, the 30-day mortality rate was 20-50%. While the peppermint study only followed patients for 14 days, the mortality rate of 1.3% (2 of 150 patients) seems unusually low.
3. Issues related to the production of the peppermint oil gel: The paper stated that the peppermint gel was prepared “according to standard methods,” as described in Appendix A. Briefly, peppermint oil was added to 5 different concentrations of carbopol 934 in water. Carbopol 934 is a polymer that is used as a thickening and gelling agent. The carbopol 934 concentration that produced the most desirable viscosity (not too runny, not too thick) was used to make the peppermint gel. The 190-word description of this “standard method” was taken verbatim (without attribution) from a 2007 paper on the use of peppermint gel to prevent nipple cracks in breastfeeding women.[17] This plagiarism included misspelling of the word “viscous” as “viscose.” Putting aside the issue of plagiarism, one wonders why the researchers bothered to go through the process of testing 5 different carbopol 934 concentrations, when the 2007 study had already determined which concentration works best.
4. Issues related to the study sites: All 4 authors listed in the paper were affiliated with Semnan University of Medical Sciences in Semnan, Iran. However, the study was conducted at 2 ICUs in Shiraz, Iran, which is more than a 9-hour drive from Semnan. None of the study authors listed an affiliation with any institution in Shiraz. In addition, the study was approved by the ethics committee of Semnan University of Medical Sciences. It is surprising that ICUs in Shiraz, which were treating critically ill patients, would have agreed to conduct a study without first seeking approval from their own ethics committees.
5. Discrepancy related to the outcome measure: The stated outcome measure in the published paper was the incidence of stage 1 pressure injury, as defined by the National Pressure Ulcer Advisory Panel. In contrast, the IRCT registration document stated that the outcome measure was the incidence of pressure ulcer. Stage 1 pressure injury is defined as “intact skin with a localized area of nonblanchable erythema, which may appear differently in darkly pigmented skin.” These lesions are not ulcers, although they may eventually progress to become ulcers.
6. Issue related to funding: Because randomized clinical trials are expensive to conduct, such trials are generally reserved for treatments for which there is some evidence of efficacy (such as case reports or uncontrolled trials). There had been no prior evidence that peppermint oil gel could prevent pressure injury.
7. Issue related to the assessment method: The paper stated that a pair of observers (nurse plus specialist) examined at-risk areas of the skin (hips, sacrum, back, elbows, knees, heels, and shoulders) once day for evidence of pressure injuries. It was not stated what was meant by “specialist;” one might assume that referred to the neurologist or neurosurgeon involved with the case. It is difficult to believe that such a “specialist,” who would be busy evaluating and treating a roomful of seriously ill patients, would agree to spend extra time checking the hips, sacrum, back, elbows, knees, heels, and shoulders every day for early evidence of pressure injuries (that would likely be the job of the nurse), particularly when the request to perform such an examination was coming from a graduate student from a faraway university who was studying a treatment for which there was no prior evidence of efficacy.

Green tea for acute cystitis

Seventy Iranian women experiencing their first episode of acute uncomplicated cystitis (diagnosed by an infectious disease specialist, based on medical history, physical examination, clinical manifestations, and urinalysis) were randomly assigned to receive, in double-blind fashion, 2 g of a dried aqueous extract of green tea or placebo once a day for 3 days. All patients were treated with trimethoprim-sulfamethoxazole. After 3 days, the proportion of women who had symptoms of cystitis was 63% in the placebo group and 2% in the green tea group (p = 0.0001).[18]

This study raises several concerns.

1. One wonders why an infectious disease specialist would spend their time evaluating and treating first episodes of acute uncomplicated cystitis, a condition that is often treated by mid-level practitioners or even over the telephone.
2. Only 72% of the women were experiencing cystitis symptoms at the time they were enrolled in the study. That fact makes one wonder why the other 28% would have gone to a medical clinic, and why they were given a diagnosis of cystitis (as opposed to asymptomatic bacteriuria).
3. Ninety-one percent of the women in the placebo group had bacteriuria at baseline, which suggests that in the vast majority of cases the acute cystitis was due to a bacterial infection. Symptoms of an acute first-episode urinary tract infection typically resolve within 24-48 hours after the start of antibiotic treatment. It is therefore implausible that 63% of the women in the placebo group remained symptomatic 72 hours after they received trimethoprim-sulfamethoxazole.
4. The research literature on the absorption and urinary excretion of antibacterial green tea catechins suggests that the dosage of green tea extract used in this study would achieve a urinary catechins concentration only about 2% as high as that needed to exert an antibacterial effect.^{18 [19]} Therefore, the dramatic benefits reported in this study are of questionable biological plausibility.

The response from journal editors

I have written to many journal editors regarding the types of concerns described in this article. Some editors failed to respond to these correspondences, and others seemed willing to close the case by accepting a flimsy or factually incorrect response from the study authors. When editors do initiate a serious investigation, the process is often painfully slow.

In March 2018, I contacted a group of researchers in New Zealand who had had a good track record of exposing questionable research. I informed them of my concerns about Asemi’s work, and they were able to obtain funding to conduct a thorough investigation. With a small amount of my input, they issued a 115-page report in July 2019, which outlined hundreds of problems in 172 of Asemi’s randomized clinical trials.

This report was sent to the editors of all 65 journals in which Asemi’s papers had been published. These efforts had very little effect for more than a year, but in October 2020 a few journals began to issue an “expression of concern” (a prelude to potential retraction) regarding some of Asemi’s papers. As of May 1, 2022, journals have issued expressions of concern for at least 35 papers and 6 have been retracted.[20] However, nearly 3 years after questions were raised about the integrity of Asemi’s entire body of work, no action has been taken on more than 75% of his papers.

Final remarks

During the past few years, the rate at which “concerning” papers from Iran, Egypt, and elsewhere have been appearing in the medical literature seems to have increased considerably. Journal editors and publishers should insist on more rigorous peer review of submitted papers. It is disconcerting that peer reviewers failed to identify any of the red flags described in this article. In addition, journal editors must overcome their reluctance to investigate research that has raised concerns among their readers.

Assuring the integrity of the medical literature is crucial to the future of the biomedical sciences.

[1] Jamilian M, Asemi Z. The effects of soy isoflavones on metabolic status of patients with polycystic ovary syndrome. J Clin Endocrinol Metab. 2016;101:3386-3394.

[2] Afshar Ebrahimi F, Foroozanfard F, Aghadavod E, Bahmani F, Asemi Z. The effects of magnesium and zinc co-supplementation on biomarkers of inflammation and oxidative stress, and gene expression related to inflammation in polycystic ovary syndrome: a randomized controlled clinical trial. Biol Trace Elem Res. 2018;184:300-307.

[3] Zadeh Modarres S, Heidar Z, Foroozanfard F, Rahmati Z, Aghadavod E, Asemi Z. The effects of selenium supplementation on gene expression related to insulin and lipid in infertile polycystic ovary syndrome women candidate for in vitro fertilization: a randomized, double-blind, placebo-controlled trial. Biol Trace Elem Res. 2018;183:218-225.

[4] Razzaghi R, Pidar F, Momen-Heravi M, Bahmani F, Akbari H, Asemi Z. Magnesium supplementation and the effects on wound healing and metabolic status in patients with diabetic foot ulcer: a randomized, double-blind, placebo-controlled trial. Biol Trace Elem Res. 2018;181:207-215.

[5] Abolhassani J, Mirhosseini N, Bahmani F, Masoud SA, Asemi Z. High-dose omega-3 fatty acid plus vitamin D3 supplementation affects clinical symptoms and metabolic status of patients with multiple sclerosis: a randomized controlled clinical trial. J Nutr. 2018;148:1380-1386.

[6] Maktabi M, Jamilian M, Asemi Z. Magnesium-zinc-calcium-vitamin D co-supplementation improves hormonal profiles, biomarkers of inflammation and oxidative stress in women with polycystic ovary syndrome: a randomized, double-blind, placebo-controlled trial. Biol Trace Elem Res. 2018;182:21-28.

[7] Samimi M, Zarezade Mehrizi M, Foroozanfard F, Akbari H, Jamilian M, Ahmadi S, Asemi Z. The effects of coenzyme Q10 supplementation on glucose metabolism and lipid profiles in women with polycystic ovary syndrome: a randomized, double-blind, placebo-controlled trial. Clin Endocrinol (Oxf). 2017;86:560-566.

[8] Heidarzadeh Z, Samimi M, Seifati SM, Ashkezari MD, Ahmadi S, Mahmoodi S, Aghadavod E, Jamilian M, Asemi Z. The effect of zinc supplementation on expressed levels of peroxisome proliferator-activated receptor gamma and glucose transporter type 1 genes in newborns of women with gestational diabetes mellitus. Biol Trace Elem Res. 2017;175:271-277.

[9] Karamali M, Dadkhah F, Sadrkhanlou M, Jamilian M, Ahmadi S, Tajabadi-Ebrahimi M, Jafari P, Asemi Z. Effects of probiotic supplementation on glycaemic control and lipid profiles in diabetes: A randomized, double-blind, placebo-controlled trial. Diabetes Metab. 2016;42:234-241.

[10] Ahmadi S, Jamilian M, Tajabadi-Ebrahimi M, Jafari P, Asemi Z. The effects of synbiotic supplementation on markers of insulin metabolism and lipid profiles in gestational diabetes: a randomised, double-blind, placebo-controlled trial. Br J Nutr. 2016;116:1394-1401.

[11] Talari HR, Zakizade M, Soleimani A, Bahmani F, Ghaderi A, Mirhosseini N, Eslahi M, Babadi M, Mansournia MA, Asemi Z. Effects of magnesium supplementation on carotid intima-media thickness and metabolic profiles in diabetic haemodialysis patients: a randomised, double-blind, placebo-controlled trial. Br J Nutr. 2019;121:809-817.

[12] Karamali M, Nourgostar S, Zamani A, Vahedpoor Z, Asemi Z. The favourable effects of long-term selenium supplementation on regression of cervical tissues and metabolic profiles of patients with cervical intraepithelial neoplasia: a randomised, double-blind, placebo-controlled trial. Br J Nutr. 2015;114:2039-2045.

[13] Gaby AR. Letter to the editor: Response to “The effects of soy isoflavones on metabolic status of patients.” J Clin Endocrinol Metab. 2017;102:1411.

[14] Jamilian M, Asemi Z. Response to letter by Gaby. J Clin Endocrinol Metab. 2017;102:1412.

[15] Masnadi Shirazi K, Sotoudeh S, Masnadi Shirazi A, Moaddab SY, Nourpanah Z, Nikniaz Z. Effect of N-acetylcysteine on remission maintenance in patients with ulcerative colitis: A randomized, double-blind controlled clinical trial. Clin Res Hepatol Gastroenterol. 2021;45:101532.

[16] Babamohamadi H, Ansari Z, Nobahar M, Mirmohammadkhani M. The effects of peppermint gel on prevention of pressure injury in hospitalized patients with head trauma in neurosurgical ICU: A double-blind randomized controlled trial. Complement Ther Med. 2019;47:102223.

[17] Melli MS, Rashidi MR, Nokhoodchi A, Tagavi, S Farzadi L, Sadaghat K, Tahmasebi Z, Sheshvan MK. A randomized trial of peppermint gel, lanolin ointment, and placebo gel to prevent nipple crack in primiparous breastfeeding women. Med Sci Monit. 2007;13:CR406-411.

[18] Kheirabadi Z, Mehrabani M, Sarafzadeh F, Dabaghzadeh F, Ahmadinia N. Green tea as an adjunctive therapy for treatment of acute uncomplicated cystitis in women: A randomized clinical trial. Complement Ther Clin Pract. 2019;34:13-16.

[19] Reygaert W, Jusufi I. Green tea as an effective antimicrobial for urinary tract infections caused by Escherichia coli. Front Microbiol. 2013;4:162.

[20] Retraction Watch Database. http://retractiondatabase.org/RetractionSearch.aspx?AspxAutoDetectCookieSupport=1#?AspxAutoDetectCookieSupport%3d1%26auth%3dAsemi%252c%2bZatollah. Accessed May 1, 2022.