Dear Editors and Authors,

I am not convinced by the use of the so-called Bayesian Confidence Propagation Neural Network (BCPNN) in this context.

In pharmacovigilance—particularly when evaluating safety signals—the use of a prior hypothesis regarding the safety of a vaccine or drug should be approached with caution. In this case, we lack reliable prior knowledge of the product’s safety profile. Assuming otherwise may be misleading and, potentially, dangerous.

Established disproportionality measures such as the Proportional Reporting Ratio (PRR) or Reporting Odds Ratio (ROR), when accompanied by confidence intervals (CIs), already provide valuable insight. If the CIs are wide, this simply reflects uncertainty—and that, in itself, is informative enough.

The main result of the paper appears to be the PRR of approximately 23 (with a lower bound exceeding 9) for preterm birth following the RSV vaccine. This is striking, yet it is not highlighted in the conclusions; one has to look in the appendix (link "supplemental material"), specifically Table S6 p.10, to find it.

Why is there such a significant discrepancy between the PRR (~23) and the Information Component (IC, ~2+)? Even at the lower bound, the PRR remains notably elevated. This likely stems from an inappropriate prior used in the Bayesian model. In fact, the paper serves as a good illustration of why Bayesian methods, particularly in the form of BCPNN, may not be suitable for pharmacovigilance signal detection.

Sincerely,
Vincent Schmitt

Responses to comments from a reviewer on a article published in BMJ Open in 2021

(https://www.pubpeer.com/publications/ABB5D808AAF436219EBFB9896A0D05)

Reader’s comment nr 1: Sample Size Calculation
The study reports: “For the estimation of HPV prevalence, a sample size of 267 participants was needed with an anticipated prevalence of HPV of 50%, 6% precision, and a 5% level of significance.”
• While 6% precision is technically possible, this departs from the standard 5% cutoff.
• Since the study employed snowball sampling —a convenience sampling method from a single center—to estimate nationwide HPV prevalence, it failed to consider the design effect and non-response rates inherent in such non-random sampling methods. After accounting for these factors, the required sample size should be approximately 446 participants for 6% precision and 642 participants for 5% precision. Not 267.
Variance Estimation, Design Effects, and Sample Size Calculations for Respondent-Driven Sampling Studies. American Journal of Epidemiology, 2006; 163(5): 471–478.
Snowball Sampling: A Review of Key Issues and Methodological Considerations. International Journal of Social Research Methodology, 2013; 16(4): 351-367. (doi:10.1080/13645579.2013.801561)

RESPONSE to comment nr 1:
Thank you so much for raising this concern.
Design effect (DE) is used for complex sampling techniques like stratified random sampling or cluster random sampling/multi-stage cluster sampling where the DE considerations are required because of additional variance due to stratification and or clustering. We went for non-random sampling, the population was homogenous, and the variation is therefore only based on the subject of interest. Thus the design effect was kept as 1 and hence got the given sample size (n=267).

Reader’s comment nr 2: Methodological Errors and Biased Estimates
The study states: "We employed a Cox proportional hazard model algorithm to estimate crude and adjusted PRs and their 95% CIs in univariate and multivariate models."
Could the authors share the actual SPSS working for fitting the Cox Proportional Hazard Model to their cross-sectional data? Would also greatly appreciate it if you could justify your analytical approach?
• Fitting the Cox proportional model on cross-sectional data is fundamentally flawed, as it treats a single time point as if it represents survival time. This misapplication violates the Cox model's assumptions, which are designed for time-to-event data with variability, not cross-sectional data. Without censoring, this model is over-parameterized and results in systematic bias. It tends to inflate study estimates, which is a common issue in longitudinal studies with high censoring rates.
• Thus, the ratios and p-values produced by SPSS stem from a misapplication of the Cox Proportional model. The software might issue warnings about insufficient variation in the time variable but will still attempt to fit the model.
• For binary outcomes (e.g., event occurrence: yes/no) in cross-sectional datasets, logistic regression should be used instead. This approach models the probability of an event as a function of covariates, which is appropriate for the study design.

RESPONSE to comment nr 2:

• Since this is a cross-sectional study with a binary outcome, we applied a Cox proportional hazard model algorithm using robust standard error (SE), which we believe is an appropriate model building technique and estimated the association between the exposure and the outcome using means of prevalence ratios (PRs).

• When adjustments for potential confounders are needed, logistic regression models are commonly used to estimate odds ratios (ORs) that are reported in a similar way as the PR. However, OR are less suitable when the outcome is very common like in our study. In these situations, interpreting ORs as if they were PRs may be inadequate.
• (ref: Barros AJ, Hirakata VN. Alternatives for logistic regression in cross-sectional studies: an empirical comparison of models that directly estimate the prevalence ratio. BMC Med Res Methodol. 2003;3:21. DOI: 10.1186/1471-2288-3-21)

• For example, if 80 out of 100 exposed subjects have a particular disease and 50 out of 100 non-exposed subjects have the disease, then the odds ratio (OR) is (80/20)/(50/50)=4. However, the prevalence ratio (PR) is (80/100)/(50/ 100)=1.6. The latter indicates that the exposed subjects are only 1.6 times as likely to have the disease as the non-exposed subjects, and this is the number in which most people would be interested.
• There is considerable literature on the advantages and disadvantages of OR versus PR that we believe supports our choice of using the PR. (Greenland, Stromberg, Axelson et al and others) , , , , .

• 1. Greenland S. Interpretation and choice of effect measures in epidemiologic studies. Am J Epidemiol 1987;125:761–8.
• 2. Stromberg U. Prevalence odds ratio v prevalence ratio. Occup Environ Med 1994;51:143–4.
• 3. Axelson O, Fredricksson M, Ekberg K. Use of the prevalence ratio v the prevalence odds ratio as a measure of risk in cross sectional studies. Occup Environ Med 1994;51:574.

• Thus, since the outcome of interest (HPV infection) was so common >10%, we preferred to use PRs and hope that the reader understands this and agrees.

• Moreover, Barros et al have reported that Cox or Poisson regression with robust variance and log-binomial regression provide correct estimates and are a better alternative for the analysis of cross-sectional studies with binary outcomes than logistic regression, since the prevalence ratio is more interpretable and easier to communicate to non-specialists than the odds ratio (Barros AJ, Hirakata VN. Alternatives for logistic regression in cross-sectional studies: an empirical comparison of models that directly estimate the prevalence ratio. BMC Med Res Methodol. 2003;3:21. DOI: 10.1186/1471-2288-3-21)
In fact, the study that the reader has referenced under bullet point 10, Wiley DJ et al (2013) published in PLOS ONE, also estimated adjusted prevalence ratios from multivariate Poisson regression analysis.
Reader comment nr 3. Unexplained Inclusion criteria
The study states:
“A total of 265 study participants with different sexual orientations, that is, MSM (homosexuals, bisexuals) and transgender women living with and without HIV were recruited. Participants were eligible for the study if they self-reported having had anal sex in the preceding at least 6 months and were age 18 and above.”
• The use of a 6-month cutoff for recent anal sex introduces significant methodological flaws, particularly when assessing HPV prevalence nationally.
• Anal HPV infections, particularly high-risk types like HPV-16, are well-documented to persist far beyond six months, even in the absence of recent sexual exposure. Limiting inclusion criteria to MSM who have engaged in anal sex within the past six months is unjustifiably restrictive and excludes individuals with persistent infections. This narrow approach biases the sample toward more sexually active individuals (by design predominantly male sex workers), inflating prevalence estimates.
Hernandez et al. (2014) and Lin et al. (2020).
• No explanation was provided for the age cutoff at 18 years. Excluding participants under 18 overlooks the reality that many MSMs initiate sexual activity at younger ages, particularly those in marginalized groups who may face early sexual abuse or exploitation, as acknowledged later in the study. This exclusion introduces further bias by failing to account for the experiences of high-risk youth.

RESPONSE to comment nr 3:
This maybe an interesting idea for another study, but we set out to study young adults (defined as 18 years and above by the WHO’s criteria) and not adolescents or minors who would require parental or garudian consent, something that ew feel would be unethical and could have jeopardized the safety of participating adolescents given that this study was conducted in a very conservative country. We focused on recruiting participants aged 18 and above in our research project also because adults are generally considered to be legally and cognitively more capable of providing informed consent, making them suitable for participation in a study that involved sensitive questions and it aws important to us that decision and consent to participate was truly informed and that they were ready to self-report sensitive sexual behaviors
Thus, this study was conducted among young adults who self-reported to have had anal sex in the last preceding at least 6 months that indeed went beyond also. But for careful readers, it is clearly stated in the paper that the mean duration of active sexual behavior of our study population was 12 years with a standard deviation of 8.2.

Reader’s Comment nr 4. Vague study procedure section
There is only one, just one sentence in the whole section about the study's own procedure, and that too is very generic!!
The following is the complete section dedicated to the "Study Procedure":
“A 35–40 min long structured questionnaire was administered at a private space to interview the study participants in order to collect data concerning sociodemographic information, sexual and reproductive history and medical history relevant to HIV, any anal disease and use of ART. A blood sample was obtained for confirmation of their HIV status (through Architect HIV Ag-Ab Combo kit by ABBOTT), viral load (real time PCR Artus Qiagen) and CD4 +T cell count (through FACS Count TM flow cytometry, Becton Dickinson, Franklin Lakes, New Jersey, USA). Two anal swabs for HPV testing were collected by a trained general physician. A water-moistened swab was inserted 5–6 cm proximal to the anal verge and the samples were obtained from anorectal transition zone around the dentate line in the anal canal. The swab was then agitated vigorously in a tube containing 3 mL of methanol- based fixative—a Sample Transport Medium (UTM-RT viral transport media with flocked polyester swabs, Copan Diagnostics, Corona, California, USA) and transported to the lab. In Molecular lab the solution was stored at −70°C before further processing and was later used for the extraction and detection of HPV DNA by the PCR. The DNA from rectal swab was extracted by using Qiagen DNA mini kit and DNA was eluted in 80 uL of elution buffer and stored at −80°C for further use. HPV DNA was detected by using MY09/MY11 primers from L1 region resulted in amplification of 450 bp of L1 gene. PCR for Housekeeping gene glyceraldehyde 3-phosphate dehydrogenase (GAPDH) was done for every extracted DNA and samples tested negative for GAPDH gene were excluded from study. Nested multiplex PCR assay was used26 for detection of HPV subtypes included LR types 6/11 and HR (group 1) types 16, 18, 31, 33, 35, 52, 56, 58 and 59 as described by Sotlar et al. 27 Primers for first-round PCR for GP-E6/E7 consensus sequence and second round PCR for HPV subtypes 16, 18, 6/11 31, 33, 35, 52, 56, 58 and 59 were synthesised from Euro- fins MWG/Operon Germany. The HPV subtype specific primers were used in four cocktails and size of the nested PCR product was used for the identification of each HPV type by gel electrophoresis.”
• The study fails to provide sufficient detail on aspects such as covariate selection (including what covariates were selected, why, their potential implications, and their confounding roles), the framework employed, and the reliability and validity of the tools used to measure exposure, outcomes, and covariates.
• Instead, strangely enough, these elements are replaced by irrelevant details on routine well-established laboratory procedures. The procedural descriptions of HIV and HPV testing could have been succinctly referenced, preserving valuable space that should have been dedicated to clarifying the study's own design and observational data procedures.
RESPONSE to comment nr 4:
We do not think this critique is relevant or warranted since we have adequately and thoroughly described the study procedures including a description of study design, study population, setting, and included variables. Since this study primarily was a prevalence study, it was necessary to elaborate carefully on the sample collection, processing, and testing protocols, as this significantly affect HPV test results, HPV prevalence data reliability and generalizability. Moreover, our intention was that the comprehensive information provided would guide future policy decisions regarding HPV screening programs.
Regarding which variables that were included in the analysis, this is also described in the result section that all reported associations were carefully analyzed for any confounding by all relevant variables such as, age, sexual and behavioral factors.

Reader’s comment nr 5. Discrepancies in Reporting
5.1. The title of Table 1 refers to the "characteristics of the study population in (city of) Karachi, Pakistan." However, the study title and aims mention the country rather than the city, suggesting that it is a nationwide study when it is not. In fact, the study doesn't even adequately represent Karachi due to its weak sampling methods, small sample size, lack of representativeness, and the use of a single center. More so, study participants were not recruited through a multistage probability random sampling approach, further undermining the study’s claims of broad relevance.
I don’t understand why the study title says "Pakistan" instead of "Karachi, Pakistan," which would better reflect the study’s limited scope. This could help avoid giving the misleading impression that the study covers the whole country when it actually only involves one center in the city of Karachi.
5.2. Table 1 shows descriptive data on the number of cigarettes smoked and the number of anal encounters. However, these factors were not included in the adjusted analyses.
Could the authors clarify why these variables were excluded? Additionally, what was the rationale behind selecting the covariates included in the multivariate models?
RESPONSE to comment nr 5.1:
We clearly describe the study setting under the methods section and have already acknowledged and justified this in our discussion section. Of course, it is a limitation that this study was not conducted nationwide, it would have been great to have the resources to conduct a much bigger study. However, as most researchers are well aware funding amnd resources constraints always influence how much research and data collection can be done. This does not make the results less valid and precise.

RESPONSE to comment nr 5.2:
The above-mentioned variables were analyzed and did not turn out to be significant in the final model. It is common practice in all statistical analyses to keep only the statistically significant variables in the final model
Reader’s comment nr 6. Discussion section
Table-3 shows statistically significant Receptive Anal Sex and HPV Risk. While there is a noticeable association between receptive anal sex and HPV16 infection, this association is weakened after adjusting for HIV status, suggesting that HIV and its associated immunosuppression might be the stronger contributing factor to HPV infection risk in this group. Yet, the effect-moderating role of ART therapy, which offers a more nuanced understanding of HPV transmission, where immunosuppression related to HIV plays a crucial role in HPV acquisition and progression, rather than sexual activity alone, was not discussed.
We know that the relationship between HPV prevalence and MSM is not as straightforward as discussed by this study. Actually, the study's findings also suggest that HIV-positive status might be a more significant risk factor for HPV16 and overall HPV infection.
Therefore, the study hypothesis that MSM population is primarily at risk of HPV-related anal lesions/cancer due to sexual behavior (e.g., anal sex role) alone does not fully capture the complexity of the issue.
RESPONSE to comment nr 6:
We never hypothesized that HPV prevalence in MSM is related to sexual practices alone. Although, previous research indicate that receptive anal intercourse is a risk factor for anal lesions, anal HPV infection is not limited to men who have receptive anal sex, it can also be acquired during non-receptive sexual activity.
Moreover, it is true that immunosuppression due to HIV plays a role in the acquisition and progression of HPV infection but exploring the effect of HIV related immunosuppression as well as the role of ART was beyond the scope of our study.

Reader’s comment nr7. Misleading statements (with no or improper citation) can be found throughout the text, e.g., the study in their intro section states:
“Currently, HPV vaccines, in which the bivalent vaccine protects against HPV16/18 and quadrivalent vaccine protects against HPV16/18 and the LR types HPV6/11,(25) are available in pharmacies in Pakistan but are not yet made available to the target population.”
Here, the authors potentially imply that the government is not facilitating its widespread use or perhaps deliberately limiting access for high-risk groups like MSWs or MSMs. [Moderator: your (over)interpretation would appear to turn on the single word "made".] However, this claim is made without sufficient clarification or citations to substantiate the assertion.
If the vaccine is indeed available in pharmacies, what mechanisms—whether policy-driven, financial, or social—are contributing to its inaccessibility for the target population? Is the limitation due to high costs, societal stigma, or a lack of targeted public health programs? Or is there evidence of intentional policy decisions to restrict access for these marginalized groups?
The lack of citation for this critical point weakens the argument and leaves significant gaps in understanding the barriers to HPV vaccination in this marginalized group. Here, the authors missed an opportunity to explore solutions, such as advocating for government-subsidized vaccination programs, awareness campaigns, or addressing systemic barriers.
(Reference cited here (25) is irrelevant: [25. Schiller JT, Castellsagué X, Garland SM. A review of clinical trials of human papillomavirus prophylactic vaccines. Vaccine 2012;30:F123–38])

REPONSE to comment nr 7:
The reader makes sweeping comments about our underlying messages in the text. He or she mentions only one specific point though, and that is an over interpretation of a single word “made” in one sentence.
It is a fact that HPV vaccine is not available to target the key population under study and beyond that we did not imply anything. Our study is not about reasons for non-availability and its solutions. This is solely the reader’s own interpretation.

Reader’s comment nr 8. Study Limitations Section
The authors provided a standard, generic list of limitations instead of a more detailed analysis that would demonstrate their expertise in both epidemiologic research methodology and the subject matter.
Most of the statements are misleading and factually incorrect. Below are a few points of concern:
8.1. The study states: “Karachi is the largest metropolitan city of Pakistan, representing all ethnicities of the country. Moreover, in a country like Pakistan, which is by and large a heteronormative society and where homophobia prevails against this sexual minority due to cultural and religious inhibitions, using above-mentioned sampling technique was the most feasible option for the recruit- ment process.”
This statement is highly problematic and fails to address critical flaws in its sampling approach and provides a misleading portrayal of Karachi’s representativeness. While Karachi is diverse, it is also the most secular city in Pakistan, and societal views around MSM and gender minorities in this metropolis may differ significantly from those in more conservative, less secular regions, challenging its suitability as a **proxy **for national representation.
The claim that snowball sampling was the most feasible technique is unfounded. Karachi, with over 27 million residents and numerous opportunities to engage directly with these populations, offers far better avenues for more robust sampling. The reliance on one center and an insufficient sample size severely undermines the study’s validity and its broader policy implications, as the findings lack depth and fail to reflect the heterogeneity of experiences across the country.

RESPONSE to comment nr 8.1:
The reader has again tried to argue that our statements are misleading and incorrect, but the first author is from Pakistan and has worked in Karachi for over 20 years with the target group of interest and is very familiar with this setting.
It is a well-known fact that Karachi being the largest metropolitan city of Pakistan represents all ethnicities in the country. It may be more secular than other cities, but this is not a study on societal views or religion. We fail to understand as why the reader thinks that the population of Karachi is not representative of the MSM and transgender people in this type of study
Despite its large population, there are very limited opportunities to engage directly with this marginalized community in the city. Given their marginalized and hidden nature, it would be difficult to identify willing participants in smaller towns, without jeopardizing their safety and confidentiality. Our study is also meant to be explorative and not intended to have broad policy implications. Now that we have reported a hidden problem, we hope that further large scale nationwide studies will be conducted to guide future policies.

Reader’s comment nr 8.2. The study states: “Moreover, the cross-sectional design precludes us to determine a temporal relationship, however, the risk factors identified in our study appear robust and have also been found in studies from other countries “
The authors again provide a generic critique of cross-sectional study designs. A well-executed cross-sectional study, with clear objectives, can generate valuable, meaningful insights comparable to longitudinal studies, particularly in terms of exploratory and prevalence data.
Response to comment nr 8.2: The reader has commented on our own comment about study limitation. We did not intend to undermine cross sectional study designs, but rather to point out the inability to draw temporal association from that sort of design. We entirely agree with the reader about the usefulness of cross-sectional studies.

Reader’s comment nr 9. Misreporting findings from published literature, e.g. the study states:
“The reported incidence of anal cancer is 1–2/100 000 in the general population,12 while in MSM it has been reported up to 35 cases per 100 00013 and even up to 131/100 000 in MSM living with HIV (14,15) regardless if they are on antiretroviral treatment or not.”
((ref 14: Silverberg MJ, et al. Risk of anal cancer in HIV-infected and HIV-uninfected individuals in North America. Clin Infect Dis. 2012 Apr;54(7):1026-34. doi: 10.1093/cid/cir1012. Epub 2012 Jan 30. PMID: 22291097; PMCID: PMC3297645.)
• The statement that the incidence rate of 131 per 100,000 among MSM with HIV is independent of ART use is not factual and reflects a lack of understanding of the data referenced. This figure is clearly the crude incidence rate reported in the reference 14 for the years 2004–2007, not an ART-independent value.
• In contrast, **the adjusted relative risk (RR) for MSM with HIV during the same period was 78.8 (40.8–152.1), ** accounting for confounding factors such as ART and immune status.
• It underscores the problematic practice of cherry-picking and misinterpreting numbers from other studies to fit preconceived narratives, which undermines the credibility and validity of scientific research.
RESPONSE to comment nr 9:
The incidence of anal cancer in MSM living with HIV, is from a clearly referenced article. It was not our intention to be selective, nor did we have any preconceived narratives. The reader is free to have a different perceived interpretation of a reference stated in our article, but this does not undermines the credibility of our scientific research.

Reader’s comment nr 10. Study Replication
Below are 4 high quality prospective cohort studies for the purpose of highlighting the points discussed in this section.
1. Hernandez, A. L., Efird, J. T., Holly, E. A., Berry, J. M., Jay, N., & Palefsky, J. M. (2014). Incidence of and risk factors for type-specific anal human papillomavirus infection among HIV-positive MSM. AIDS (London, England), 28(9), 1341–1349. https://doi-org.ezproxy.u-pec.fr/10.1097/QAD.0000000000000254 Similar, with correct methodology:
2. Wiley DJ, Li X, Hsu H, Seaberg EC, Cranston RD, et al. (2013) Factors Affecting the Prevalence of Strongly and Weakly Carcinogenic and Lower-Risk Human Papillomaviruses in Anal Specimens in a Cohort of Men Who Have Sex with Men (MSM). PLOS ONE 8(11): e79492. https://doi-org.ezproxy.u-pec.fr/10.1371/journal.pone.0079492
3. Anal human papillomavirus infection and associated neoplastic lesions in men who have sex with men: a systematic review and meta-analysis Machalek, Dorothy A et al. The Lancet Oncology, Volume 13, Issue 5, 487 – 500 2012
4. Silverberg MJ, et al. Risk of anal cancer in HIV-infected and HIV-uninfected individuals in North America. Clin Infect Dis. 2012 Apr;54(7):1026-34. doi: 10.1093/cid/cir1012. Epub 2012 Jan 30. PMID: 22291097; PMCID: PMC3297645.
10.1. Significant differences exist in the methodological rigor between the current study and Studies 1-4 above. The current study is essentially a replica of the well-established studies but differs in its flawed methodology, smaller sample size, restrictive and non-representative single-center sampling approach, and cross-sectional design. For instance, Study 4 involved 13 cohorts with a total of 34,189 HIV-infected and 114,260 HIV-uninfected individuals, using robust longitudinal methods and Poisson regression, yielding valid and generalizable results. Similarly, Study 3, which pooled data from 53 studies, found that while anal HPV and anal cancer precursors are common in MSM, progression to cancer appears significantly lower than for cervical pre-cancerous lesions.
As a result, the current study fails to critically appraise high-quality literature, missing the opportunity to address existing knowledge and methodological gaps that it could potentially fill.
Could the authors provide adequate and concrete evidence of any unique contribution their study brings to the existing literature, particularly regarding unique study variables or covariates that were not previously assessed?
RESPONSE to comment nr 10:
Again, our study was the first of its kind to be conducted in an extremely conservative society where medical problems by this marginalized population are rarely considered or mentioned let alone research conducted on such a sensitive issue. We thought that with limited resources, and uncertainty of follow-up visits, a small-scale cross-sectional study could provide valuable initial information and be of interest to readers (which it obviously really is!).
We, however, never intended to imitate or excel above-mentioned cohort studies. The assertion by the reader that we should have filled the knowledge gap on a larger scale and explored new variables is unjustified, considering the scope of our study.

Following the publication of the original article, it has come to the authors’ attention that the timing of the analysis was still based on the wording of the original funding application, and had not been updated prior to publication of the trial protocol paper.

The original funding application where both the short- and long-term outcomes were deemed as co-primary outcomes and as such would have been analysed at study end. At point of funding by the NIHR, we were requested to consider only the short-term outcome to be the primary outcome, and as such the timing of the analysis should have been changed so that short term outcomes were analysed first and longer-term outcomes after 2 years post-partum. The analysis plan was adjusted at that time, according to the NIHRs request.

The article currently states (under the Main analysis section) that: “All analyses will be undertaken after database lock following data collection at 2 years.”

However, the wording in this section should read: “Analysis of the short-term outcomes will be carried out after database lock following data collection at birth. The longer-term outcomes will be analysed after database lock following data collection at 2 years post-partum.”

To ensure that knowledge of the short-term outcomes will not impact the scientific integrity of the longer-term outcomes, we will continue to adhere to strict retention protocols to follow up the mothers at 2 years for the longer-term health and development outcomes of the child. In addition, we believe that there is no risk of the long-term outcomes being affected by the knowledge of the short-term outcomes as data collection is via a survey completed by the mothers on the health and development of the child.

This clarification has been approved by the Trial Steering Committee.

This rapid response correction is submitted whilst recruitment to the trial is still ongoing.

Yours sincerely,

Dr Rebecca Cannings-John, on behalf of the TRUFFLE 2 Trial Management Group

Comments related to the findings reported by Warrington and Holm regarding the use of Artificial Intelligence (AI) by UK General Medical Council (GMC) registered doctors (Warrington DJ, Holm S. BMJ Open 2024;14:e089090. doi:10.1136/bmjopen-2024-089090).
A key observation regarding the study is its apparent lack of distinction between participants' use of regulated AI products (classified as medical devices) and non-regulated AI tools (such as general-purpose LLMs). The wide range of respondent specialties reported further highlights this potential issue; for instance, clinicians in radiology or pathology are more likely to encounter regulated, task-specific AI, whereas those in public health or psychiatry might be more likely to experiment with non-regulated, general-purpose models.
During the review process, I used Gemini Advanced (specifically, the model designated by the user as 2.5 Pro Experimental) to assist with processing screenshots of table1, table 2 and fig 1 into spreadsheet processable data. The same Large Language Model (LLM) was also prompted to categorize the clinical risks associated with the AI uses listed in Figure 1 of the original paper. The author is of the opinion that the LLM’s risk categorisation (column 2 in table A below) adopted a patient-centric perspective.
However, the author is of the opinion that a "composite" clinical risk assessment, which considers both the nature of the specific usage instance and the potential scale of patient impact arising from a flawed AI output (e.g., contrasting the impact of an error in analyzing an individual patient's test results with that of disseminating flawed guidance derived from an AI-assisted literature review), is more appropriate. Author assumed most probable categorisation of software and composite risk assessments (low, medium and high scale) are recorded in columns 3 and 4 of table A

Table A

Use Case (ref Figure 1,Warrington DJ, Rank (gemini Author assumed Author's
Holm S. BMJ Open 2024;14:e089090. clinical risk most probable composite risk
doi:10.1136/bmjopen-2024-089090) prompt - output) categorisation of assessment
software (low, medium
and high
scale)
search the scientific literature 1 Non-regulated Low
stay up to date with my medical knowledge 2 Non-regulated Low
write educational material 3 Non-regulated High
write research papers or essays 4 Non-regulated High
write reflective pieces for my portfolio 5 Non-regulated Low
automate administerial tasks 6 Non-regulated Medium
perform data analytics 7 Non-regulated Medium
write patient letters or notes 8 Non-regulated Medium
interpret pathology slides 9 Regulated Low
interpret scans or results 10 Regulated Low
make diagnoses 11 Non-regulated Medium
make treatment recommendations 12 Non-regulated Medium
plan or perform radiotherapy 13 Regulated Low
plan or perform surgical operations 14 Regulated Low
Other 15 Non-regulated Medium

Furthermore, regarding the survey questions exploring AI's role in decision-making (specifically referencing the implications of question 5d in Table 2, concerning autonomous decisions), this author is of the view that such questions may implicitly understate the clinician's ultimate responsibility. Under the current UK regulatory system, it is my understanding that the clinician, not the AI device, remains accountable for the clinical decision. Consequently, the onus is on the clinician to critically evaluate and, if necessary, seek "a second opinion" on the AI's output before integrating it into their decision-making. The patient retains the right to request a second opinion on the clinician's final clinical judgment.
Finally, as a point of good practice for maintaining professional accountability and audit trails, healthcare registrants using LLMs for clinical decision support might consider exporting AI outputs to secure documents (e.g., Google Docs or similar platforms), particularly when using tools lacking integrated tracking features.