1
0
Peña ME y cols. Chatbots basados en inteligencia arꢁꢂcial: la importancia de saber preguntar. Rev Argent Cir. 2026;118(2):e-1961
■
TABLE 2
Descripꢀon of the variables related to response quality
Variable
Deꢁniꢀon
Assessment
Accuracy of the informaꢀon provided in the
Likert scale Score from 1 to 5 (1: Completely incorrect, 2: Mostly incorrect,
Accuracy
response
3: Equally correct and incorrect, 4: Mostly correct, 5: Completely correct).
Presence of all necessary informaꢀon within the
response to address the clinical case.
Likert scale. Score from 1 to 5 (1: Highly deꢁcient, 2: Incomplete, 3:
Parꢀally complete, 4: Substanꢀally complete, 5: Fully comprehensive
Completeness
Likert scale Score from 1 to 5 (1: Completely irrelevant, 2: Mostly
irrelevant, 3: Parꢀally relevant, 4: Mostly relevant, 5: Completely
relevant).
Relevance of the informaꢀon provided by the
response to address the clinical case
Relevance
Sum of the scores obtained for accuracy,
completeness, and relevance.
Total response score
Score from 3 to 15
Likert scales, and the total score — ranging from 3 to regression using the Brant test. A p-value < 0.05 was
5 points — was calculated by adding up the individual considered staꢀsꢀcally signiꢁcant. All the staꢀsꢀcal
scores. For both scales, a higher score indicates greater calculaꢀons were performed using SPSS v.25® and
prompt or response quality. RStudio® soꢂware packages.
The prompts and responses were assessed by
two evaluators (AG and FI), who are general surgeons.
1
Both surgeons had received training in prompt Results
engineering using the same method as that employed
with residents. They were administered the same set of
Of the 18 residents in the department of
prompts and encoded responses, with no informaꢀon general surgery, 16 parꢀcipated in the study. Seventy-
about the parꢀcipants or the intervenꢀon provided. ꢁve percent had used an AiCB before, with 63% having
They were asked to assess the prompts and responses used it in their professional acꢀviꢀes. None of the
using the menꢀoned scales. The scores calculated by residents had any prior knowledge of PE (Table 3). All
both evaluators were averaged to obtain a ꢁnal score.
residents solved the three clinical cases before and
For staꢀsꢀcal analysis, quanꢀtaꢀve variables following PE training, providing a total of 96 prompts
were reported as mean and standard deviaꢀon to and 96 responses.
facilitate their interpretaꢀon. Qualitaꢀve variables
Prompts’ quality improved signiꢁcantly
were expressed as absolute and relaꢀve frequencies. following PE training [total score before training: 7.9
The Wilcoxon test for dependent samples was used to (1.8); total score following training: 10.4 (2.1); p <
compare prompts and responses before and following 0.01]. This improvement was noted in completeness,
the training intervenꢀon.
context, input data and output format categories
Inter-rater agreement was assessed using (Table 4). In clinical case 1, a signiꢁcant increase was
Cohen’s weighted kappa coefficient, given the ordinal observed in the total score, completeness, context,
nature of the scales used. The total scores obtained and output format categories (Appendix 2, Table A1).
from prompts and responses, as well as the speciꢁc In clinical case 2, a signiꢁcant increase was observed
dimensions, were analyzed both before and following in the total score, completeness and context categories
the intervenꢀon. The magnitude of the agreement (Appendix 2, Table A3). In clinical case 3, the total score,
was interpreted according to the criteria of Landis and completeness, context, and input data categories
1
2
Koch .
increased signiꢁcantly (Appendix 2, Table A5).
The quality of responses improved signiꢁcantly
The associaꢀon between prior experience
of AiCB (for both general and medical purposes) followingPEtraining[totalscorebeforetraining:10.2(2);
and year in the residency program with the quality total score following training: 11.9 (1.8); p < 0.01]. This
of prompts and responses generated by ChatGPT improvement was noted in the accuracy, completeness,
was evaluated using ordinal mulꢀvariate regression. and relevance categories (Table 5). In clinical cases
The dependent variables were the total prompt and 1 and 2, the total score, accuracy, completeness, and
response quality scores, assessed before and following relevance categories increased signiꢁcantly (Appendix
prompt engineering training. The year of residency 2, Tables A2 and A4). In clinical case 3, there was a
was modeled as an ordinal variable, and its effect signiꢁcant increase in the total score, accuracy, and
was assessed as a linear trend. Odds raꢀos (OR) were completeness categories (Appendix 3, Table A6).
esꢀmated, along with their corresponding conꢁdence
The analysis of inter-rater agreement revealed
intervals and p-values. The model assumpꢀons were that agreement was low before training (κ = 0.04) and
veriꢁed, including the absence of mulꢀcollinearity acceptable following training for the total prompt score
among independent variables, using the variance (κ = 0.25). When analyzing the speciꢁc dimensions,
inflaꢀon factor (VIF), and the assumpꢀon of parallel moderate and substanꢀal levels of agreement were