Artificial intelligence-based chatbots: the importance of asking the right questions

Abstract

Background: Artificial intelligence-based chatbots (CBIA) are a widely used source of medical information. Prompt engineering (PE) focuses on designing and optimizing the questions asked of AI/ CBs to improve responses.
Objective: The aim of this study was to compare the quality of prompts and the responses provided by an AiCB for clinical case resolution in general surgery, before and following PE training.
Materials and methods: Three fictional clinical cases were developed for residents in general surgery to solve using ChatGPT-4®. After they were trained in PE, the participants solved the cases again. The quality of the prompts was evaluated using a scale (5-15 points) that explored completeness, context, input data, output format, and instructions. The chatbot’s answers were assessed using a scale (3- 15 points) that included accuracy, completeness, and relevance. The results obtained before and following PE training were compared.
Results: Sixteen postgraduate year 1 to 4 residents participated in the study. The quality of prompts improved significantly following PE training, as assessed by total score [7.9 (1.8) vs. 10.4 (2.1), p< 0.01] for completeness, context, data input, and output format categories. Chatbot’s responses also improved across the categories and total score [10.2 (2) vs. 11.9 (1.8), p<0.01].
Conclusion: Training in PE significantly improved the quality of prompts and AiCB’s responses for solving general surgery clinical cases.