Evaluating Accuracy of ChatGPT-4o in Automated Carbohydrate Estimation From Images as a Self-Management Tool for Adolescents With Type 1 Diabetes

Abstract

Introduction:

Type 1 diabetes mellitus (T1D) requires precise carbohydrate estimation to manage blood glucose and prevent chronic and acute complications to hyperglycemia or hypoglycemia. This study evaluates the accuracy of ChatGPT in estimating carbohydrate content in images of meals, compared with the considered gold standard of manually counting carbohydrates.

Method:

Carbohydrate content of 60 fruits and vegetables (F&V) and 60 composite meals was manually counted as the reference standard. Images (n = 240), with and without a size reference, were uploaded to ChatGPT-4o with a standardized prompt in separate sessions. ChatGPT’s estimates were then compared with the manual counts to assess accuracy.

Results:

The performance of ChatGPT-4o compared with the manual calculation was assessed primarily using mean absolute error, percentage of agreement (PoA), and Bland-Altman analysis. ChatGPT-4o achieved a PoA of 93.3% for F&V’s estimates, increasing to 95% with a size reference, while composite meal estimates yielded a PoA of 46.7%, reducing to 43.3% with a size reference, based on a ±10 g carbohydrates limit. Bland-Altman analysis showed a slight bias tendency in both ChatGPT-4o’s estimates of F&V and composite meals with a size reference. ChatGPT-4o’s estimate for F&V and composite meals without a size reference exhibited a systematic bias, with both overestimation and underestimation of the carbohydrate content.

Conclusion:

This study suggests that adolescents living with T1D should employ ChatGPT-4o for carbohydrate estimating with caution. ChatGPT-4o showed inaccuracies in its application to composite meals, increasing the risk of inaccurate insulin administration and potentially causing postprandial hyperglycemia or hypoglycemia.