A linguistic lens into vision-language models for open-ended question-answers in medical visual question answering

Abstract

Objectives

Medical Visual Question Answering (MedVQA) systems are predominantly evaluated using exact-match accuracy, which fails to account for partially correct or clinically insightful answers, particularly in open-ended question settings. This study aims to investigate the effectiveness of language-based evaluation metrics, specifically BLEU scores at multiple n-gram levels, in providing a more nuanced perspective of model performance beyond strict accuracy.

Methods

We conducted a systematic benchmarking of multimodal vision–language models by combining three transformer-based visual encoders (ViT, BEiT, and DeiT) with three transformer-based text encoders (BERT, RoBERTa, and ALBERT). All model variants were fine-tuned and evaluated on the VQA-Med 2020 and VQA-Med 2021 datasets. Performance was assessed using exact-match accuracy alongside BLEU-1 to BLEU-4 and cumulative BLEU scores to capture varying degrees of lexical and semantic overlap between predicted and reference answers.

Results

Experimental results demonstrate that DeiT-based visual backbones consistently outperform ViT and BEiT across both datasets. On VQA-Med 2020, the DeiT + BERT model achieved the highest accuracy of 52.27% with a cumulative BLEU score of 26.87. On VQA-Med 2021, DeiT + RoBERTa reached the highest accuracy of 54.87%, while DeiT + ALBERT achieved the highest cumulative BLEU score of 25.50. Qualitative analyses further show that BLEU scores effectively capture partial correctness, synonymous terminology, and clinically relevant overlaps that are ignored by accuracy-based evaluation.

Conclusion

This study demonstrates that language-based evaluation metrics, particularly BLEU scores, provide important complementary insights beyond accuracy in open-ended MedVQA tasks. By capturing semantic similarity and partial correctness, such language-based metrics support a more clinically meaningful assessment of model outputs. These findings emphasize the need for multi-metric evaluation frameworks, as exact-match accuracy alone is insufficient for open-ended responses.

1. Introduction

Medical Visual Question Answering, also known as MedVQA, is a specialized type of Visual Question Answering (VQA) that involves integrating image and language information and interpreting medical-related visual concepts. In other words, MedVQA systems take medical images and clinical questions as inputs and generate answers as responses. Such systems can assist doctors in getting a second opinion on a diagnosis, ¹ potentially reducing the costs associated with further medical appointments and consultations with other professionals. This process can also help to alleviate the burden on medical facilities, reserving valuable resources for the most critical patients. Moreover, patients can benefit from MedVQA by receiving immediate responses to their questions and making better decisions.

A challenging aspect of MedVQA, in contrast to VQA in general, is questions in this domain are more complex and require high precision since they are related to health and safety. In general, there are two types of questions in MedVQA; one is closed-ended questions and the other is open-ended questions. Closed-ended questions involve a Yes/No answer or common perceptual abilities related to the image. On the other hand, open-ended questions are more challenging to solve and provide accurate answers. Questions, such as, “What is under the right hemidiaphragm?” or “Where are the kidney in this image?” are more difficult to answer since it requires visual understanding and reasoning ability at the same time. MedVQA generally suffers when clinicians and medical experts ask more problem-specific queries that demand detailed and descriptive answers. ² As a result, these questions are more difficult to answer compared with the closed-ended ones. ³

The development of MedVQA has been limited by the availability of high-quality labeled datasets. Early efforts, such as ImageCLEF 2018, ⁴ automatically generated questions and answers from PubMed Central image captions, but this approach suffered from noisy data. Subsequent VQA-Med datasets progressively refined the task, focusing on radiology images and clinically relevant questions.^5–7 These datasets typically classify questions into closed-ended (classification) and open-ended (generative) categories, with the latter being more challenging to evaluate. Most prior MedVQA studies have emphasized accuracy as the primary metric.^8,9 While accuracy measures exact matches between predicted and reference answers, it fails to capture partially correct or clinically informative responses. For example, when asked “Where is the brain tumor?”, a model prediction of “right lobe” contains partial insight even if it does not exactly match the reference “upper right lobe”. ¹⁰ Furthermore, traditional metrics often overlook the linguistic nuances of open-ended responses, such as synonymous terms or semantically related phrasings.

Later, several studies have attempted to address the challenges of MedVQA tasks. Frameworks like HQS-VQA ² and CGMVQA ³ explored hierarchical and hybrid classification-generation architectures. Multimodal pretraining approaches, such as MMBERT ¹¹ and PubMedCLIP-based models ¹² leveraged large-scale image-text pairs to improve vision-language alignment. More recent generative MedVQA models, including LLaVA-Med, ¹³ Med-MoE, ¹⁴ and FAVP ¹⁵ introduced instruction tuning, mixture-of-experts frameworks, and region-level prompt adaptation to enhance open-ended VQA. While these models advanced the field, their evaluations of open-ended MedVQA questions have largely relied on recall, which measures the proportion of ground-truth tokens present in the generated answer. Although recall provides insight into content coverage at the token level, it fails to fully account for syntactic coherence, semantic fidelity, or clinically relevant partial answers, potentially underestimating model performance when responses are only partly correct or phrased differently from the reference.

To address the limitations of relying solely on accuracy for evaluating MedVQA systems, this work does not primarily aim to introduce a novel model architecture; rather, it contributes from an evaluation and assessment standpoint for clinical vision–language tasks. Accordingly, we focus on a more comprehensive evaluation framework that captures the subtleties of generated responses. Specifically, we analyze the role of language-based evaluation scores in assessing predicted answers. Rather than relying solely on exact string matches, which may overlook clinically relevant but partially correct outputs, we explore how BLEU scores across multiple n-gram levels (BLEU-1 through BLEU-4) can provide a more nuanced evaluation perspective. This approach allows us to measure not just whether a prediction matches the ground-truth answer verbatim, but also the degree of semantic overlap, the preservation of key medical terminology, and the overall clinical relevance of partially correct responses. By situating different BLEU scores within the context of MedVQA, we aim to highlight their ability to account for fine-grained variations such as synonymous terminology, different phrasings, or related yet distinct conditions, thereby offering insights that accuracy alone cannot capture. The main contributions of this work can be summarized as follows.

• Comprehensive Benchmarking of Vision-Language Models: We conduct an extensive experimental study evaluating multiple vision–language encoder combinations, including BEiT, ViT, and DeiT paired with BERT, RoBERTa, and ALBERT, on the VQA-Med 2020 and VQA-Med 2021 datasets. This systematic benchmarking highlights the strengths, weaknesses, and comparative performance of these models in MedVQA, providing valuable guidance for future model selection and design.

• BLEU-based Evaluation in MedVQA: In contrast to prior studies that predominantly report accuracy, we emphasize the importance of language scores for evaluating generated responses. By exploring BLEU at different n-gram levels, we demonstrate how this metric captures varying aspects of answer quality, from word-level matches to multi-word semantic coherence. This analysis reveals how BLEU complements accuracy by uncovering degrees of partial correctness and overlaps in medically relevant terms that remain clinically significant despite not being exact matches.

• Quantitative and Qualitative Insights: Our study goes beyond numerical benchmarking by incorporating case-based qualitative analysis. We examine how BLEU scores reflect subtle distinctions such as synonymous expressions (“myocardial infarction” vs. “heart attack”) or related conditions that accuracy would simply classify as incorrect. By juxtaposing accuracy with BLEU trends, we show that multi-metric evaluation provides a more balanced and clinically meaningful assessment of MedVQA models. This dual perspective highlights the need for evaluation methods that capture not only correctness but also the relevancy and clinical utility of predicted answers.

The remainder of the paper is organized as follows. Section 2 reviews the related work. Section 3 describes the methodology, including architectural details, model training, and evaluation. Section 4 presents the experimental results and discussion, covering quantitative evaluation on two datasets and key insights from the model predictions. Section 5 concludes the study.

2. Methods

2.1. Architecture overview

As illustrated in Figure 1, the proposed MedVQA framework follows a simple fusion-based architecture designed to jointly interpret medical images and clinical questions. The input radiology image is first encoded into a sequence of visual feature tokens using a vision encoder, while the text question is processed by a language encoder to obtain contextualized word embeddings. Self-attention is applied within each modality to refine intra-modal relationships, focusing on salient visual regions in the image and key semantic phrases in the question. The resulting embeddings are then aggregated into compact image and text feature vectors. These modality-specific representations are combined through a fusion step, implemented using feature concatenation to preserve complementary information. The fused multi-modal features are then passed through a Multi-Layer Perceptron (MLP) network that learns higher-level interactions between visual and textual cues before being fed into a classifier to produce the final answer.

Figure 1.

The overview of the model architecture.

2.2. Datasets

We used two MedVQA datasets, VQA-Med 2020 and VQA-Med 2021, which offer clinically relevant image–question–answer triplets. These datasets are distinct from general-domain VQA benchmarks by focusing on medical content and associated clinical reasoning challenges. We verified dataset integrity prior to analysis and found no missing values for the required fields (image, question, reference answer) across the training, validation, and test splits.

VQA-Med 2020: The dataset used for this VQA task was constructed by automatically filtering relevant medical images from the MedPix database and generating QA pairs based on diagnostic information. Only cases where the diagnosis was made directly from imaging modalities such as CT, MRI, ultrasound, angiography, and radiographs were included. A curated list of 330 medical abnormalities was used to form QA pairs, with each condition appearing at least 10 times. Common examples include pulmonary embolism, acute appendicitis, angiomyolipoma, osteochondroma, adenocarcinoma of the lung, and sarcoidosis. ⁶ The final dataset comprises 4,000 QA pairs for training, 500 for validation, and 500 for testing, with the test set manually validated by a medical expert to ensure quality.

VQA-Med 2021: The VQA-Med 2021 dataset, developed as part of the ImageCLEF 2021 challenge, focuses on answering questions about abnormalities in radiology images. It was constructed by automatically selecting relevant medical images from the MedPix database using filters based on image captions, anatomical localities, and diagnosis methods, ensuring that only cases with image-based diagnoses were included. The dataset emphasizes frequently occurring abnormality-related question categories and consists of a training set with 4,500 QA pairs (reused from VQA-Med 2020), a newly curated validation set with 500 QA pairs, and a new test set with 500 radiology images and corresponding abnormality-related questions. ⁷ To ensure high quality, the test set reference answers were manually validated by a medical expert.

2.2.1. Image and question encoding

In this architecture, the self-attention mechanisms are independently applied within medical images and related questions. Firstly, the image input, constituting pixel intensity values, is encoded into visual features that capture semantic information relevant for subsequent processing. Simultaneously, the image related questions or queries are transformed into a sequence of tokenized units for effective interpretation. Then the model proceeds to encode these inputs. The image encoder processes images, yielding encoded image features. We evaluated three prominent transformer-based vision models.

• Vision Transformer (ViT): A transformer encoder model trained on ImageNet-21k (14M images, 21k classes) and fine-tuned on ImageNet-1k (1M images, 1k classes) at 224×224 resolution. ¹⁶

• Bidirectional Encoder representation from Image Transformers (BEiT): A ViT model pretrained in a self-supervised manner using masked image modeling on ImageNet-21k and fine-tuned on ImageNet-1k. BEiT uses a visual tokenizer from DALL-E’s Vector Quantized Variational Autoencoder (VQ-VAE) during pretraining, making it particularly strong in extracting semantically rich image features. ¹⁷

• Data-efficient image Transformer (DeiT): A lightweight ViT variant, pre-trained and fine-tuned on ImageNet-1k using knowledge distillation and optimized for training with fewer resources. ¹⁸

Concurrently, the text encoder encodes questions, generating corresponding encoded textual representations. For text encoding, we evaluated the following transformer models.

• BERT: BERT is a transformer-based model pre-trained on a large English corpus using two self-supervised objectives: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). It is an uncased model that learns deep bidirectional representations by predicting randomly masked tokens in a sentence and determining whether two input sentences follow each other. This approach enables BERT to capture rich contextual information without requiring manually labeled data. ¹⁹

• RoBERTa: RoBERTa is a case-sensitive transformer-based model pretrained on a large English corpus using a self-supervised MLM objective. It learns deep bidirectional representations by predicting masked tokens within sentences, enabling a strong contextual understanding without human-labeled data. Unlike autoregressive models, RoBERTa considers the full sentence context during training, enhancing its ability to model language dependencies. ²⁰

• ALBERT: ALBERT is a transformer-based, uncased language model pretrained on a large English corpus using self-supervised learning. It employs two key objectives: MLM, which enables the model to learn bidirectional contextual representations by predicting masked tokens, and Sentence Ordering Prediction (SOP), where the model learns the correct sequence of two sentence segments, improving coherence understanding over the traditional NSP used in BERT. ²¹

Each variant of the architecture consists of a ViT-based visual encoder and a BERT-based textual encoder. This simultaneous encoding approach enables the extraction of meaningful information from both visual and textual inputs, enabling the model to respond to image-related questions in MedVQA.

2.2.2. Fusion of features

The model uses a feature-level fusion module to combine the encoded image and textual features for MedVQA. In particular, the pooled output of the text encoder and the pooled output of the image encoder are concatenated along the feature dimension to form a unified multimodal representation. This concatenated feature vector is then passed through an MLP-based fusion network composed of linear layers, layer normalization, ReLU activations, and dropout. The first projection reduces the fused representation from 1536 to 1024 dimensions, while the second projection maps it to an intermediate latent space, enabling the model to learn compact and discriminative joint features from both modalities. Dropout is further applied to improve generalization. Finally, the fused representation is fed into a linear classifier to predict the answer. This strategy allows the model to integrate high-level visual and textual information in a unified space for answer prediction.

2.2.3. Answer prediction

Finally, the integrated representation obtained from the MLP network is passed through a classifier layer to predict the answer. This classifier layer produces a set of scores, with each score corresponding to a potential answer label. The model computes cross-entropy loss to quantify the dissimilarity between the predicted answers and the actual answers. The computed loss is considered to the output, making it integral for model training.

2.3. Model training

The model was optimized using the cross-entropy loss, with parameters updated through backpropagation to improve predictive accuracy and language-based evaluation scores on the benchmark datasets. Training was conducted using the Hugging Face Transformers library. For reproducibility, each run was initialized with a fixed random seed, and all experiments were repeated three times, with the reported results corresponding to the average across runs. The batch size was set to 16 for both training and evaluation, and the model was trained for 100 epochs. Mixed-precision training was enabled to improve computational efficiency and reduce memory consumption. In addition, a learning rate annealing strategy was employed through the built-in optimization and scheduling mechanisms of the Hugging Face Transformers framework, whereby the learning rate was progressively adjusted during training to promote stable convergence and improved performance.

2.4. Model evaluation

We evaluated a series of multimodal models by combining different pre-trained visual and textual backbones, all sourced from Hugging Face. The goal was to assess their performance on the VQA-Med 2020 and VQA-Med 2021 datasets for MedVQA. All the variants of the model were fine-tuned end-to-end on the targeted datasets without additional pretraining on radiology-specific image-text pairs.

The evaluation of the MedVQA models was conducted using both accuracy and language generation metrics to comprehensively assess performance. Accuracy was used as the primary metric to measure the proportion of correctly answered questions. In addition, we adopted BLEU scores, ranging from BLEU-1 to BLEU-4, to evaluate the quality of generated answers based on n-gram precision with respect to ground truth responses. While BLEU-1 captures unigram matches and indicates basic word-level correctness, BLEU-4 accounts for 4-gram overlaps, reflecting the fidelity and syntactic structure of the predictions. To further summarize performance across different n-gram levels, we computed the cumulative BLEU score, which provides a geometric mean of BLEU-1 through BLEU-4 and offers an overall indication of answer quality in terms of semantic fidelity and linguistic coherence. This multi-metric evaluation strategy ensures a robust analysis of both the factual correctness and linguistic clarity of model-generated answers.

3. Results

3.1. Quantitative evaluation on VQA-Med 2020

The results in Table 1 demonstrate the performance of various visual and textual encoder combinations on the VQA-Med 2020 dataset, evaluated using accuracy and BLEU metrics (BLEU-1 to BLEU-4 and cumulative BLEU). Among the combinations, DeiT + BERT achieved the highest accuracy of 52.27 ± 0.38%, along with strong BLEU scores (BLEU-4 of 12.10 ± 0.57 and cumulative BLEU of 26.87 ± 0.45). This suggests that the pair benefits from both the discriminative visual features of DeiT and the robust contextual understanding of BERT. The ViT + BERT pair also showed competitive performance with an accuracy of 50.67 ± 0.34%, reflecting the effectiveness of the ViT in capturing global visual patterns. On the contrary, combinations using ALBERT consistently yielded lower accuracy and BLEU scores across all vision backbones. For example, ViT + ALBERT achieved an accuracy of 48.47 ± 0.75% with a BLEU-4 of 7.98 ± 0.42, indicating a noticeable performance gap compared to BERT-based models. This may be due to ALBERT’s parameter-reduction strategies (factorized embedding parameterization and cross-layer parameter sharing), which, while making the model lightweight, might limit its capacity to capture fine-grained medical semantics in MedVQA tasks.

Table 1.

Accuracy and BLEU scores on VQA-Med 2020.

Visual + textual backbone	Accuracy	Bleu-1	Bleu-2	Bleu-3	Bleu-4	Cumulative bleu
BEiT + BERT	48.93±0.84	69.73±1.00	29.57±1.88	16.97±1.64	10.48±1.09	24.53±1.59
BEiT + RoBERTa	47.33±0.82	69.87±0.50	25.53±1.13	13.73±1.42	7.99±1.54	20.97±1.67
BEiT + ALBERT	47.73±1.82	69.33±0.66	26.07±1.54	13.90±0.59	7.51±0.34	20.80±0.75
DeiT + BERT	52.27±0.38	72.40±1.07	31.90±0.16	18.70±0.33	12.10±0.57	26.87±0.45
DeiT + RoBERTa	50.87±0.25	72.33±1.09	29.23±0.74	16.10±0.91	10.02±0.92	24.13±1.07
DeiT + ALBERT	49.47±0.25	71.07±1.00	28.17±1.43	15.03±1.55	8.87±1.52	22.67±1.80
ViT + BERT	50.67±0.34	73.07±0.90	30.03±0.90	16.83±1.24	10.48±1.52	24.87±1.47
ViT + RoBERTa	47.47±0.94	70.40±0.82	25.13±1.53	12.87±1.97	7.58±1.83	20.30±2.32
ViT + ALBERT	48.47±0.75	69.53±1.46	26.27±1.22	13.67±1.07	7.98±0.42	21.10±0.93

Bold indicates the best results.

The performance of RoBERTa-based combinations lies between BERT and ALBERT. While DeiT + RoBERTa achieved a reasonable accuracy of 50.87 ± 0.25%, its cumulative BLEU score was slightly lower (24.13 ± 1.07) than that of DeiT + BERT, suggesting that although RoBERTa is trained on a larger corpus, its lack of NSP might make it less suitable for understanding inter-sentence dependencies, which are often crucial in MedVQA tasks. In general, DeiT-based visual encoders outperform ViT and BEiT counterparts across textual backbones, possibly due to DeiT’s distillation-based pretraining that enables learning with fewer resources yet achieving high data efficiency. BEiT-based models lagged slightly behind, likely because BEiT’s discrete visual tokenization may not align well with the free-form textual modality of medical questions. The standard deviations across experiments are relatively small (mostly <2%), indicating stable and reproducible performance. These results highlight the importance of choosing both an expressive visual backbone and a semantically rich textual encoder for achieving high accuracy and better language generation quality in multimodal MedVQA tasks.

3.2. Quantitative evaluation on VQA-Med 2021

Table 2 presents the performance of different multimodal model configurations on the VQA-Med 2021 dataset, evaluated using accuracy and BLEU metrics (BLEU-1 to BLEU-4 and cumulative BLEU). Across all models, DeiT-based visual backbones consistently yielded the best results, with DeiT + RoBERTa achieving the highest overall accuracy of 54.87 ± 0.74% and a strong cumulative BLEU score of 24.97 ± 0.78. Close behind was DeiT + ALBERT, with the highest cumulative BLEU score of 25.50 ± 1.63 and an accuracy of 54.27 ± 0.41%. These findings indicate that DeiT’s efficient visual feature extraction and transformer-based representation contribute positively to both classification and answer generation tasks in VQA-Med 2021. Unlike the results of VQA-Med 2020 dataset, ALBERT-based models generally showed more competitive performance on this dataset. The improvement in BLEU scores for DeiT + ALBERT combination (e.g., 18.03 BLEU-3, 10.33 BLEU-4) suggests that ALBERT’s parameter-sharing structure may benefit from the characteristics of the dataset, which could contain more repetitive patterns or shorter sequences suited to ALBERT’s reduced capacity. Despite this, BERT-based models like DeiT + BERT still performed consistently well with a balance between accuracy (54.27 ± 0.94%) and cumulative BLEU (24.83 ± 1.39), showcasing the general strength of BERT’s deep and bidirectional representations.

Table 2.

Accuracy and BLEU scores on VQA-Med 2021.

Visual + textual backbone	Accuracy	Bleu-1	Bleu-2	Bleu-3	Bleu-4	Cumulative bleu
BEiT + BERT	53.07±0.09	69.07±1.48	29.30±0.59	16.17±0.77	9.19±0.66	23.37±0.74
BEiT + RoBERTa	51.20±0.59	68.13±1.23	28.10±0.71	15.83±0.62	9.19±0.25	22.97±0.31
BEiT + ALBERT	50.87±0.25	68.07±0.77	28.70±0.43	16.50±0.43	9.33±0.70	23.40±0.43
DeiT + BERT	54.27±0.94	70.20±0.85	30.90±1.13	17.83±1.46	9.87±1.02	24.83±1.39
DeiT + RoBERTa	54.87±0.74	71.53±0.98	31.10±0.99	17.83±0.81	9.87±0.45	24.97±0.78
DeiT + ALBERT	54.27±0.41	72.53±0.41	31.50±0.91	18.03±1.46	10.33±1.53	25.50±1.63
ViT + BERT	52.93±0.84	69.53±1.76	30.43±1.23	17.10±1.31	9.68±1.25	24.27±1.44
ViT + RoBERTa	54.40±1.34	70.80±0.49	31.10±1.34	17.63±1.67	9.61±1.55	24.67±1.90
ViT + ALBERT	50.27±1.09	67.67±1.46	27.80±0.57	14.40±1.75	7.44±1.57	21.10±1.99

Bold indicates the best results.

The ViT-based combinations offered moderate performance overall, with ViT + RoBERTa achieving a notable 54.40 ± 1.34% accuracy and 24.67 ± 1.90 cumulative BLEU. However, ViT + ALBERT underperformed with both the lowest accuracy (50.27 ± 1.09%) and cumulative BLEU (21.10 ± 1.99) among all combinations. This indicates that ViT’s reliance on global self-attention may not synergize well with ALBERT’s lighter-weight architecture, potentially due to insufficient modeling of fine-grained textual dependencies. Among the BEiT-based models, BEiT + BERT slightly outperformed others in terms of accuracy (53.07 ± 0.09%) and achieved a moderate cumulative BLEU score (23.37 ± 0.74). BEiT’s self-supervised pretraining approach, which involves reconstructing masked visual tokens, might be less aligned with the supervised VQA task, especially when fine-tuning on domain-specific medical data. Nonetheless, its pairing with BERT still yields decent performance, possibly due to BERT’s dual pretraining objectives (MLM and NSP) which enhance its ability to capture contextual relationships in the textual input (Figures 2 and 3).

Figure 2.

Accuracy Comparison on VQA-Med 2020 vs VQA-Med 2021.

Figure 3.

Cumulative BLEU Comparison on VQA-Med 2020 vs VQA-Med 2021.

Figures 2 and 3 illustrate the comparison of model accuracies and cumulative BLUE performance on the VQA-Med 2020 and VQA-Med 2021 datasets across different Visual–Textual backbone combinations. The plot clearly highlights that DeiT-based models, particularly DeiT + BERT, outperform other architectures in accuracy. The DeiT + ALBERT and DeiT + RoBERTa models achieved the highest accuracies, demonstrating the robustness of DeiT-based approaches across datasets. In summary, DeiT-based models consistently outperform others across most evaluation metrics, confirming the advantage of data-efficient transformers in vision-language tasks. Additionally, RoBERTa and BERT remain strong text encoders, with ALBERT showing competitive BLEU performance but slightly reduced accuracy. These results emphasize the importance of choosing well-balanced vision and language encoders to optimize both accuracy and answer generation quality in MedVQA systems.

4. Discussion

4.1. Comparisons with state-of-the-arts

Previous studies in VQA-Med 2020 primarily relied on Convolutional Neural Network (CNN)-based architectures such as ResNet, DenseNet, or Inception for extracting visual features, combined with Recurrent Neural Networks (RNNs) or simple transformer-based models for question encoding. Teams such as AIML, HCP-MIC, and NLM achieved accuracies in the range of 14.2% to 49.6%, with BLEU scores reported between 0.177 and 0.542.^22–24 These approaches were mostly classification-focused and optimized for exact-match accuracy, with BLEU used as a secondary measure. However, the reported BLEU values did not clarify whether they corresponded to BLEU-1, BLEU-4, or an aggregated score, limiting their interpretability. In contrast, our work advances beyond these efforts by employing DeiT-based visual backbones with transformer text encoders, achieving higher accuracy (52.27%) and investigating a comprehensive BLEU evaluation across multiple n-gram levels and cumulative BLEU. This provides deeper insights into semantic correctness, capturing synonymous terms and clinically meaningful overlaps that prior studies did not explicitly address.

Building on the VQA-Med 2020 analysis, submissions in VQA-Med 2021 challenge continued to rely primarily on CNN-based visual encoders combined with RNNs or shallow transformer text encoders. The highest accuracy was achieved by SYSU-HCP at 0.382, with the top reported BLEU score being 0.416. ²⁵ Other notable approaches such as TeamS, Lijie, and TAM achieved accuracies of 34.8%, 31.6%, and 22.2%, respectively, with corresponding BLEU scores of 0.391, 0.352, and 0.255.^26–28 Similar to the previous year, these BLEU values do not clarify whether they correspond to BLEU-1, BLEU-4, or a cumulative metric, making it difficult to assess semantic correctness or clinically meaningful partial predictions. In contrast, our DeiT-based framework paired with transformer text encoders achieves substantially higher accuracy of 54.87%, while providing a comprehensive BLEU evaluation across multiple n-gram levels and cumulative BLEU, enabling a more nuanced assessment of semantic alignment and clinical relevance compared to prior approaches.

4.2. Language score vs. accuracy

Accuracy provides a binary evaluation, classifying a predicted answer as either correct when it exactly matches the ground truth or incorrect otherwise. While this works for rigid tasks, it falls short in MedVQA, where answers often involve synonyms, paraphrases, or clinically equivalent terms. For example, a prediction such as “In the right lower lung” versus the ground truth “Right lower lobe” would be marked incorrect by accuracy, even though both refer to the same anatomical region. Language scores, such as BLEU, complement accuracy by capturing the degree of overlap between predicted response and reference answer across different n-gram levels. This allows for recognition of partial correctness and semantic similarity, rewarding predictions that are not exact matches but still clinically meaningful. Together, accuracy and language scores provide a more comprehensive evaluation, balancing strict correctness with nuanced assessment of medical relevance.

Across all models, the BLEU scores show a natural decline from BLEU-1 to BLEU-4 due to increasing n-gram overlap constraints. The standard deviations remain low across metrics, signifying consistent behavior over multiple runs. Besides, the results highlight the importance of synergizing visual and textual encoder choices to maximize both answer accuracy and language generation quality in domain-specific MedVQA tasks.

4.3. Analysis of model predictions

The results of MedVQA samples from Figure 4 highlight how BLEU scores, particularly n-gram-based evaluations, offer a more graded and meaningful assessment of MedVQA predictions compared to strict accuracy. In the first case, for the question “What is the primary abnormality in this image?”, the model predicted “Pulmonary embolism” instead of the ground-truth “Pulmonary embolus.” While the medical terminology differs slightly in suffix, the semantic meaning is identical. This leads to high BLEU-1 and BLEU-2 scores, as unigrams and bigrams such as “pulmonary” and “embol” are shared between the two answers, even though accuracy would mark this prediction as incorrect. In the second case involving the knee X-ray, the model responded with “Osteochondroma” to the expected “Idiopathic synovial osteochondromatosis.” Although the prediction is not a perfect match, it captures the core pathological term “osteochondro-”, suggesting partial understanding of the underlying disease class. Here, BLEU-1 might be moderate, but higher n-grams (BLEU-2 to BLEU-4) will likely yield lower scores due to the omission of modifiers like “idiopathic” and “synovial,” which reduce overlap in longer n-grams. In the third case, the model’s prediction “Inguinal hernia involving bladder” partially overlaps with the correct answer “Amyand hernia.” Although it fails to label the hernia subtype accurately, it reflects some level of anatomical insight. Given this, BLEU-1 and BLEU-2 scores may still be non-zero, capturing shared terms like “hernia,” but BLEU-3 and BLEU-4 scores would be low or zero due to a lack of multi-word alignment with the ground truth (Figure 5).

Figure 4.

VQA-Med 2020 samples indicating n-gram BLEU scores’ effectiveness in evaluating predictions.

Figure 5.

VQA-Med 2021 samples indicating n-gram BLEU scores’ effectiveness in evaluating predictions.

The predicted answers in Figure 5 further demonstrate the effectiveness of language-based metrics in capturing partial correctness and semantic similarity in MedVQA tasks. In the first example, the model predicted “Pulmonary sarcoidosis” for the ground-truth answer “Sarcoid”. Here, BLEU-1 and BLEU-2 scores are expected to be high, as the prediction not only includes the core medical term “sarcoid” but also provides more anatomical specificity. This indicates a clinically accurate and linguistically enriched response, which exact match metrics would incorrectly penalize. In the second case, the model predicted “Metastatic disease from colonic carcinoma” instead of the correct response, “Carcinoma, small cell.” Although both answers refer to malignancies and share the word “carcinoma,” they describe entirely different disease origins. Consequently, BLEU-1 may reflect some unigram overlap, but BLEU-2 to BLEU-4 scores are expected to be low, accurately indicating a weak match in multi-word sequences. In the third example, the model predicted “Zenker diverticulum” in place of the ground-truth “Killian-jamieson diverticulum”. Despite the anatomical misclassification, both are esophageal diverticula, meaning BLEU-1 will capture partial lexical overlap (e.g., “diverticulum”), but higher-order BLEU scores will drop, highlighting the distinction in specificity. These cases collectively illustrate how BLEU scores offer a graded and interpretable metric for evaluating MedVQA predictions, distinguishing between closely related but imperfect answers and entirely incorrect ones (Figure 6).

Figure 6.

Samples from VQA-Med 2021 where predicted answers match ground-truths but differ in preprocessed form, illustrating the need for language-aware evaluation beyond exact match.

As shown in Figure 6, the predicted answers match the ground-truth responses, although they are presented in a preprocessed form rather than their original representation. This indicates that the model has correctly identified the clinical concept, yet the difference lies in tokenization, or normalization applied during preprocessing. For instance, medical terms may be standardized by removing punctuations, and parentheses, which does not alter the underlying meaning but changes the surface form of the text. Such cases highlight the limitations of using strict exact-match metrics, as they would incorrectly penalize predictions that are semantically and clinically equivalent but textually different. Metrics that account for partial overlap or semantic similarity, such as BLEU or embedding-based measures, are better suited for evaluating these scenarios because they recognize that the essential medical information has been preserved. These samples underscore the importance of adopting language-aware evaluation strategies in MedVQA tasks, ensuring that clinically accurate answers are not unfairly judged as incorrect due to preprocessing or superficial differences in wording.

These results demonstrate how n-gram BLEU scores serve as a continuous and informative metric, reflecting the semantic closeness and medical relevance of the model’s outputs. They reward partial correctness, which is crucial in MedVQA tasks, where multiple valid phrasings and levels of specificity are common.

4.4. Limitations and future directions

While BLEU scores provide a graded measure of lexical and phrase-level overlap between predicted and reference answers, they have inherent limitations in clinical language evaluation. BLEU primarily measures surface-level n-gram similarity and is not designed to assess semantic similarity beyond lexical matching. Consequently, answers that convey the same clinical meaning through different terminology or phrasing may be unfairly penalized, whereas responses with greater lexical overlap may still be clinically incorrect. In medical applications, such discrepancies are critical, as plausible but erroneous responses could lead to misinterpretation or unsafe clinical conclusions. Therefore, BLEU should not be considered a standalone evaluation criterion. Instead, it should be interpreted alongside strict accuracy measures and complementary evaluation approaches to ensure both linguistic quality and clinical validity.

Another important limitation of this study is the absence of evaluation by clinical experts. Although automated metrics enable scalable and reproducible performance assessment, they cannot adequately determine the clinical correctness, diagnostic relevance, or safety of model-generated answers. Expert-driven qualitative evaluation is particularly crucial for open-ended MedVQA tasks, where subtle semantic differences may have significant diagnostic implications. Moreover, the objective of this work is not to establish statistically significant performance superiority over existing MedVQA methods, but rather to provide a complementary perspective on MedVQA evaluation by examining the behavior and limitations of commonly used automatic metrics. Accordingly, the reported experiments are intended to offer descriptive insight into evaluation trends and variability rather than to serve as a rigorous hypothesis-testing framework. Future work will therefore include statistical significance analysis in more explicitly comparative studies, as well as structured clinician-in-the-loop evaluation protocols to assess the clinical relevance, factual correctness, interpretability, and potential safety risks of generated responses. Such extensions are essential for improving the reliability, trustworthiness, and real-world applicability of MedVQA systems.

5. Conclusion

In this study, we conducted a systematic benchmarking of multiple vision–language model combinations for MedVQA on the VQA-Med 2020 and VQA-Med 2021 datasets, highlighting their comparative strengths and weaknesses. To complement accuracy, which remains the predominant evaluation metric, we explored the use of BLEU scores at different n-gram levels to capture dimensions of partial correctness and semantic overlap. Our experiments demonstrated that BLEU provides additional insights beyond exact-match accuracy by rewarding clinically meaningful predictions that share key terminology or convey semantically equivalent information, even when phrasing differs.

Our experimental evaluation revealed that models using DeiT-based visual backbones in combination with transformer text encoders consistently achieved top performance. In particular, BERT reached 52.27% accuracy with a cumulative BLEU of 26.87 on VQA-Med 2020, while RoBERTa achieved 54.87% accuracy and ALBERT recorded the highest cumulative BLEU of 25.50 on VQA-Med 2021. Through both quantitative evaluation and qualitative case analyses, we showed how BLEU trends reveal insights such as synonymous terms and related conditions, thereby offering a more comprehensive perspective on model performance. By incorporating BLEU-based evaluation, we move toward a more clinically aligned and linguistically sensitive assessment framework that better reflects the usefulness of MedVQA systems. Our findings advocate for the adoption of such language metrics in future research and benchmarking efforts, promoting the development of models that are not only accurate but also contextually aware and clinically meaningful.