Can Fine-Tuned Vision-Language Models Generalize Across Tasks on the Same Images?

A detailed caption can generate many questions about an image. But how many of those questions do we need for useful training? If a model learns to count objects, does it also learn to locate them? If it learns to count objects of one color, can it count objects of a particular shape?

These questions matter because training data often treats each image-question pair as a separate example, even when several questions draw on the same visual content. I ran a small experiment to see whether fine-tuning on one task transfers to other tasks over the same images.

Many vision-language models (VLMs) use a vision encoder aligned with text through a contrastive objective such as CLIP. Pooling patch-level image features can discard fine detail, while captions used during alignment may leave out details that later tasks require. Despite this, VLMs can perform well on OCR. Their weaknesses are more visible on tasks that depend on precise visual relationships, such as chart understanding and spatial reasoning.

Experiment question

The experiment asks whether training on a subset of questions can help a model use an image more broadly. For example, does training on object counts transfer to locating objects, or does training on color counts transfer to shape counts?

The central idea is image reuse during training: when one image supports many possible questions, does a model benefit from seeing more of those questions, or can a few representative tasks teach it to use the image more effectively?

Synthetic task design

To control the visual details and question types, I generated 1,000 scenes in Blender using this script. Each scene contains 3–8 objects with randomized shapes, colors, materials, and sizes. Here is one example of the scene attributes:

{
  "shapes": ["cube", "sphere", "cylinder"],
  "colors": ["gray", "red", "blue", "green", "brown", "purple", "cyan", "yellow"],
  "materials": ["rubber", "metal"],
  "sizes": ["large", "small"]
}

I designed eight tasks:

• Extreme material, color, or shape: identify the property of the leftmost, rightmost, farthest, or nearest object.
• Extreme all: identify the leftmost, rightmost, farthest, or nearest object and report its size, color, material, and shape.
• Count material, color, or shape: count objects with a specified property.
• Count relative to: count objects positioned to one side of, closer than, or farther than a specified object.

For people, I expected relative counting to be hardest because it requires finding a reference object before counting. Simple property counts should be easier, followed by identifying one property at an extreme. Identifying all four properties of an extreme object also requires a longer, precisely formatted answer, which may make it difficult for a VLM.

Data and models

I used 800 images for training and held out 200 for testing. For evaluation, I sampled up to 1,500 questions per task from each split. I compared training on all eight tasks with training on one task at a time, including a balanced setting with about 188 examples per task (roughly 1,500 total).

The models were Qwen2-VL-2B-Instruct and Llama-3.2-11B-Vision-Instruct, both in 4-bit form. I fine-tuned each model for two epochs with QLoRA using Unsloth, keeping the vision encoder frozen. The in-domain evaluation uses images seen during training; the out-of-domain evaluation uses the held-out images.

Results at a glance

The results suggest that fine-tuning can transfer across tasks on these synthetic scenes, though the size and direction of that transfer depend on the model and task. In the Llama evaluation, training on the Extreme Color task reached about 58% of the gain from training on all eight tasks, while also improving several other tasks.

A few patterns stood out:

• Training on all tasks together generally performed best.
• For Llama, performance was similar on the in-domain and held-out image sets, suggesting that the gains were not limited to memorizing the training images.
• With a budget of roughly 1,500 examples, training on Extreme Color performed comparably to splitting the budget across all eight tasks in the reported comparison.
• Qwen2-VL-2B showed little or inconsistent improvement on several tasks, suggesting that transfer also depends on the model’s starting capabilities.

These are pilot results from a controlled synthetic setting. They do not establish that the same transfer patterns will hold for natural images or other models, but they offer a useful way to study how much visual skill a model can reuse across tasks.

Detailed results

In-domain evaluation

Baseline and fine-tuned accuracy for Llama-3.2-11B-Vision-Instruct. Each cell shows accuracy.

Cross-task transfer after fine-tuning Llama-3.2-11B-Vision-Instruct on one task. Each cell shows the fraction of the improvement from full fine-tuning recovered by one-task fine-tuning:

(one-task accuracy − baseline accuracy) / (all-task accuracy − baseline accuracy) × 100%

Held-out images

Baseline and fine-tuned accuracy for Llama-3.2-11B-Vision-Instruct on held-out images. Each cell shows accuracy.

Cross-task transfer on held-out images after fine-tuning on one task, using the same transfer score defined above.

Strongest cross-task transfer

Held-out images. Yellow marks the test task that benefited most from each training task, excluding the matching task.