Instruct Me More! Random Prompting for Visual In-Context Learning
Jiahao Zhang Bowen Wang Liangzhi Li Yuta Nakashima Hajime Nagahara
Large-scale models trained on extensive datasets, have emerged as the
preferred approach due to their high generalizability across various tasks.
In-context learning (ICL), a popular strategy in natural language processing,
uses such models for different tasks by providing instructive prompts but
without updating model parameters. This idea is now being explored in computer
vision, where an input-output image pair (called an in-context pair) is
supplied to the model with a query image as a prompt to exemplify the desired
output. The efficacy of visual ICL often depends on the quality of the prompts.
We thus introduce a method coined Instruct Me More (InMeMo), which augments
in-context pairs with a learnable perturbation (prompt), to explore its
potential. Our experiments on mainstream tasks reveal that InMeMo surpasses the
current state-of-the-art performance. Specifically, compared to the baseline
without learnable prompt, InMeMo boosts mIoU scores by 7.35 and 15.13 for
foreground segmentation and single object detection tasks, respectively. Our
findings suggest that InMeMo offers a versatile and efficient way to enhance
the performance of visual ICL with lightweight training. Code is available at
https://github.com/Jackieam/InMeMo.