Enxin/Video-MMLU
Viewer
β’
Updated
β’
1.07k
β’
2.39k
β’
8
EnxinSong
Enxin
AI & ML interests
None yet
Recent Activity
updated
a dataset
1 day ago
Enxin/Video-MMLU
published
a dataset
6 days ago
Enxin/lmms_video_mmlu
updated
a collection
6 days ago
Video-MMLU
Organizations
Enxin's activity
updated
a
dataset
1 day ago
published
a
dataset
6 days ago
updated
a
collection
6 days ago
posted
an
update
6 days ago
Post
1149
π Introducing Video-MMLU, a new benchmark for evaluating large multimodal models on classroom-style lectures in math, physics, and chemistry!
π§βπ«πVideo-MMLU requires strong reasoning capabilities and world knowledge compared to the previous benchmarks for video LMMs.
Each video comes with two tasks:
π Take Notes β detailed captioning of multi-discipline lectures
π§ Do Quiz β open-ended QA to test reasoning over visuals & proofs
We evaluated 90+ models, including vision-blind baselines, open-source models and proprietary ones.
π We find that existing models generally perform poorly, with accuracy ranging from only 10% to 50%.
πWe also explore how the number of visual tokens and the base LLMs influence performance, offering insights into the interplay between multimodal perception and reasoning in lecture comprehension.
For more details, please check belowοΌ
π Paper: https://arxiv.org/abs/2504.14693
π» Code: https://github.com/Espere-1119-Song/Video-MMLU
π§ Data: Enxin/Video-MMLU
π Website: https://enxinsong.com/Video-MMLU-web/
π§βπ«πVideo-MMLU requires strong reasoning capabilities and world knowledge compared to the previous benchmarks for video LMMs.
Each video comes with two tasks:
π Take Notes β detailed captioning of multi-discipline lectures
π§ Do Quiz β open-ended QA to test reasoning over visuals & proofs
We evaluated 90+ models, including vision-blind baselines, open-source models and proprietary ones.
π We find that existing models generally perform poorly, with accuracy ranging from only 10% to 50%.
πWe also explore how the number of visual tokens and the base LLMs influence performance, offering insights into the interplay between multimodal perception and reasoning in lecture comprehension.
For more details, please check belowοΌ
π Paper: https://arxiv.org/abs/2504.14693
π» Code: https://github.com/Espere-1119-Song/Video-MMLU
π§ Data: Enxin/Video-MMLU
π Website: https://enxinsong.com/Video-MMLU-web/
liked
a
dataset
12 days ago
upvoted
a
paper
about 1 month ago
updated
4
datasets
about 1 month ago