Special Topic: Large Multimodal Models
RESEARCH PAPER Webpage Webpage-cn SpringerLink Google Scholar

How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites
Chen Z, Wang W Y, Tian H, et al
Sci China Inf Sci, 2024, 67(12): 220101
Keywords: multimodal model; open-source; vision encoder; dynamic resolution; bilingual dataset; LMM
Cite as: Chen Z, Wang W Y, Tian H, et al. How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites. Sci China Inf Sci, 2024, 67(12): 220101, doi: 10.1007/s11432-024-4231-5

Special Topic: Large Multimodal Models
RESEARCH PAPER Webpage Webpage-cn SpringerLink Google Scholar

OCRBench: on the hidden mystery of OCR in large multimodal models
Liu Y L, Li Z, Huang M X, et al
Sci China Inf Sci, 2024, 67(12): 220102
Keywords: large multimodal model; LMM; OCR; text recognition; scene text-centric VQA; document-oriented VQA; key information extraction; handwritten mathematical expression recognition
Cite as: Liu Y L, Li Z, Huang M X, et al. OCRBench: on the hidden mystery of OCR in large multimodal models. Sci China Inf Sci, 2024, 67(12): 220102, doi: 10.1007/s11432-024-4235-6

Special Topic: Large Multimodal Models
RESEARCH PAPER Webpage Webpage-cn SpringerLink Google Scholar

MMInstruct: a high-quality multi-modal instruction tuning dataset with extensive diversity
Liu Y Z, Cao Y, Gao Z W, et al
Sci China Inf Sci, 2024, 67(12): 220103
Keywords: instruction tuning; multi-modal; multi-domain; dataset; vision large language model; LMM
Cite as: Liu Y Z, Cao Y, Gao Z W, et al. MMInstruct: a high-quality multi-modal instruction tuning dataset with extensive diversity. Sci China Inf Sci, 2024, 67(12): 220103, doi: 10.1007/s11432-024-4187-3

Special Topic: Large Multimodal Models
RESEARCH PAPER Supplementary Webpage Webpage-cn SpringerLink Google Scholar

Woodpecker: hallucination correction for multimodal large language models
Yin S K, Fu C Y, Zhao S R, et al
Sci China Inf Sci, 2024, 67(12): 220105
Keywords: multimodal learning; multimodal large language models; hallucination correction; large language models; vision and language; LMM
Cite as: Yin S K, Fu C Y, Zhao S R, et al. Woodpecker: hallucination correction for multimodal large language models. Sci China Inf Sci, 2024, 67(12): 220105, doi: 10.1007/s11432-024-4251-x

Special Topic: Large Multimodal Models
RESEARCH PAPER Webpage Webpage-cn SpringerLink Google Scholar

DocPedia: unleashing the power of large multimodal model in the frequency domain for versatile document understanding
Feng H, Liu Q, Liu H, et al
Sci China Inf Sci, 2024, 67(12): 220106
Keywords: document understanding; large multimodal model; LMM; OCR-free; high-resolution; frequency
Cite as: Feng H, Liu Q, Liu H, et al. DocPedia: unleashing the power of large multimodal model in the frequency domain for versatile document understanding. Sci China Inf Sci, 2024, 67(12): 220106, doi: 10.1007/s11432-024-4250-y

Special Topic: Large Multimodal Models
RESEARCH PAPER Webpage Webpage-cn SpringerLink Google Scholar

Modality-experts coordinated adaptation for large multimodal models
Zhang Y, Ji Z, Pang Y W, et al
Sci China Inf Sci, 2024, 67(12): 220107
Keywords: large multimodal model; LMM; multimodal learning; vision-language pretraining; parameter-efficient fine-tuning; adapter; modality expert
Cite as: Zhang Y, Ji Z, Pang Y W, et al. Modality-experts coordinated adaptation for large multimodal models. Sci China Inf Sci, 2024, 67(12): 220107, doi: 10.1007/s11432-024-4234-4

Special Topic: Large Multimodal Models
LETTER Supplementary Webpage Webpage-cn SpringerLink Google Scholar

COMET: "cone of experience" enhanced large multimodal model for mathematical problem generation
Liu S N Y, Feng J T, Yang Z K, et al
Sci China Inf Sci, 2024, 67(12): 220108
Keywords: mathematical problem generation; mathematical problem solving; large multimodal model; LMM; educational application; smart education
Cite as: Liu S N Y, Feng J T, Yang Z K, et al. COMET: "cone of experience" enhanced large multimodal model for mathematical problem generation. Sci China Inf Sci, 2024, 67(12): 220108, doi: 10.1007/s11432-024-4242-0

Special Topic: Large Multimodal Models
LETTER Supplementary Webpage Webpage-cn SpringerLink Google Scholar

ChemDFM-X: towards large multimodal model for chemistry
Zhao Z H, Chen B, Li J P, et al
Sci China Inf Sci, 2024, 67(12): 220109
Keywords: LMM; AI for Science; Instruction-Tuning; Cross-Modality; Chemistry
Cite as: Zhao Z H, Chen B, Li J P, et al. ChemDFM-X: towards large multimodal model for chemistry. Sci China Inf Sci, 2024, 67(12): 220109, doi: 10.1007/s11432-024-4243-0