MiniCPM-o

图像理解能力

注意：闭源模型的 Token Density 由 API 收费方式估算得到。

多图和视频理解能力

语音理解能力

语音生成能力。

端到端声音克隆能力。

多模态流式交互能力: StreamingBench 分数

MiniCPM-V 2.6

MiniCPM-V 2.6 是 MiniCPM-V 系列中最新、性能最佳的模型。该模型基于 SigLip-400M 和 Qwen2-7B 构建，共 8B 参数。与 MiniCPM-Llama3-V 2.5 相比，MiniCPM-V 2.6 性能提升显著，并引入了多图和视频理解的新功能。MiniCPM-V 2.6 的主要特点包括：

• 🔥 领先的性能。 MiniCPM-V 2.6 在最新版本 OpenCompass 榜单上（综合 8 个主流多模态评测基准）平均得分 65.2，以8B量级的大小在单图理解方面超越了 GPT-4o mini、GPT-4V、Gemini 1.5 Pro 和 Claude 3.5 Sonnet 等主流商用闭源多模态大模型。

• 🖼️ 多图理解和上下文学习。 MiniCPM-V 2.6 还支持多图对话和推理。它在 Mantis-Eval、BLINK、Mathverse mv 和 Sciverse mv 等主流多图评测基准中取得了最佳水平，并展现出了优秀的上下文学习能力。

• 🎬 视频理解。 MiniCPM-V 2.6 还可以接受视频输入，进行对话和提供涵盖时序和空间信息的详细视频描述。模型在 有/无字幕 评测场景下的 Video-MME 表现均超过了 GPT-4V、Claude 3.5 Sonnet 和 LLaVA-NeXT-Video-34B等商用闭源模型。

• 💪 强大的 OCR 能力及其他功能。 MiniCPM-V 2.6 可以处理任意长宽比的图像，像素数可达 180 万（如 1344x1344）。在 OCRBench 上取得最佳水平，超过 GPT-4o、GPT-4V 和 Gemini 1.5 Pro 等商用闭源模型。基于最新的 RLAIF-V 和 VisCPM 技术，其具备了可信的多模态行为，在 Object HalBench 上的幻觉率显著低于 GPT-4o 和 GPT-4V，并支持英语、中文、德语、法语、意大利语、韩语等多种语言。

• 🚀 卓越的效率。 除了对个人用户友好的模型大小，MiniCPM-V 2.6 还表现出最先进的视觉 token 密度（即每个视觉 token 编码的像素数量）。它仅需 640 个 token 即可处理 180 万像素图像，比大多数模型少 75%。这一特性优化了模型的推理速度、首 token 延迟、内存占用和功耗。因此，MiniCPM-V 2.6 可以支持 iPad 等终端设备上的高效实时视频理解。

• 💫 易于使用。 MiniCPM-V 2.6 可以通过多种方式轻松使用：(1) llama.cpp 和 ollama 支持在本地设备上进行高效的 CPU 推理，(2) int4 和 GGUF 格式的量化模型，有 16 种尺寸，(3) vLLM 支持高吞吐量和内存高效的推理，(4) 针对新领域和任务进行微调，(5) 使用 Gradio 快速设置本地 WebUI 演示，(6) 在线demo即可体验。

性能评估

典型示例

点击查看更多示例。

我们将 MiniCPM-V 2.6 部署在iPad Pro上，并录制了以下演示视频。

python
1import torch
2from PIL import Image
3from transformers import AutoModel, AutoTokenizer
4
5model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True,
6    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
7model = model.eval().cuda()
8tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True)
9
10image1 = Image.open('image1.jpg').convert('RGB')
11image2 = Image.open('image2.jpg').convert('RGB')
12question = 'Compare image 1 and image 2, tell me about the differences between image 1 and image 2.'
13
14msgs = [{'role': 'user', 'content': [image1, image2, question]}]
15
16answer = model.chat(
17    msgs=msgs,
18    tokenizer=tokenizer
19)
20print(answer)

python
1import torch
2from PIL import Image
3from transformers import AutoModel, AutoTokenizer
4
5model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True,
6    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
7model = model.eval().cuda()
8tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True)
9
10question = "production date" 
11image1 = Image.open('example1.jpg').convert('RGB')
12answer1 = "2023.08.04"
13image2 = Image.open('example2.jpg').convert('RGB')
14answer2 = "2007.04.24"
15image_test = Image.open('test.jpg').convert('RGB')
16
17msgs = [
18    {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]},
19    {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]},
20    {'role': 'user', 'content': [image_test, question]}
21]
22
23answer = model.chat(
24    msgs=msgs,
25    tokenizer=tokenizer
26)
27print(answer)

python
1import torch
2from PIL import Image
3from transformers import AutoModel, AutoTokenizer
4from decord import VideoReader, cpu    # pip install decord
5
6model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True,
7    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
8model = model.eval().cuda()
9tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True)
10
11MAX_NUM_FRAMES=64 # if cuda OOM set a smaller number
12
13def encode_video(video_path):
14    def uniform_sample(l, n):
15        gap = len(l) / n
16        idxs = [int(i * gap + gap / 2) for i in range(n)]
17        return [l[i] for i in idxs]
18
19    vr = VideoReader(video_path, ctx=cpu(0))
20    sample_fps = round(vr.get_avg_fps() / 1)  # FPS
21    frame_idx = [i for i in range(0, len(vr), sample_fps)]
22    if len(frame_idx) > MAX_NUM_FRAMES:
23        frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)
24    frames = vr.get_batch(frame_idx).asnumpy()
25    frames = [Image.fromarray(v.astype('uint8')) for v in frames]
26    print('num frames:', len(frames))
27    return frames
28
29video_path="video_test.mp4"
30frames = encode_video(video_path)
31question = "Describe the video"
32msgs = [
33    {'role': 'user', 'content': frames + [question]}, 
34]
35
36# Set decode params for video
37params = {}
38params["use_image_id"] = False
39params["max_slice_nums"] = 2 # use 1 if cuda OOM and video resolution > 448*448
40
41answer = model.chat(
42    msgs=msgs,
43    tokenizer=tokenizer,
44    **params
45)
46print(answer)

python
1import torch
2import librosa
3from transformers import AutoModel, AutoTokenizer
4
5model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True,
6    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
7model = model.eval().cuda()
8tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True)
9
10model.init_tts()
11model.tts.float()

• Mimick 任务反映了模型的端到端语音建模能力。模型接受音频输入，输出语音识别（ASR）转录结果，并随后以高相似度重建原始音频。重建的音频相似度和原始音频越高，表明模型有越高的语音端到端建模基础能力。

python
1mimick_prompt = "Please repeat each user's speech, including voice style and speech content."
2audio_input, _ = librosa.load('xxx.wav', sr=16000, mono=True)
3msgs = [{'role': 'user', 'content': [mimick_prompt,audio_input]}]
4res = model.chat(
5    msgs=msgs,
6    tokenizer=tokenizer,
7    sampling=True,
8    max_new_tokens=128,
9    use_tts_template=True,
10    temperature=0.3,
11    generate_audio=True,
12    output_audio_path='output.wav', # save the tts result to output_audio_path
13)

python
1ref_audio, _ = librosa.load('./assets/voice_01.wav', sr=16000, mono=True) # load the reference audio
2
3# Audio RolePlay:  # With this mode, model will role-play the character based on the audio prompt.
4sys_prompt = model.get_sys_prompt(ref_audio=ref_audio, mode='audio_roleplay', language='en')
5user_question = {'role': 'user', 'content': [librosa.load('xxx.wav', sr=16000, mono=True)[0]]}
6
7# Audio Assistant: # With this mode, model will speak with the voice in ref_audio as a AI assistant.
8# sys_prompt = model.get_sys_prompt(ref_audio=ref_audio, mode='audio_assistant', language='en') 
9# user_question = {'role': 'user', 'content': [librosa.load('xxx.wav', sr=16000, mono=True)[0]]} # Try to ask something!

python
1msgs = [sys_prompt, user_question]
2res = model.chat(
3    msgs=msgs,
4    tokenizer=tokenizer,
5    sampling=True,
6    max_new_tokens=128,
7    use_tts_template=True,
8    generate_audio=True,
9    temperature=0.3,
10    output_audio_path='result.wav',
11)
12
13# round two
14history = msgs.append({'role': 'assistant', 'content': res})
15user_question = {'role': 'user', 'content': [librosa.load('xxx.wav', sr=16000, mono=True)[0]]}
16msgs = history.append(user_question)
17res = model.chat(
18    msgs=msgs,
19    tokenizer=tokenizer,
20    sampling=True,
21    max_new_tokens=128,
22    use_tts_template=True,
23    generate_audio=True,
24    temperature=0.3,
25    output_audio_path='result_round_2.wav',
26)
27print(res)

python
1'''
2Audio Understanding Task Prompt:
3Speech:
4    ASR with ZH(same as AST en2zh): 请仔细听这段音频片段，并将其内容逐字记录。
5    ASR with EN(same as AST zh2en): Please listen to the audio snippet carefully and transcribe the content.
6    Speaker Analysis: Based on the speaker's content, speculate on their gender, condition, age range, and health status.
7General Audio:
8    Audio Caption: Summarize the main content of the audio.
9    Sound Scene Tagging: Utilize one keyword to convey the audio's content or the associated scene.
10'''
11task_prompt = "\n"
12audio_input, _ = librosa.load('xxx.wav', sr=16000, mono=True)
13
14msgs = [{'role': 'user', 'content': [task_prompt,audio_input]}]
15
16res = model.chat(
17    msgs=msgs,
18    tokenizer=tokenizer,
19    sampling=True,
20    max_new_tokens=128,
21    use_tts_template=True,
22    generate_audio=True,
23    temperature=0.3,
24    output_audio_path='result.wav',
25)
26print(res)

python
1'''
2Speech Generation Task Prompt:
3    Human Instruction-to-Speech: see https://voxinstruct.github.io/VoxInstruct/
4    Example:
5        # 在新闻中，一个年轻男性兴致勃勃地说：“祝福亲爱的祖国母亲美丽富强！”他用低音调和低音量，慢慢地说出了这句话。
6        # Delighting in a surprised tone, an adult male with low pitch and low volume comments:"One even gave my little dog a biscuit" This dialogue takes place at a leisurely pace, delivering a sense of excitement and surprise in the context. 
7
8    Voice Cloning or Voice Creation: With this mode, model will act like a TTS model. 
9'''
10# Human Instruction-to-Speech:
11task_prompt = '' #Try to make some Human Instruction-to-Speech prompt
12msgs = [{'role': 'user', 'content': [task_prompt]}] # you can try to use the same audio question
13
14# Voice Cloning mode: With this mode, model will act like a TTS model. 
15# sys_prompt = model.get_sys_prompt(ref_audio=ref_audio, mode='voice_cloning', language='en')
16# text_prompt = f"Please read the text below."
17# user_question = {'role': 'user', 'content': [text_prompt, "content that you want to read"]} # using same voice in sys_prompt to read the text. (Voice Cloning)
18# user_question = {'role': 'user', 'content': [text_prompt, librosa.load('xxx.wav', sr=16000, mono=True)[0]]} # using same voice in sys_prompt to read 'xxx.wav'. (Voice Creation)
19
20msgs = [sys_prompt, user_question]
21res = model.chat(
22    msgs=msgs,
23    tokenizer=tokenizer,
24    sampling=True,
25    max_new_tokens=128,
26    use_tts_template=True,
27    generate_audio=True,
28    temperature=0.3,
29    output_audio_path='result.wav',
30)
31
32

python
1import math
2import numpy as np
3from PIL import Image
4from moviepy.editor import VideoFileClip
5import tempfile
6import librosa
7import soundfile as sf
8import torch
9from transformers import AutoModel, AutoTokenizer
10
11def get_video_chunk_content(video_path, flatten=True):
12    video = VideoFileClip(video_path)
13    print('video_duration:', video.duration)
14    
15    with tempfile.NamedTemporaryFile(suffix=".wav", delete=True) as temp_audio_file:
16        temp_audio_file_path = temp_audio_file.name
17        video.audio.write_audiofile(temp_audio_file_path, codec="pcm_s16le", fps=16000)
18        audio_np, sr = librosa.load(temp_audio_file_path, sr=16000, mono=True)
19    num_units = math.ceil(video.duration)
20    
21    # 1 frame + 1s audio chunk
22    contents= []
23    for i in range(num_units):
24        frame = video.get_frame(i+1)
25        image = Image.fromarray((frame).astype(np.uint8))
26        audio = audio_np[sr*i:sr*(i+1)]
27        if flatten:
28            contents.extend(["<unit>", image, audio])
29        else:
30            contents.append(["<unit>", image, audio])
31    
32    return contents
33
34
35model = AutoModel.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True,
36    attn_implementation='sdpa', torch_dtype=torch.bfloat16)
37model = model.eval().cuda()
38tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-o-2_6', trust_remote_code=True)
39
40model.init_tts()
41
42# If you are using an older version of PyTorch, you might encounter this issue "weight_norm_fwd_first_dim_kernel" not implemented for 'BFloat16', Please convert the TTS to float32 type.
43# model.tts.float()
44
45# https://huggingface.co/openbmb/MiniCPM-o-2_6/blob/main/assets/Skiing.mp4
46video_path="assets/Skiing.mp4"
47sys_msg = model.get_sys_prompt(mode='omni', language='en')
48# if use voice clone prompt, please set ref_audio
49# ref_audio_path = '/path/to/ref_audio'
50# ref_audio, _ = librosa.load(ref_audio_path, sr=16000, mono=True)
51# sys_msg = model.get_sys_prompt(ref_audio=ref_audio, mode='omni', language='en')
52
53contents = get_video_chunk_content(video_path)
54msg = {"role":"user", "content": contents}
55msgs = [sys_msg, msg]
56
57# please set generate_audio=True and output_audio_path to save the tts result
58generate_audio = True
59output_audio_path = 'output.wav'
60
61res = model.chat(
62    msgs=msgs,
63    tokenizer=tokenizer,
64    sampling=True,
65    temperature=0.5,
66    max_new_tokens=4096,
67    omni_input=True, # please set omni_input=True when omni inference
68    use_tts_template=True,
69    generate_audio=generate_audio,
70    output_audio_path=output_audio_path,
71    max_slice_nums=1,
72    use_image_id=False,
73    return_dict=True
74)
75print(res)

注意：流式推理存在轻微的性能下降，因为音频编码并非全局的。

python
1# a new conversation need reset session first, it will reset the kv-cache
2model.reset_session()
3
4contents = get_video_chunk_content(video_path, flatten=False)
5session_id = '123'
6generate_audio = True
7
8# 1. prefill system prompt
9res = model.streaming_prefill(
10    session_id=session_id,
11    msgs=[sys_msg], 
12    tokenizer=tokenizer
13)
14
15# 2. prefill video/audio chunks
16for content in contents:
17    msgs = [{"role":"user", "content": content}]
18    res = model.streaming_prefill(
19        session_id=session_id,
20        msgs=msgs, 
21        tokenizer=tokenizer
22    )
23
24# 3. generate
25res = model.streaming_generate(
26    session_id=session_id,
27    tokenizer=tokenizer,
28    temperature=0.5,
29    generate_audio=generate_audio
30)
31
32audios = []
33text = ""
34
35if generate_audio:
36    for r in res:
37        audio_wav = r.audio_wav
38        sampling_rate = r.sampling_rate
39        txt = r.text
40
41        audios.append(audio_wav)
42        text += txt
43        
44    res = np.concatenate(audios)
45    sf.write("output.wav", res, samplerate=sampling_rate)
46    print("text:", text)
47    print("audio saved to output.wav")
48else:
49    for r in res:
50        text += r['text']
51    print("text:", text)

python
1# test.py    Need more than 16GB memory to run.
2import torch
3from PIL import Image
4from transformers import AutoModel, AutoTokenizer
5
6model = AutoModel.from_pretrained('openbmb/MiniCPM-Llama3-V-2_5', trust_remote_code=True, low_cpu_mem_usage=True)
7model = model.to(device='mps')
8
9tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-Llama3-V-2_5', trust_remote_code=True)
10model.eval()
11
12image = Image.open('./assets/hk_OCR.jpg').convert('RGB')
13question = 'Where is this photo taken?'
14msgs = [{'role': 'user', 'content': question}]
15
16answer, context, _ = model.chat(
17    image=image,
18    msgs=msgs,
19    context=None,
20    tokenizer=tokenizer,
21    sampling=True
22)
23print(answer)

运行:

shell
1PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py

2. 之前版本的 MiniCPM-V
   1. 安装 vLLM(>=0.5.4):

   shell
   1pip install vllm

   3. 安装 timm 库: （可选，MiniCPM-V 2.0需安装）

   shell
   1pip install timm=0.9.10

   4. 运行示例代码:（注意：如果使用本地路径的模型，请确保模型代码已更新到Hugging Face上的最新版)

   python
   1from transformers import AutoTokenizer
   2from PIL import Image
   3from vllm import LLM, SamplingParams
   4
   5MODEL_NAME = "openbmb/MiniCPM-V-2_6"
   6# MODEL_NAME = "openbmb/MiniCPM-O-2_6"
   7# Also available for previous models
   8# MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5"
   9# MODEL_NAME = "HwwwH/MiniCPM-V-2"
   10
   11image = Image.open("xxx.png").convert("RGB")
   12tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
   13llm = LLM(
   14    model=MODEL_NAME,
   15    trust_remote_code=True,
   16    gpu_memory_utilization=1,
   17    max_model_len=2048
   18)
   19
   20messages = [{
   21    "role":
   22    "user",
   23    "content":
   24    # Number of images
   25    "(<image>./</image>)" + \
   26    "\nWhat is the content of this image?" 
   27}]
   28prompt = tokenizer.apply_chat_template(
   29    messages,
   30    tokenize=False,
   31    add_generation_prompt=True
   32)
   33
   34# Single Inference
   35inputs = {
   36    "prompt": prompt,
   37    "multi_modal_data": {
   38        "image": image
   39        # Multi images, the number of images should be equal to that of `(<image>./</image>)`
   40        # "image": [image, image] 
   41    },
   42}
   43# Batch Inference
   44# inputs = [{
   45#     "prompt": prompt,
   46#     "multi_modal_data": {
   47#         "image": image
   48#     },
   49# } for _ in 2]
   50
   51
   52# 2.6
   53stop_tokens = ['<|im_end|>', '<|endoftext|>']
   54stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]
   55# 2.0
   56# stop_token_ids = [tokenizer.eos_id]
   57# 2.5
   58# stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
   59
   60sampling_params = SamplingParams(
   61    stop_token_ids=stop_token_ids, 
   62    use_beam_search=True,
   63    temperature=0, 
   64    best_of=3,
   65    max_tokens=1024
   66)
   67
   68outputs = llm.generate(inputs, sampling_params=sampling_params)
   69
   70print(outputs[0].outputs[0].text)

   4. 点击此处查看带视频推理和其他有关 vLLM 的信息。