能否支持glm-4v-9b模型

[liyuan1208]

Motivation

尝试适配glm-4v-9b模型（其实是13.9b，视觉部分有4.9B），发现glm4v里面对输入的position_ids做了特殊处理：
new_input_embeds.append(torch.cat(
(inputs_embeds[i, :boi_token_pos], images_features[i], inputs_embeds[i, eoi_token_pos + 1:])))
new_position_ids.append(torch.cat(
(position_ids[i, :boi_token_pos + 1], position_ids[i, boi_token_pos + 1].repeat(num_patches),
position_ids[i, eoi_token_pos:])
))
其将视觉特征部分的position_ids统一设定为同一个值，在计算RoPE时会用到。
turbomind引擎好像没有修改position_ids的接口。我们对glm-4v在我们的场景里全参数微调后效果是开源模型里面最好的，希望官方能支持glm-4v-9b模型

Related resources

GLM-4模型链接：
https://github.com/THUDM/GLM-4

Additional context

No response

[RunningLeon]

@liyuan1208 hi, glm-4v-9b will be supported by lmdeploy's pytorch engine. Will update once the pr is created.

[danxuan2022]

@liyuan1208 hi, glm-4v-9b will be supported by lmdeploy's pytorch engine. Will update once the pr is created.

大概要多久可以上线，期待😚

[danxuan2022]

@liyuan1208 hi, glm-4v-9b will be supported by lmdeploy's pytorch engine. Will update once the pr is created.

个人感觉哈，如果比vllm先支持会吸引一大波用户 。。。

[RunningLeon]

@danxuan2022 hi, you could try this PR #1947