Skip to content

Commit 7a28ab3

Browse files
n1ck-guon1ck-guo
authored
cogvlm doc (#339)
Signed-off-by: n1ck-guo <[email protected]>
1 parent 182dd3f commit 7a28ab3

File tree

2 files changed

+128
-1
lines changed

2 files changed

+128
-1
lines changed

‎docs/cogvlm2-llama3-chat-19B_sym.md

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
2+
## Model Details
3+
4+
This model is an int4 model with group_size 128 and symmetric quantization of [THUDM/cogvlm2-llama3-chat-19B](https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B).
5+
## How To Use
6+
### INT4 Inference
7+
```python
8+
import torch
9+
from PIL import Image
10+
from auto_round import AutoRoundConfig ##must import for auto-round format
11+
from transformers import AutoModelForCausalLM, AutoTokenizer
12+
import requests
13+
14+
MODEL_PATH = "Intel/cogvlm2-llama3-chat-19B-inc-private"
15+
DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
16+
17+
tokenizer = AutoTokenizer.from_pretrained(
18+
MODEL_PATH,
19+
trust_remote_code=True
20+
)
21+
model = AutoModelForCausalLM.from_pretrained(
22+
MODEL_PATH,
23+
torch_dtype="auto",
24+
trust_remote_code=True,
25+
device_map=DEVICE
26+
).to(DEVICE).eval()
27+
28+
image_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
29+
content = "Describe this image."
30+
31+
text_only_template = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {} ASSISTANT:"
32+
query = text_only_template.format(content)
33+
34+
image = Image.open(requests.get(image_url, stream=True).raw)
35+
input_by_model = model.build_conversation_input_ids(
36+
tokenizer,
37+
query=query,
38+
images=[image],
39+
template_version='chat'
40+
)
41+
inputs = {
42+
'input_ids': input_by_model['input_ids'].unsqueeze(0).to(DEVICE),
43+
'token_type_ids': input_by_model['token_type_ids'].unsqueeze(0).to(DEVICE),
44+
'attention_mask': input_by_model['attention_mask'].unsqueeze(0).to(DEVICE),
45+
'images': [[input_by_model['images'][0].to(DEVICE).to(model.dtype)]] if image is not None else None,
46+
}
47+
gen_kwargs = {
48+
"max_new_tokens": 2048,
49+
"pad_token_id": 128002,
50+
}
51+
52+
with torch.no_grad():
53+
outputs = model.generate(**inputs, **gen_kwargs)
54+
outputs = outputs[:, inputs['input_ids'].shape[1]:]
55+
response = tokenizer.decode(outputs[0])
56+
response = response.split("<|end_of_text|>")[0]
57+
print(response)
58+
##INT4:
59+
## The image depicts a serene beach scene during what appears to be the golden hour, just before sunset. A woman is seated on the sandy shore, facing the vast expanse of the ocean. The waves are gently crashing on the beach, and the sky is painted with hues of orange and blue. The woman seems to be in a relaxed posture, possibly enjoying the tranquility of the moment. Beside her is a large, golden-colored dog, possibly a Labrador, wearing a colorful harness. The dog is sitting upright, looking at the woman, and they seem to share a bond of trust and affection. The overall mood of the image is peaceful, reflective, and heartwarming.
60+
61+
##BF16:
62+
## The image showcases a serene beach setting during what appears to be either sunrise or sunset. In the foreground, a woman sits on the sandy beach, dressed in casual attire, including a checkered shirt and jeans. She is engaged in a moment of connection with a golden retriever dog, which is seated beside her. The dog wears a colorful harness and is looking up at the woman, possibly in anticipation of a treat or a playful gesture. The vast expanse of the ocean can be seen in the background, with gentle waves crashing onto the shore. The sky is clear, and the warm hues of the setting or rising sun cast a soft glow over the scene, creating a tranquil and heartwarming atmosphere.
63+
64+
image_url = "http://images.cocodataset.org/train2017/000000411975.jpg"
65+
content = "图片中的棒球场上有多少人？"
66+
##INT4:
67+
## In the image provided, there are four individuals on the baseball field. There are two children in the foreground, presumably engaged in some playful activity or perhaps participating in a game. One child is bending over, possibly picking something up off the ground, while the other stands beside them. Additionally, there is an adult standing further back, holding a microphone, suggesting they might be an announcer or commentator for the game. Lastly, there is another adult, seen from the back, sitting in the stands. So, in total, there are four people on the baseball field.
68+
69+
##BF16:
70+
## In the image provided, there are five people visible on the baseball field.
71+
72+
image_url = "https://intelcorp.scene7.com/is/image/intelcorp/processor-overview-framed-badge:1920-1080?wid=480&hei=270"
73+
content = "这张图片代表哪家公司？"
74+
##INT4:
75+
## The image represents the company Intel. The logo in the image is the Intel Inside logo, which is commonly used by Intel to signify the presence of their processors in various electronic devices.
76+
77+
##BF16:
78+
## The image represents the company Intel.
79+
```
80+
81+
82+
83+
## Evaluation the model
84+
pip3 install lmms_eval
85+
```bash
86+
auto-round-mllm --lmms --model Intel/cogvlm2-llama3-chat-19B-inc-private --tasks pope,textvqa_val,scienceqa,mmbench_en --output_dir "./eval_result" --device cuda:0
87+
```
88+
89+
### Generate the model
90+
Here is the sample command to reproduce the model.
91+
```bash
92+
pip install auto_round
93+
auto-round-mllm
94+
--model THUDM/cogvlm2-llama3-chat-19B \
95+
--device 0 \
96+
--group_size 128 \
97+
--bits 4 \
98+
--iters 1000 \
99+
--nsample 512 \
100+
--seqlen 2048 \
101+
--format 'auto_gptq,auto_round' \
102+
--output_dir "./tmp_autoround"
103+
```
104+
105+
## Ethical Considerations and Limitations
106+
107+
The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
108+
109+
Therefore, before deploying any applications of the model, developers should perform safety testing.
110+
111+
## Caveats and Recommendations
112+
113+
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
114+
115+
Here are a couple of useful links to learn more about Intel's AI software:
116+
117+
- Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
118+
119+
## Disclaimer
120+
121+
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
122+
123+
## Cite
124+
125+
@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }
126+
127+
[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)

‎docs/llava-v1.5-7b_sym.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ messages = [{"from": "human", "value": "How many people and animals are there in
8282
## Evaluation the model
8383
pip3 install lmms_eval
8484
```bash
85-
auto_round --lmms --model llava-v1.5-7b-inc-private --tasks pope,textvqa_val,scienceqa,mmbench_en --output_dir "./eval_result" --device cuda:0
85+
auto-round-mllm --lmms --model Intel/llava-v1.5-7b-inc-private --tasks pope,textvqa_val,scienceqa,mmbench_en --output_dir "./eval_result" --device cuda:0
8686
```
8787
|Metric |16bits|Pile Calib INT4 | Llava Calib INT4 |
8888
|:-------------------|:------|:------|:--------------|

0 commit comments

Comments
 (0)