Update README.md

README.md

@@ -12,4 +12,107 @@ tags:
 - Multimodal
 - Embedding
 pipeline_tag: image-text-to-text
----
+---
+
+# Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
+<a href="https://github.com/GaryGuTC">Tiancheng Gu*</a>,</span>
+<a href="https://kaicheng-yang0828.github.io">Kaicheng Yang*</a>,</span>
+Ziyong Feng,</span>
+Xingjun Wang,</span>
+Yanzhao Zhang,</span>
+Dingkun Long,</span>
+Yingda Chen,</span>
+<a href="https://weidong-tom-cai.github.io/">Weidong Cai</a>,</span>
+<a href="https://jiankangdeng.github.io">Jiankang Deng</a></span>
+
+[🏡 Project Page](https://garygutc.github.io/UniME) |  [📄 Paper]() | [💻 Github](https://github.com/deepglint/UniME)
+
+
+<p align="center">
+    <img src="figures/fig1.png" width="85%" height="85">
+</p>
+
+
+## 🎺 News
+- [2025/04/24]: ✨We release the evaluate and demo code.
+- [2025/04/24]: ✨The paper of UniME is submitted to arxiv.
+- [2025/04/22]: ✨We release the model weight of UniME in [🤗 Huggingface](https://huggingface.co/collections/DeepGlint-AI/unime-6805fa16ab0071a96bef29d2)
+
+## 💡 Highlights
+To enhance the MLLM's embedding capability, we propose textual discriminative knowledge distillation. The training process involves decoupling the MLLM's LLM component and processing text with the prompt "Summarize the above sentences in one word.", followed by aligning the student (MLLM) and teacher (NV-Embed V2) embeddings via KL divergence on batch-wise similarity distributions. **Notably, only the LLM component is fine-tuned during this process, while all other parameters remain frozen**. 
+
+<p align="center">
+    <img src="figures/fig2.png" width="85%" >
+</p>
+
+After that, we propose hard negative enhanced instruction tuning enhances multimodal systems by improving visual sensitivity, strengthening cross-modal alignment, and boosting instruction-following capabilities. At its core are two key innovations: a false negative filtering mechanism using a similarity threshold to eliminate misleading samples, and an automatic hard negative sampling strategy that selects top-k similar but non-matching examples to increase training difficulty. 
+<p align="center">
+    <img src="figures/fig3.png" width="85%" >
+</p>
+
+
+## 🧭 Quick Start
+```bash
+git clone https://github.com/deepglint/UniME.git
+cd UniME
+conda create -n uniME python=3.10 -y
+conda activate uniME
+pip install -r requirements.txt
+```
+
+```python
+import torch
+from PIL import Image
+from torch.nn import functional as F
+from transformers import AutoProcessor, AutoModelForCausalLM
+
+base_model_path="DeepGlint-AI/UniME-Phi3.5-V-4.2B"
+img_prompt = '<|user|>\n<|image_1|>\nSummary above image in one word: <|end|>\n<|assistant|>\n'
+text_prompt = '<|user|>\n<sent>\nSummary above sentence in one word: <|end|>\n<|assistant|>\n'
+
+text = "A man is crossing the street with a red car parked nearby."
+image_path = "figures/demo.png"
+input_texts = text_prompt.replace('<sent>', text)
+input_image_prompt = img_prompt
+input_image = [Image.open(image_path)]
+
+transform = AutoProcessor.from_pretrained(base_model_path, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(base_model_path,device_map="cuda", trust_remote_code=True，torch_dtype=torch.float16, _attn_implementation='flash_attention_2')
+transform.tokenizer.padding_side = "left"
+transform.tokenizer.padding = True
+
+inputs_text = transform(text=input_texts,
+                    images=None,
+                    return_tensors="pt", 
+                    padding=True)
+for key in inputs_text: inputs_text[key] = inputs_text[key].to("cuda")
+inputs_image = transform(text=input_image_prompt,
+                    images=input_image, 
+                    return_tensors="pt", 
+                    padding=True).to("cuda")
+
+with torch.no_grad():
+  emb_text = model(**inputs_text, output_hidden_states=True, return_dict=True).hidden_states[-1][:, -1, :]
+  emb_image = model(**inputs_image, output_hidden_states=True, return_dict=True).hidden_states[-1][:, -1, :]
+  emb_text = F.normalize(emb_text, dim=-1)
+  emb_image = F.normalize(emb_image, dim=-1)
+  Score = emb_image @ emb_text.T
+print("Score: ", Score)
+```
+
+## 🔢 Results
+### Diverse Retrieval
+<p align="center">
+    <img src="figures/res1.png" width="85%" >
+</p>
+
+### MMEB
+<p align="center">
+    <img src="figures/res2.png" width="85%" >
+</p>
+
+## 📖 Citation
+If you find this repository useful, please use the following BibTeX entry for citation.
+```latex
+Coming soon
+```