Exploring Response Uncertainty in MLLMs : An Empirical Evaluation Under Misleading Scenarios

  

---

📌 Key Features

• 🔍 Evaluation of MLLMs under misleading inputs
• 📊 Uncertainty quantification metrics
• 🎯 Explicit & Implicit misleading experiments
• 🔬 Comprehensive model comparison
• 📝 Reproducible results and visualization

---

⚙️ Environment Installation

Before running the code, set up the required environments for: glm , llava, MiniCPM-V, mmstar.

📥 Installation Steps:

1. Navigate to the env folder.
2. Download and install the corresponding .yml environment files:

   conda env create -f env/glm.yml
   conda env create -f env/llava.yml
   conda env create -f env/MiniCPM-V.yml
   conda env create -f env/mmstar.yml

3. Activate the required environment:

   conda activate <ENV_NAME>

---

📂 Data Preparation

Download the Multimodal Uncertainty Benchmark (MUB) dataset here.
Extract and place the downloaded images into the extract_img_all folder.

Evaluated Open-source and Close-source Models:
MiniCPM-v-v2; Phi-3-vision; YiVL-6b; Qwen-VL-Chat; Deepseek-VL-7b-Chat; LLaVA-NeXT-7b-vicuna; MiniCPM-Llama3-v2.5; GLM4V-9Bchat; CogVLM-chat; InternVL-Chat-V1-5; LLaVA-Next-34b; Yi-VL-34b; GPT-4o; Gemini-Pro; Claude3-OpusV; Glm-4V

---

🔍 Explicit Misleading Experiment

bash MR_test.sh

---

🔄 Implicit Misleading Experiment

🛠️ Step 1: Generate Implicit Misleading Samples

1. Open implicit/misleading_generate/my_tool.py.
2. Fill in your API Key.
3. Run:

   bash implicit/misleading_generate/mislead_generate.sh

📊 Step 2: Run the Implicit Misleading Test

Use the generated data in implicit/mislead_output:

bash implicit/Implicit_MR_test/implicit_MR_test.sh

---

📈 Results & Visualization

Results are saved in:

• 📁 result/test_dataset_6
  • .jsonl → Detailed outputs
  • .txt → Model's Misleading Rate (MR)

📊 Convert Results to Table Format

1. Open extract2table/extract2table.py
2. Modify txt_folder_paths as needed.
3. Run:

   python extract2table/extract2table.py

4. The formatted table is saved in: 📁 extract2table/Tables/

---

📚 Citation

If you use this work, please cite:

@article{yourpaper2024,
  title={Exploring Response Uncertainty in MLLMs: An Empirical Evaluation Under Misleading Scenarios},
  author={Authors' Names},
  journal={arXiv preprint arXiv:2411.02708},
  year={2024}
}

---

📬 Contact

For any issues, please open a GitHub issue or reach out via email: yunkaidang1@gmail.com