DINOv2 + UAD algorithm for assistive robot project [BRIN 2025]

All credit to:

Yihe Tang¹, Wenlong Huang¹, Yingke Wang¹, Chengshu Li¹, Roy Yuan¹, Ruohan Zhang¹, Jiajun Wu¹, Li Fei-Fei¹

Environment Setup

• (Optioal) If you are using Omnigibson for rendering Behavior1K assets, or Blender for rendering Objaverse assets, please follow their installation guide respectively.
  • Note the rendering libraries may have version conflicts with the data pipeline / model training code, consider using a separate env in that case.
• (Optional) If you are using open-sourced sentence-transformers for language embedding, please follow their installation guide.
  • We recommend installing from source
• Create your conda environment and install torch

  conda create -n uad python=3.9
  conda activate uad
  pip install torch torchvision torchaudio
  

• Install unsup-affordance in the same conda env

  git clone https://github.com/TangYihe/unsup-affordance.git
  cd unsup-affordance
  pip install -r requirements.txt
  pip install -e .
  

Contributions

• Do some changes on the OPENAPI, now using GEMINI, as the api key is free.
• Optimization code,as trying to change input to live camera stream.

To Be Change

• Everything with !TO CHANGE comment, change it by yourself.