About the data

[LHRYANG]

Great work!

I want to know when will you release the data and the details of the data, e..g, sources, captioner.

Thanks a lot!

[wusize]

We are still seeking approval to upload data. They will be coming soon!

[stdKonjac]

Hi thanks for the great work. I wonder if current training codes and docs are ready for finetuning on customized datasets？

[wusize]

coming soon

[stdKonjac]

coming soon

Sorry for my ambiguous comments. I mean, if I can load data myself, are train_ddp.sh/train.py the full training scripts used in the paper?

[stdKonjac]

coming soon

Sorry for my ambiguous comments. I mean, if I can load data myself, are train_ddp.sh/train.py the full training scripts used in the paper?

BTW, how to get the reported GenEval results in the paper? I run the evaluation code provided in the docs for OpenUni-L-1024, using the original GenEval prompts, and only got a score of 72.9. The detailed scores are as follows:

Summary
=======
Total images: 2212
Total prompts: 553
% correct images: 72.20%
% correct prompts: 92.04%

Task breakdown
==============
single_object    = 98.75% (316 / 320)
two_object       = 87.37% (346 / 396)
counting         = 69.69% (223 / 320)
colors           = 59.84% (225 / 376)
position         = 80.25% (321 / 400)
color_attr       = 41.50% (166 / 400)

Overall score (avg. over tasks): 0.72900

Could you please share your GenEval prompts in case I missed something like prompt-rewriting? Thank you!

[wusize]

color_attr = 41.50% (166 / 400)

Maybe train_ddp.sh/train.py works but I cannot guarantee since I am still organizing training scripts.

For evaluation, can I have a look at your script to run the inference?

[stdKonjac]

color_attr = 41.50% (166 / 400)

Maybe train_ddp.sh/train.py works but I cannot guarantee since I am still organizing training scripts.

For evaluation, can I have a look at your script to run the inference?

For running inference, I use the following scripts:

bash ACCRUN ./scripts/evaluation/gen_eval.py ./configs/models/openuni_l_internvl3_2b_sana_1_6b_1024_hf.py \
  --checkpoint ./cache/openuni_l_internvl3_2b_sana_1_6b_1024_hf_blip3o60k.pth  \
  --batch_size 8  \
  --output ./cache/inference/gen_eval/openuni_l_internvl3_2b_sana_1_6b_1024_hf_blip3o60k \
  --height 1024 \
  --width 1024 \
  --seed 42 \
  --data ./geneval/prompts/evaluation_metadata.jsonl

where I use ACCRUN to set the environments on my machine. The core eval scripts are not modified.

The evaluation scripts are as follows:

EVAL_MODEL=openuni_l_internvl3_2b_sana_1_6b_1024_hf_blip3o60k
EVAL_TASK=gen_eval
IMAGE_FOLDER=./cache/inference/$EVAL_TASK/$EVAL_MODEL
RESULTS_FOLDER=./geneval/results/${EVAL_MODEL}_${EVAL_TASK}
MODEL_PATH=./geneval/mask2former

python3 ./geneval/evaluation/evaluate_images.py \
    $IMAGE_FOLDER \
    --outfile $RESULTS_FOLDER/results.jsonl \
    --model-path $MODEL_PATH

python3 ./geneval/evaluation/summary_scores.py \
    $RESULTS_FOLDER/results.jsonl

which follows the official repo.

[stdKonjac]

I indeed find some bad cases on GenEval. For No.552 prompt "a photo of a blue pizza and a yellow baseball glove", the generated images are weird.

[wusize]

This is my result, using the same config and checkpoint

[stdKonjac]

Yes, I do get the normal images for the first and second generated ones. But the third and fourth ones are weird.

[stdKonjac]

It seems this happens for all samples. The 1st and 2nd ones are normal, while the 3rd the 4th ones are weird.

[wusize]

I looked at all the four images and they are all normal ones. Can I know the versions of your packages? Including transformers, flash attention and diffusers.

[stdKonjac]

My transformers version is transformers-4.45.2, flash attention is 2.7.4.post1, and diffusers is 0.33.1. Anyway, I will re-evaluate the results using only the first two images.

[wusize]

My transformers version is transformers-4.45.2, flash attention is 2.7.4.post1, and diffusers is 0.33.1. Anyway, I will re-evaluate the results using only the first two images.

Can you add my wechat or open another issue for this specific problem?

[stdKonjac]

My transformers version is transformers-4.45.2, flash attention is 2.7.4.post1, and diffusers is 0.33.1. Anyway, I will re-evaluate the results using only the first two images.

Can you add my wechat or open another issue for this specific problem?

Sure. I will send you my WeChat account to your NTU email, thank you for your patience :)