Could you help me to reproduce your code from scratch?

[woozch]

Hi. Thanks so much for sharing your impressive work.
I am trying to re-implement your recent work, but it seems a little difficult for me to re-implement your work.

Currently, I am re-implementing your GTA5->Cityscapes experiment based on drn26 network.

At first, I generated GTA5->Cityscapes images following the guide.

python3 train.py --dataroot='./datasets/data_semseg' --gpu_ids='0' --model='cycle_gan' --display_freq=100 --display_ncols=2 --save_latest_freq=2000 --save_epoch_freq=1 --save_by_iter --niter=5 --niter_decay=15 --batch_size=1 --soft_phase=True --norm='instance' --normGNET='instance' --netG='resnet_9blocks' --display_port=18099 --name='gta2city' --lambda_A=10.0 --lambda_B=10.0 --lambda_identity=5.0 --lambda_P=50000.0 --lr=1e-5 --lrG=5e-5

python3 export_images.py --dataroot='./datasets/data_semseg' --phase='train' --soft_phase=True --model='cycle_gan' --netG='resnet_9blocks' --norm='instance' --normGNET='instance' --name='gta2city' --load_iter=10000 --num_test=20 --results_dir='./results/gta2city'

I just simply modify export_images script to convert all gta5 images to cityscapes style(a.k.a. cyclegta5 dataset in your repo). Here I used last updated generater. The generated images are looking good overall, and it seems work correctly.

---

Second, I cloned recent CyCADA (https://github.com/jhoffman/cycada_release) repo. Then I merge your UNet_CPN.py and train_cpn_cycada.py script file to the main folder of CyCADA.
At train_fcn_adda_cpn.sh, It seems that it requires pretrained cycada model for initial weights, so I downloaded "drn26_cycada_cyclegta2cityscapes.pth file" from CyCADA repo.

After that, I trained CPN network with the generated cyclegta5 dataset with train_cpn_cycada.py, just simply run this script file.

python train_cpn_cycada.py

With the trained CPN network, I execute train_fcn_ada_cpn.sh file.

bash ./train_fcn_adda_cpn.sh

I evaluate the network of "net-itercurr.pth", but it gives bad performance (around 22% of mIoU).
Throughout the reproducing process, I didn't change any code except dataset directories and CPN permute functions that I mentioned on previous issue. I use python3.6.6, pytorch 1.6.0 with cuda 10.2, NVIDIA P40 GPU.

I also evaluate your trained network with evaluation script(eval_fcn.py) on CyCADA repo, and it gives good performance (42.6% mIoU). I think I miss some important steps, could you help me to reproduce it so I can verify your work?

[donglao]

Hi, I went through your steps. I think we are almost there. The image generation process seems correct. Could you please turn off CPN and train the segmentation network until the performance is saturated (and maybe verify if the performance is correct), and then turn on CPN? Another way to verify is to download the pre-trained weights from CyCADA, apply CPN, and see if there is a performance increase. The role of CPN is to clean up irregular segmentation output and align better to the edges in the image. If the initial prediction is incorrect, it could get stuck in local minima leading to bad segmentation performance. Thanks for pointing out this issue. During the paper submission, we had some explanations regarding CPN in the supplementary materials due to the space limitations so that it is currently not visible to the readers. I will include more details in the readme. Please let me know if you get some new results! Cheers, Lao

…

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[woozch]

Hi. Lao. Thanks for replying.
Following your suggestions, I trained the segmentation network(drn26) without CPN (use train_fcn_adda.sh script in CyCADA repo). The performance of the network was 38.27 mIoU.
However, after applying CPN, the performance degrades. I also tried to use the pre-trained
weights from CyCADA, then applied CPN, but the performance degrades again.
It seems that there may be something wrong on my training CPN process.

[donglao]

Hi, Could you please share with me your pre-trained model (without CyCADA 38.27 mIoU)? Let me train from it and check if there's anything wrong. Thanks, Lao

…

On Mon, Feb 15, 2021 at 7:25 AM Woong-Gi Chang ***@***.***> wrote: Hi. Lao. Thanks for replying. Following your suggestions, I trained the segmentation network(drn26) without CPN (use train_fcn_adda.sh script in CyCADA repo). The performance of the network was 38.27 mIoU. However, after applying CPN, the performance degrades. I also tried to use the pre-trained weights from CyCADA, then applied CPN, but the performance degrades again. It seems that there may be something wrong on training CPN process. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

-- Dong Lao Master/Ph.D. student Visual Computing Center King Abdullah University of Science and Technology (KAUST) Building 1, 2106-WS15 Thuwal, Kingdom of Saudi Arabia

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[woozch]

Hi Lao. Thanks for replying. :)

I trained the network using our company's computing resources.
Due to company security concerns, I cannot email the pre-trained weights to you. Also, company mail is only for receiving, so it seems difficult to respond with company mail.

I use pytorch 1.6, Tesla P40 gpu with cuda 10.1.

As you may have done this process(train without phase consistency), I'll also try to use the CyCADA released pretrained model to train CPN.
I will use below weights and translated image to check my CPN script.

I will follow the below process.

1. download pretrained weights and translated images
   pretrained weight: drn26-cyclegta5-iter115000.pth
   Translated Images: cyclegta.zip

2. train CPN_network by running train_cpn_cycada.py
   I'll use 100000.pth as CPN_cycada.pth for next step.

3. train segmentation network with trained CPN network on 2)

gpu=0
######################
# loss weight params #
######################
lr=1e-6
momentum=0.09
lambda_d=1
lambda_g=0.1

################
# train params #
################
max_iter=100000
crop=1024
snapshot=5000
batch=1

weight_share='weights_shared'
discrim='discrim_score'

########
# Data #
########
src='cyclegta5'
tgt='cityscapes'
datadir='./data/'

resdir="results_cycada/${src}_to_${tgt}/adda_sgd_cpn/${weight_share}_nolsgan_${discrim}"

# init with pre-trained cyclegta5 model
model='drn26'
baseiter=115000
#model='fcn8s'
#baseiter=100000

# base_model="base_models/drn26_cycada_cyclegta2cityscapes.pth"
base_model="base_models/drn26-cyclegta5-iter115000.pth"

outdir="${resdir}/${model}/lr${lr}_crop${crop}_ld${lambda_d}_lg${lambda_g}_momentum${momentum}"

# Run python script #
export PYTHONPATH=.

CUDA_VISIBLE_DEVICES=${gpu} python scripts/train_fcn_adda_cpn.py \
    ${outdir} \
    --dataset ${src} --dataset ${tgt} --datadir ${datadir} \
    --lr ${lr} --momentum ${momentum} --gpu 0 \
    --lambda_d ${lambda_d} --lambda_g ${lambda_g} \
    --weights_init ${base_model} --model ${model} \
    --"${weight_share}" --${discrim} --no_lsgan \
    --max_iter ${max_iter} --crop_size ${crop} --batch ${batch} \
--snapshot $snapshot

I'll let you know the experiment result.

[woozch]

The result of the above procedure are following.

<Evaluation Result(eval_fpn.py)>
road,sidewalk,building,wall,fence,pole,traffic light,traffic sign,vegetation,terrain,sky,person,rider,car,truck,bus,train,motorcycle,bicycle
49.740,31.484,43.071,6.175,2.834,16.248,8.737,3.053,83.659,34.853,77.992,56.782,0.383,64.167,8.363,5.933,0.008,6.894,0.592
26.366718461122538 52.59252430450583 67.60381523779724 32.47464192042296

Even though I used translated images from CyCADA, the evaluation result with CPN is worse than the one of CyCADA.
I am not sure where I am doing wrong. There may be bugs that I couldn't find.
Thanks for your help, though, I decided not to move forward on my personal project with this repo.

Sincerely