Best way to train a lora and use with controlnet - Z Image

[gabriele-dominici]

Hi,

I was wondering which is the best way to train a Lora on Z Image in the current repo for using with Controlnet Tile.

The situation is the following:

• I have several images (slightly out of distribution from the training of the model - very specific), both at low resolution and high resolution
• The goal is going from a low resolution to a high resolution version of those images (I thought of using Z Image Controlnet Tile), where very little details matter

What is the best configuration both for training and inference in such a scenario?
I don't mean the exact hyperparam, but more from high level pov.

Some of the questions that I have are the following, but feel free to add additional crucial details that I might miss:

• Should I train the Lora on Z Image or Z Image Turbo? (The controlnet has been trained on top of Z Image Turbo afaik)
  I know that Z Image Turbo should be more difficult to train (as it is a distilled model), but then there would be a mismatch with the ControlNet
• Should I train a Lora on Z Image (or Z Image Turbo) and apply that on controlnet, or should I train a new controlnet (I think it might be a bit too expensive)? Should I do something different?
• Is there any specific suggested procedure to preprocess the images to train the tile controlnet?
• Assuming I did one of those trainings, is there any parameter I should modify (or care a lot) during inference? (e.g., If I train a lora on Turbo, the control guidance and number of steps should remain the same (0 and 8, respectively)?)

I already had a few tests, but I don't want to influence your opinion (as I might made some mistakes in the middle)

[bubbliiiing]

For image preprocessing when training a tile ControlNet, I suggest you do it based on the actual input conditions.

If you're planning to use it in the ControlNet way, ControlNet itself only contains some parameters and doesn't change the base model; I find training a LoRA a bit odd for this purpose, so I suggest training the Control part of ControlNet directly.

When using it after training is complete, there may be a lack of acceleration capability, requiring the setting of CFG and a larger number of steps.