Question about the release of the tokenizer training code

[Arispur-311]

Hi,
Thank you for your excellent work on this project. It's truly impressive. I am particularly interested in the tokenizer training part of your work. I noticed that you have plans to release the training code for the tokenizer. I was wondering if you could share any information about the estimated timeline for its release. No pressure at all, of course, just very keen to learn from your approach.
Thank you again for your contribution to the community!
Best regards

[roman-bachmann]

Hi @Arispur-311 ,

Thanks for your kind words! Indeed, we're thinking of releasing it too, but I don't have a concrete timeline right now. Stay tuned!

Best, Roman

[Arispur-311]

Thanks for your reply. I was wondering if you could point me to any resources or examples on how to train a Rectified Flow model? Any guidance would be greatly appreciated.

[roman-bachmann]

Of course! These are the main three resources I've come back to again and again:

• https://github.com/cloneofsimo/minRF/blob/main/rf.py: We base our rectified flow implementation on this minimal version. It's really clean and works great. You can also check in https://github.com/apple/ml-flextok/tree/main/flextok/flow_matching, where we put all of our rectified flow utils, and modified this minimal version to also allow for SD3's mode noise schedule.
• https://arxiv.org/abs/2403.03206: The SD3 paper is a great resource to learn the basics and has some great ablations on modeling choices around the noise schedule.
• Diffusers implementation of SD3: https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_flow_match_euler_discrete.py and relevant utils in https://github.com/huggingface/diffusers/blob/main/src/diffusers/training_utils.py. We also trained FlexTok models with this implementation and it works just as well. We just found minRF to be simpler since it doesn't need the diffusers dependency and is very easy to understand.

Besides that I can also recommend the following flow matching and diffusion resources:

• https://arxiv.org/abs/2412.06264 and code at https://github.com/facebookresearch/flow_matching
• https://arxiv.org/abs/2406.08929
• https://arxiv.org/abs/2403.18103
• https://diffusionflow.github.io/
• https://mlg.eng.cam.ac.uk/blog/2024/01/20/flow-matching.html

Hope this helps, and good luck!
Best, Roman

[Arispur-311]

Thanks a lot! I'll dive into the materials you shared. I believe this will be very helpful.
Best regards

[DTennant]

Hey,

Our codebase on a very similar idea have both training and testing codes, you are welcomed to check it out!
https://github.com/visual-gen/semanticist

[Arispur-311]

@DTennant thank you so much for sharing the resource !

[knightnemo]

Hi @roman-bachmann , I'm kindly following on the status of tokenizer training code:)