Add support for LTX-Video model in ImageToVideo Pipeline#394
Add support for LTX-Video model in ImageToVideo Pipeline#394RUFFY-369 wants to merge 9 commits intolivepeer:mainfrom
Conversation
| seed = kwargs.pop("seed", None) | ||
| safety_check = kwargs.pop("safety_check", True) | ||
|
|
||
| if self.pipeline_name == "LTXImageToVideoPipeline": |
There was a problem hiding this comment.
Can we do something more generic that looks at the pipeline class args and del the kwargs keys if not present in the pipeline? If a heavy lift we can do in separate PR.
There was a problem hiding this comment.
Good point would love to make all pipelines more generic. We can do this in seperate PR if to heavy.
There was a problem hiding this comment.
@ad-astra-video That's a nice mention. Lets remove this stone age hard coded block and get something more generic.
Pushed the changes in the latest commit in this PR itself. 👍 Now it can work even when we make the whole pipeline generic based on task like i2i generic pipeline.
ad-astra-video
left a comment
ad-astra-video
left a comment
There was a problem hiding this comment.
PR looks good, have couple comments below and left suggestions/questions in the files:
- LTXVideo requires diffusers 0.32.0 right? Suggest we update to diffusers 0.32.1 with this PR.
- If fix the multipart.go suggested change I can build and test tomorrow
Co-authored-by: Brad | ad-astra <99882368+ad-astra-video@users.noreply.github.com>
Update: addressed all the suggested changes in the recent commits |
|
@RUFFY-369 I rebased this and removed the parts moved to livepeer/go-livepeer |
3 participants
What does this PR do?
This PR adds support for the
LTX-Videomodel in theImageToVideoPipeline. It introduces a mechanism to dynamically load either theLTXImageToVideoPipelineorStableVideoDiffusionPipeline, based on the providedmodelID. This change enhances the flexibility of theImageToVideoPipeline, allowing it to handle bothLTX-VideoandStableVideoDiffusionmodels and with addition ofLTXImageToVideoPipeline, the video generation can now be prompt text guided.After building the Docker image, the setup has been tested by running it locally on a Uvicorn server.
cc @rickstaa