Add Holocine Structured Prompt & Shot Attention Support #1615
Conversation
…_embeds 参数以支持帧数推断
…_embeds 和 total_frames 参数以增强帧数推断能力；修复 WanVideoSampler 中的错误信息以提供更清晰的调试信息；调整 WanModel 中的 global_tokens 处理逻辑以支持自动模式。
…数推断能力；移除 total_frames 参数，改为通过 image_embeds 推断帧数；调整 global_tokens 处理逻辑以支持新的参数结构。
…o_or_number 参数的验证逻辑；确保其值大于 0，并支持比率和绝对值模式；调整 pooled_tokens 计算逻辑以适应新的参数结构。
Introduce per-shot lora. Should work even without Holocine.
- Better lora handling - Fix a couple of Attention Discrepencies (not fully aligned yet)
|
Loading transformer parameters to cuda:0: 100%|███████████████████████████████████| 1095/1095 [00:07<00:00, 140.43it/s] Prompt executed in 12.94 seconds after updated met error |
|
Oh hmmm. Try not to use torch compile for now. I'll look at if we can use compile at all. |
|
I've tried to recreate the scene with the painter from Holocines project page. with 241 (15sec) frames in 720x408 but encountered an error. I disabled torch. 23:29:22.643 [Info] Sent Comfy backend direct prompt requested to backend #0 (from user local) |
Would you share your workflow - I can't replicate... |
|
Dango233, thank you for your implementation, it is competent. WanVideo VRAM Management node not working. Can you fix it in the Kijai Wrapper? |
|
issue with lora , it was fixed on main branch , but for now toggle merge lora ON |
|
In latest version, the error is as follows: Can multiple different context windows adopt multiple rounds of inference? One-time inference consumes a lot of memory. |
thanks for the suggestion, but at least with the lora loader in my workflow it did not change the result still getting the error. meanwhile I've installed a fresh comfyui python 3.12.10 , cuda=12.8 , pytorch 2.8 |
|
@a3nima thanks for sharing your workflow, I trialed it and got the same error but changing the resolution to the one in the root holocine project fixed it for me at least (480x832). |
|
I sincerely hope that kijai can merge it into the main branch, because holoCine can truly narrow the gap with closed-source models. plz look at this: WanVideo2_2_I2V_00004.mp4 |
|
@kijai Can you review this? ❤️ |
|
If applying a lora I need to use "base_precision" fp32 for the WanVideo Model Loader , otherwise I get |
|
Since Kijai is swamped with other things, I did a small workaround to use HoloCine but also Bindweave and the other updates. I downloaded your repo and deleted the node names in the python node files except the loader (renamed it with HoloCine Loader), the sampler (also renamed it), and the four HoloCine nodes. In the init, I deleted most lines except for the model_loader_nodes and the sampler_nodes. Now, it works for HoloCine and everything else too. |
|
@railep can you put the working holocine repo up for trying? |
Here you go: https://github.com/railep/ComfyUI-WanVideoWrapper-Multishot |
|
I was too busy these weeks but i can tidy it up as soon as I have some time... |
|
@railep The memory usage is too high. I used to fit the example fully in VRAM with Dango's repo. The float/half issue is also present by default |
|
To those experiencing issues with the workflow make the following changes model loader (I question the default setting of the text encoder since its at fp8 not scaled) then if you have a larger video card like myself I have the rtx 6000 pro blackwell , set block swaps to 0 blocks to swap. I did 6 steps -> 832x480 @241 frames, the video generated according to the prompts but it seems a bit soft with grainy artifacts that could be sharper not sure if it's because i need to gen at higher resolutions , also i noticed the enhance node but it's tied to teacache which as far as I'm aware of is only compatible with wan 2.1 , unsure how to proceed. Lastly the motion of the subject almost seems too fast or unnatural such as taking a jacket off or a hat it just peels off like a fruit rollup. |
|
@Dango233 any chance we can have some multigpu support here , I modded my computer to house a rtx5090 along with a rtx 6000 pro blackwell 96gb vram , on a second note the first pass in the ksampler seems to be insanely long like 15 minutes , while the second sampler takes about 70-120 seconds which makes me wonder what the hell is the computer working on in that span of time I am working with the default number of scenes making each one 5 seconds FYI. |
Hey Dango, you think you can make this ready to become part of the wrapper? It would be so nice 🤩 |
|
@Dango233 is this project dead ? Also can you go over how your nodes allow holocine to work but a regular t2v workflow isn't that great ? Trying to wrap my head around the mechanics outside of lora issues applying to the entire scene which makes sense to split those out per segment |
|
It's not dead... It's just me too busy with my day job. I'll try catch up with the issues and sort them. |
Please..... |
that would be really awesome! |
|
@Dango233 I'm really hoping you'll be able to get back to this now that Christmas has passed. HoloCine looks like a much better implementation than any of the other options for long-video I've seen recently. |
One of the issues I experienced is that it seems if you are just using a usual workflow t2v with holocine and apply loras as you normally would it kind of ruins the output unless you are using the workflow and nodes dango233 made, at this point it seems easier to use svi 2.0 that works with wan 2.2 , amount of time and effort is rather high. On another note that way I originally installed this node seems to have screwed up my original wanvideowrapper node , not really sure if that is intentional or not. On a rtx 6000 pro blackwell 96gb vram I struggled to produce anything at 400 frames. Seems like a giant waste of time and hoping someone has some rectifying thoughts or opinions. |
Unfortunately we asian people does not have Chirstmas holiday LOL |
|
Turning to draft in order not to confuse people |
13 participants
Recreated from #1566 #1566, rebased to align with Upstream.
For those who has installed previously - Please delete the previous main and pull this instead.