Custom nodes, imports fails. Moving mamba env to workspace?

[dkampien]

I installed some custom nodes but they keep getting import failed. I think this is due to the mambe env not being persistent in the workspace folder. You mentioned in the base image docks something about moving the env to workspace and that it's the default to true. But I can't find the env in workspace. Do I need to do something specific?

[robballantyne]

Which nodes? Can you provide a log?

Persisting the environment is only useful if you want the node dependencies to remain installed after the container has been removed. It's not usually necessary as nodes will have their dependencies installed on ComfyUI start.

I need to update the documentation because the default is now WORKSPACE_MAMBA_SYNC=false (also supports using old var WORKSPACE_SYNC)

If you set that variable to true the micromamba environments will be moved from /opt/micromamba to /workspace/environments/ but I don't think it'll fix whatever is causing the import failure

[dkampien]

Yes that is exactly what I want. For my environment to be persistent after the container has been removed.

I imagine installing many custom nodes and dependencies at ComfyUI start would take some time. By the way, Im using the template from Runpod.io and the image pull everytime I want to use it is very slow. About 40 mins for the 4.8GB. Any solution for this?

I will try and find a log and come back.

[dkampien]

Here's my log: The module not found errors aren't related to their dependencies?

** ComfyUI startup time: 2024-04-18 02:49:12.047262
[2024-04-18 02:49] ** Platform: Linux
[2024-04-18 02:49] ** Python version: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
[2024-04-18 02:49] ** Python executable: /opt/micromamba/envs/comfyui/bin/python
[2024-04-18 02:49] ** Log path: /workspace/ComfyUI/comfyui.log
[2024-04-18 02:49] 
Prestartup times for custom nodes:
[2024-04-18 02:49]    0.0 seconds: /workspace/ComfyUI/custom_nodes/rgthree-comfy
[2024-04-18 02:49]    0.3 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
[2024-04-18 02:49] 
Total VRAM 24217 MB, total RAM 257590 MB
[2024-04-18 02:49] xformers version: 0.0.25.post1
[2024-04-18 02:49] Set vram state to: NORMAL_VRAM
[2024-04-18 02:49] Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
[2024-04-18 02:49] VAE dtype: torch.bfloat16
[2024-04-18 02:49] Using xformers cross attention
[2024-04-18 02:49] Traceback (most recent call last):
  File "/workspace/ComfyUI/nodes.py", line 1864, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/workspace/ComfyUI/custom_nodes/ComfyUI-KJNodes/__init__.py", line 1, in <module>
    from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
  File "/workspace/ComfyUI/custom_nodes/ComfyUI-KJNodes/nodes.py", line 7, in <module>
    import matplotlib.pyplot as plt
ModuleNotFoundError: No module named 'matplotlib'

[2024-04-18 02:49] Cannot import /workspace/ComfyUI/custom_nodes/ComfyUI-KJNodes module for custom nodes: No module named 'matplotlib'
[2024-04-18 02:49] 
[2024-04-18 02:49]� [92m[rgthree] Loaded 37 exciting nodes.�[0m
[2024-04-18 02:49] �[92m[rgthree] Will use rgthree's optimized recursive execution.�[0m
[2024-04-18 02:49] 
[2024-04-18 02:49] [VideoHelperSuite] - �[0;33mWARNING�[0m - Failed to import imageio_ffmpeg
[2024-04-18 02:49] [AnimateDiffEvo] - �[0;31mERROR�[0m - No motion models found. Please download one and place in: ['/workspace/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/models', '/workspace/ComfyUI/models/animatediff_models']
[2024-04-18 02:49] Traceback (most recent call last):
  File "/workspace/ComfyUI/nodes.py", line 1864, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/workspace/ComfyUI/custom_nodes/ComfyUI_FizzNodes/__init__.py", line 57, in <module>
    from .ScheduledNodes import (
  File "/workspace/ComfyUI/custom_nodes/ComfyUI_FizzNodes/ScheduledNodes.py", line 4, in <module>
    import numexpr
ModuleNotFoundError: No module named 'numexpr'

[2024-04-18 02:49] Cannot import /workspace/ComfyUI/custom_nodes/ComfyUI_FizzNodes module for custom nodes: No module named 'numexpr'
[2024-04-18 02:49] Traceback (most recent call last):
  File "/workspace/ComfyUI/nodes.py", line 1864, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/workspace/ComfyUI/custom_nodes/was-node-suite-comfyui/__init__.py", line 1, in <module>
    from .WAS_Node_Suite import NODE_CLASS_MAPPINGS
  File "/workspace/ComfyUI/custom_nodes/was-node-suite-comfyui/WAS_Node_Suite.py", line 37, in <module>
    from numba import jit
ModuleNotFoundError: No module named 'numba'

[2024-04-18 02:49] Cannot import /workspace/ComfyUI/custom_nodes/was-node-suite-comfyui module for custom nodes: No module named 'numba'
[2024-04-18 02:49] 
�[36mEfficiency Nodes:�[0m Attempting to add Control Net options to the 'HiRes-Fix Script' Node (comfyui_controlnet_aux add-on)...�[92mSuccess!�[0m
[2024-04-18 02:49] �[93mEfficiency Nodes Warning:�[0m Failed to import python package 'simpleeval'; related nodes disabled.
[2024-04-18 02:49] 
[2024-04-18 02:49] 

Full error log from comfyui_controlnet_aux: 
Traceback (most recent call last):
  File "/workspace/ComfyUI/custom_nodes/comfyui_controlnet_aux/__init__.py", line 32, in load_nodes
    module = importlib.import_module(
  File "/opt/micromamba/envs/comfyui/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/workspace/ComfyUI/custom_nodes/comfyui_controlnet_aux/node_wrappers/dwpose.py", line 5, in <module>
    from controlnet_aux.dwpose import DwposeDetector, AnimalposeDetector
  File "/workspace/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/controlnet_aux/dwpose/__init__.py", line 15, in <module>
    from . import util
  File "/workspace/ComfyUI/custom_nodes/comfyui_controlnet_aux/src/controlnet_aux/dwpose/util.py", line 3, in <module>
    import matplotlib
ModuleNotFoundError: No module named 'matplotlib'


[2024-04-18 02:49] 
[2024-04-18 02:49] Traceback (most recent call last):
  File "/workspace/ComfyUI/nodes.py", line 1864, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/workspace/ComfyUI/custom_nodes/ComfyUI-Crystools/__init__.py", line 10, in <module>
    from .core import version, logger
  File "/workspace/ComfyUI/custom_nodes/ComfyUI-Crystools/core/__init__.py", line 5, in <module>
    from .common import *
  File "/workspace/ComfyUI/custom_nodes/ComfyUI-Crystools/core/common.py", line 4, in <module>
    from deepdiff import DeepDiff
ModuleNotFoundError: No module named 'deepdiff'

[2024-04-18 02:49] Cannot import /workspace/ComfyUI/custom_nodes/ComfyUI-Crystools module for custom nodes: No module named 'deepdiff'
[2024-04-18 02:49] ### Loading: ComfyUI-Manager (V2.19)
[2024-04-18 02:49] ### ComfyUI Revision: 2129 [719fb2c8] *DETACHED | Released on '2024-04-14'
[2024-04-18 02:49] 
Import times for custom nodes:
[2024-04-18 02:49]    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Image-Selector
[2024-04-18 02:49]    0.0 seconds: /workspace/ComfyUI/custom_nodes/websocket_image_save.py
[2024-04-18 02:49]    0.0 seconds: /workspace/ComfyUI/custom_nodes/mikey_nodes
[2024-04-18 02:49]    0.0 seconds: /workspace/ComfyUI/custom_nodes/comfyui_controlnet_aux
[2024-04-18 02:49]    0.0 seconds (IMPORT FAILED): /workspace/ComfyUI/custom_nodes/ComfyUI-KJNodes
[2024-04-18 02:49]    0.0 seconds (IMPORT FAILED): /workspace/ComfyUI/custom_nodes/was-node-suite-comfyui
[2024-04-18 02:49]    0.0 seconds (IMPORT FAILED): /workspace/ComfyUI/custom_nodes/ComfyUI_FizzNodes
[2024-04-18 02:49]    0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus
[2024-04-18 02:49]    0.1 seconds (IMPORT FAILED): /workspace/ComfyUI/custom_nodes/ComfyUI-Crystools
[2024-04-18 02:49]    0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts
[2024-04-18 02:49]    0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet
[2024-04-18 02:49]    0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
[2024-04-18 02:49]    0.2 seconds: /workspace/ComfyUI/custom_nodes/Derfuu_ComfyUI_ModdedNodes
[2024-04-18 02:49]    0.2 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved
[2024-04-18 02:49]    0.2 seconds: /workspace/ComfyUI/custom_nodes/rgthree-comfy
[2024-04-18 02:49]    0.2 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
[2024-04-18 02:49]    0.5 seconds: /workspace/ComfyUI/custom_nodes/efficiency-nodes-comfyui
[2024-04-18 02:49] 
[2024-04-18 02:49] Starting server

[2024-04-18 02:49] To see the GUI go to: http://127.0.0.1:18188
[2024-04-18 02:49] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[2024-04-18 02:49] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[2024-04-18 02:49] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[2024-04-18 02:49] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json

[dkampien]

@robballantyne

[jgal14]

@dkampien did you ever find a solution for this custom node installation failure?

[dougbtv]

Running into the same issue... for example:

2024-05-08T22:32:19.176400003Z Traceback (most recent call last):
2024-05-08T22:32:19.176401343Z   File "/workspace/ComfyUI/nodes.py", line 1864, in load_custom_node
2024-05-08T22:32:19.176402693Z     module_spec.loader.exec_module(module)
2024-05-08T22:32:19.176403943Z   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
2024-05-08T22:32:19.176405373Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2024-05-08T22:32:19.176406803Z   File "/workspace/ComfyUI/custom_nodes/comfy-image-saver/__init__.py", line 1, in <module>
2024-05-08T22:32:19.176408113Z     from .nodes import NODE_CLASS_MAPPINGS
2024-05-08T22:32:19.176409353Z   File "/workspace/ComfyUI/custom_nodes/comfy-image-saver/nodes.py", line 5, in <module>
2024-05-08T22:32:19.176410673Z     import piexif
2024-05-08T22:32:19.176412203Z ModuleNotFoundError: No module named 'piexif'
2024-05-08T22:32:19.176413473Z 
2024-05-08T22:32:19.176414693Z Cannot import /workspace/ComfyUI/custom_nodes/comfy-image-saver module for custom nodes: No module named 'piexif'
2024-05-08T22:32:19.176421263Z Traceback (most recent call last):
2024-05-08T22:32:19.176422563Z   File "/workspace/ComfyUI/nodes.py", line 1864, in load_custom_node
2024-05-08T22:32:19.176423823Z     module_spec.loader.exec_module(module)
2024-05-08T22:32:19.176425093Z   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
2024-05-08T22:32:19.176427583Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2024-05-08T22:32:19.176429573Z   File "/workspace/ComfyUI/custom_nodes/was-node-suite-comfyui/__init__.py", line 1, in <module>
2024-05-08T22:32:19.176431113Z     from .WAS_Node_Suite import NODE_CLASS_MAPPINGS
2024-05-08T22:32:19.176432423Z   File "/workspace/ComfyUI/custom_nodes/was-node-suite-comfyui/WAS_Node_Suite.py", line 37, in <module>
2024-05-08T22:32:19.176433683Z     from numba import jit
2024-05-08T22:32:19.176434953Z ModuleNotFoundError: No module named 'numba'

I can work around it temporarily by using the "install pip packages" from the manager UI node, but... currently looking into doing something to work around this in the meanwhile to make it persistent

[robballantyne]

Theres a bug with the default provisioning script. It should run an install for all declared nodes, but it won't unless AUTO_UPDATE is enabled. I'll fix this asap - I can also extend the behaviour to install requirements for all nodes which seems sensible.

Should be able to do this within 24 hours

[dougbtv]

I tried my hand at it in #73 -- feedback welcome, or, if you were thinking of something else feel free to close/replace it. (I'm still in the process of testing it...)

[dougbtv]

@dkampien , re:

By the way, Im using the template from Runpod.io and the image pull everytime I want to use it is very slow. About 40 mins for the 4.8GB. Any solution for this?

I think 4.8 gig is actually a "not too bad" of a image size for this kind of app (and in this case, might only be one layer? I just pulled the whole image locally and I got 13 gig for :latest), I've struggled to get a1111 and kohya builds down to that size, also was in the ~15 gig range.

In some cases, it might be runpod (and/or the image registry?), I also have times where the image pull takes forever randomly.

Also -- potentially you could log into the ghcr registry using runpod.io? e.g. with a github personal access token? I'm wondering if it's getting ratelimited by not logging in.

[benlau]

Theres a bug with the default provisioning script. It should run an install for all declared nodes, but it won't unless AUTO_UPDATE is enabled. I'll fix this asap - I can also extend the behaviour to install requirements for all nodes which seems sensible.

Should be able to do this within 24 hours

I am curious; is it fixed? Seem that it is still not reinstalling the packages . I need to write a provisioning script to make it works. (running in runpod)

[robballantyne]

Unfortunately not. There have been very few updates pushed actually but there will be a big update in the next few days hopefully that will alleviate this problem along with other small issues. I'm removing micromamba and moving everything to venv.

Remember, the provisioning script is just a bash script and it can be written in any style. You can just use a list of commands as you'd write them into a terminal if you'd like. The default script was only ever meant as an example, but it will definitely be fixed when my update is ready

[benlau]

got it. thx

By the way, out of curiosity, what is the reason to drop micromamba?

[robballantyne]

It causes more issues than it solves:

• No ROCm distribution of PyTorch
• Installing jupyter (nodejs dependency) breaks nano editor
• Pulls system libraries from untrusted source (forge)
• Pulled system libraries are occasionally broken

I like micromamba and outside of containers it's still my go-to, but my new solution is cleaner, leaner and less likely to give me a future headache - I hope!

[robballantyne]

@benlau New method to sync environment:

venv-sync comfyui

Run it manually whenever it's convenient to backup your container environment to a persistent disk