spandrel model

[rahmirtrt]

How can I use the big-lama.pt model in dgenerate's spandrel? Could you provide a command line?

dgenerate --sub-command image-process my-image-seed.png my-mask-image.png --output my-image-seedout.png --align 1 --processors "upscaler;model=models/big-lama.pt"

I couldn't get it to work this way.

https://github.com/Sanster/models/releases/download/add_big_lama/big-lama.pt

[Teriks]

I have not implemented an inpainting processor for using inpainting arches from spandrel, though might be worth. spandrel supported arches are only usable through the upscaler image processor plugin, which is just set up for image upscaling and not inpainting.

[Teriks]

Went ahead and added this capability through a new inpaint processor: fa3a7ba

This processor supports inpainting architectures implemented in spandrel

Reinstall using the network installer (branch version_5.0.0) or through a manual install, and it should be available to use.

This new processor has a similar interface to the cpu based patchmatch processor.

# (bash syntax / line continuations)

# Subject image as primary input
# ====================

# Inpaint using spandrel + LaMa

dgenerate --sub-command image-process examples/media/dog-on-bench.png \
--output dog-bench-lama-inpaint1.png \
--processors "inpaint;model=https://github.com/Sanster/models/releases/download/add_big_lama/big-lama.pt;mask=examples/media/dog-on-bench-mask.png"


# Inpaint using spandrel + MAT

dgenerate --sub-command image-process examples/media/dog-on-bench.png \
--output dog-bench-mat-inpaint1.png \
--processors "inpaint;model=https://github.com/Sanster/models/releases/download/add_mat/Places_512_FullData_G.pth;mask=examples/media/dog-on-bench-mask.png"


# Reverse (mask as primary input)
# ====================

# Inpaint using spandrel + LaMa

dgenerate --sub-command image-process examples/media/dog-on-bench-mask.png \
--output dog-bench-lama-inpaint2.png \
--processors "inpaint;model=https://github.com/Sanster/models/releases/download/add_big_lama/big-lama.pt;image=examples/media/dog-on-bench.png"


# Inpaint using spandrel + MAT

dgenerate --sub-command image-process examples/media/dog-on-bench-mask.png \
--output dog-bench-mat-inpaint2.png \
--processors "inpaint;model=https://github.com/Sanster/models/releases/download/add_mat/Places_512_FullData_G.pth;image=examples/media/dog-on-bench.png"

Also see script example: https://github.com/Teriks/dgenerate/blob/version_5.0.0/examples/image_process/inpaint-config.dgen

dgenerate --image-processor-help inpaint

inpaint:
    arguments:
        model: str
        mask: str | None = None
        mask-processors: str | None = None
        image: str | None = None
        image-processors: str | None = None
        dtype: str = "float32"
        pre-resize: bool = False
        device: str | None = None
        output-file: Optional[str] = None
        output-overwrite: bool = False
        model-offload: bool = False

    Inpaint an image using inpainting model architectures supported by Spandrel (AI-based inpainting).

    This processor uses inpainting models such as LaMa, MAT, and other architectures supported by Spandrel for
    advanced AI-based inpainting to fill in missing or masked areas in images.

    This processor requires either a mask or subject image to be provided via the "mask" or "image" arguments.
    These arguments are mutually exclusive.

    When using the "mask" argument, the incoming image is considered the subject image to be inpainted, and
    the "mask" argument provides a grayscale mask where white pixels (255) indicate areas to inpaint and black
    pixels (0) indicate areas to preserve.

    When using the "image" argument, the incoming image is considered the mask, and the "image" argument
    provides the subject image to be inpainted. The incoming mask image should be a grayscale image where
    white pixels (255) indicate areas to inpaint and black pixels (0) indicate areas to preserve.

    The "mask" or "image" argument should point to a file path on disk or a URL that can be downloaded. Both
    local files and remote URLs are supported. The mask or image will be resized to match the dimensions of
    the corresponding target image if they are not the same size.

    The "mask-processors" argument allows you to pre-process the "mask" argument with an arbitrary image
    processor chain, for example: invert, gaussian-blur, etc. This arguments value must be quoted (single or
    double string quotes) if you intend to supply arguments to the processors in the chain. The pixel
    alignment of this processor chain defaults to 1, meaning no forced alignment will occur, you can force
    alignment using the "resize" image processor if desired.

    The "image-processors" argument allows you to pre-process the "image" argument with an arbitrary image
    processor chain, for example: invert, gaussian-blur, etc. This arguments value must be quoted (single or
    double string quotes) if you intend to supply arguments to the processors in the chain. The pixel
    alignment of this processor chain defaults to 1, meaning no forced alignment will occur, you can force
    alignment using the "resize" image processor if desired.

    The "model" argument specifies the path to an inpainting model file supported by Spandrel, such as LaMa,
    MAT, or other compatible architectures. This can be a local file path or a URL that can be downloaded.

    The "dtype" argument can be used to specify the datatype to use for the model in memory, it can be either
    "float32" or "float16". Using "float16" will result in a smaller memory footprint if supported by the
    model. If a model doesn't support FP16, it will automatically fall back to FP32 with a warning message.

     The "pre-resize" argument determines if the processing occurs before or after dgenerate resizes the
    image. This defaults to False, meaning the image is processed after dgenerate is done resizing it.

    The "device" argument can be used to set the device the inpainting model and
    mask-processors/image-processors will run on, for example: cpu, cuda, cuda:1.

    The "output-file" argument can be used to set the output path for a processor debug image, this will save
    the processed image to a path of your choosing.

    The "output-overwrite" argument can be used to enable overwrite for a processor debug image. If this is
    not enabled, new images written by the processor while it is being used will be written with a numbered
    suffix instead of being overwritten.

    The "model-offload" argument can be used to enable cpu model offloading for the inpainting model and
    mask-processors/image-processors. If this is disabled, any torch tensors or modules placed on the GPU will
    remain there until the processor is done being used, instead of them being moved back to the CPU after
    each invocation. Enabling this may help save VRAM when using multiple processors that make use of the GPU.

==============================================================================================================

[rahmirtrt]

Can you add it to dgenerate lite version as well?
Inpaint using spandrel + LaMa and Inpaint using spandrel + mat

[Teriks]

The code base for 5.0.0 has diverged very significantly from that branch so I was not planning on maintaining it