exception when trying to generate data

[weimzh]

tried running:
sh script/data_generation_coco_instance.sh

and got this error:

Traceback (most recent call last):
File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/whistler/DatasetDM/tools/parallel_generate_Instance_COCO_AnyClass.py", line 284, in sub_processor
scheduler = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", use_auth_token=MY_TOKEN).to(device).scheduler
File "/whistler/DatasetDM/model/diffusers/pipeline_utils.py", line 377, in from_pretrained
load_method = getattr(class_obj, load_method_name)
TypeError: getattr(): attribute name must be string

[weimzh]

I replaced the SD1.4 version with the one downloaded form here:
https://drive.google.com/file/d/12lrOexljsyvFB30-ltbYXnIpQ8oP4lrW/view?usp=sharing

however the error now changed:

Traceback (most recent call last):
File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/whistler/DatasetDM/tools/parallel_generate_Instance_COCO_AnyClass.py", line 414, in sub_processor
images_here, x_t = ptp_utils.text2image(unet,vae,tokenizer,text_encoder,scheduler, prompts, controller, num_inference_steps=NUM_DIFFUSION_STEPS, guidance_scale=5, generator=g_cpu, low_resource=LOW_RESOURCE, Train=False)
File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/whistler/DatasetDM/ptp_utils.py", line 214, in text2image
image = latent2image(vae, latents)
File "/whistler/DatasetDM/ptp_utils.py", line 90, in latent2image
image = vae.decode(latents)['sample']
File "/whistler/DatasetDM/model/diffusers/models/vae.py", line 552, in decode
dec = self.decoder(z)
File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/whistler/DatasetDM/model/diffusers/models/vae.py", line 202, in forward
sample = up_block(sample)
File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/whistler/DatasetDM/model/diffusers/models/unet_blocks.py", line 1212, in forward
hidden_states = resnet(hidden_states, temb=None)
File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/whistler/DatasetDM/model/diffusers/models/resnet.py", line 337, in forward
hidden_states = self.norm1(hidden_states.float()).type(hidden_states.dtype)
AttributeError: 'tuple' object has no attribute 'float'

[weijiawu]

Sorry, it's difficult to determine where the problem occurred based on the current information. Could you try printing out the hidden_states of type tuple for me to see? Perhaps we can diagnose the issue from there.

[weimzh]

thanks for reply

I've printed the type of hidden_states, and normally it's torch.Tensor, eventuallly it became 'class tuple' and the error occures:

<class 'torch.Tensor'>
<class 'torch.Tensor'>
<class 'tuple'>
Process SpawnProcess-2:
Traceback (most recent call last):
  File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/whistler/DatasetDM/tools/parallel_generate_Instance_COCO_AnyClass.py", line 414, in sub_processor
    images_here, x_t = ptp_utils.text2image(unet,vae,tokenizer,text_encoder,scheduler, prompts, controller,  num_inference_steps=NUM_DIFFUSION_STEPS, guidance_scale=5, generator=g_cpu, low_resource=LOW_RESOURCE, Train=False)
  File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/whistler/DatasetDM/ptp_utils.py", line 214, in text2image
    image = latent2image(vae, latents)
  File "/whistler/DatasetDM/ptp_utils.py", line 90, in latent2image
    image = vae.decode(latents)['sample']
  File "/whistler/DatasetDM/model/diffusers/models/vae.py", line 552, in decode
    dec = self.decoder(z)
  File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/whistler/DatasetDM/model/diffusers/models/vae.py", line 202, in forward
    sample = up_block(sample)
  File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/whistler/DatasetDM/model/diffusers/models/unet_blocks.py", line 1212, in forward
    hidden_states = resnet(hidden_states, temb=None)
  File "/whistler/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/whistler/DatasetDM/model/diffusers/models/resnet.py", line 337, in forward
    hidden_states = self.norm1(hidden_states.float()).type(hidden_states.dtype)
AttributeError: 'tuple' object has no attribute 'float'

and the contents of hidden_states is like this:

(tensor([[[[  5.9298,  -3.1241,  -6.5244,  ...,  10.8497,  10.3805,   3.1242],
          [  8.3238,  -9.6560, -14.9727,  ...,  16.8409,  11.2739,  -2.1180],
          [ 12.1110, -10.4051, -19.2935,  ...,  10.8612,   3.4495,  -5.7012],
          ...,
          [  9.0838,  -1.3200,  -0.6628,  ...,  -0.2742,  -3.9212,  12.8734],
....

seems to be a tuple of tensor

[weimzh]

this is when the tensor became a tuple:

at diffusers/models/vae.py:202

        # up                                                                                                                                                                                                       
        for up_block in self.up_blocks:
            print(type(sample))
            print(up_block)
            sample = up_block(sample)
            print(type(sample))
            print('---------')

<class 'torch.Tensor'>
UpDecoderBlock2D(
  (resnets): ModuleList(
    (0): ResnetBlock2D(
      (norm1): GroupNorm(32, 512, eps=1e-06, affine=True)
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm2): GroupNorm(32, 512, eps=1e-06, affine=True)
      (dropout): Dropout(p=0.0, inplace=False)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (nonlinearity): SiLU()
    )
    (1): ResnetBlock2D(
      (norm1): GroupNorm(32, 512, eps=1e-06, affine=True)
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm2): GroupNorm(32, 512, eps=1e-06, affine=True)
      (dropout): Dropout(p=0.0, inplace=False)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (nonlinearity): SiLU()
    )
    (2): ResnetBlock2D(
      (norm1): GroupNorm(32, 512, eps=1e-06, affine=True)
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm2): GroupNorm(32, 512, eps=1e-06, affine=True)
      (dropout): Dropout(p=0.0, inplace=False)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (nonlinearity): SiLU()
    )
  )
  (upsamplers): ModuleList(
    (0): Upsample2D(
      (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    )
  )
)
<class 'tuple'>

[weijiawu]

It's strange, I've never encountered this bug before. It seems like an error related to mismatched VAE versions, although there haven't been any modifications to the VAE within this method, and the VAE section doesn't require training or optimization.

If there's no way to resolve it, I suggest you try running Grounded-Diffusion. Our work is based on this code base, which might be helpful to you. You can find it here: Grounded-Diffusion