Spaces:
Running
on
Zero
Seek help when encountering problems
Hello, first of all, thank you very much for providing this space as a developer. This is the best inpaint model I have ever used.
I want to run your model on AWS g6.xlarge, but after clicking inpaint, I found an error message. Here is my error message. Can you help me find out where the problem is? Thank you.
usr/local/lib/python3.10/dist-packages/gradio/analytics.py:106: UserWarning: IMPORTANT: You are using gradio version 4.42.0, however version 4.44.1 is available, please upgrade.
warnings.warn(
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/gradio/queueing.py", line 536, in process_events
response = await route_utils.call_process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/route_utils.py", line 321, in call_process_api
output = await app.get_blocks().process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1935, in process_api
result = await self.call_function(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1532, in call_function
prediction = await utils.async_iteration(iterator)
File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 671, in async_iteration
return await iterator.anext()
File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 664, in anext
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 962, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 647, in run_sync_iterator_async
return next(iterator)
File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 809, in gen_wrapper
response = next(iterator)
File "/root/diffusers-fast-inpaint/./app.py", line 55, in fill_image
) = pipe.encode_prompt(prompt, "cuda", True)
File "/root/diffusers-fast-inpaint/pipeline_fill_sd_xl.py", line 158, in encode_prompt
prompt_embeds = text_encoder(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/clip/modeling_clip.py", line 1490, in forward
text_embeds = self.text_projection(pooled_output)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py", line 117, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half
Oh, to fix this, you might have to modify the app.py file.
Change this line
pipe = StableDiffusionXLFillPipeline.from_pretrained(
"SG161222/RealVisXL_V5.0_Lightning",
torch_dtype=torch.float16,
vae=vae,
controlnet=model,
variant="fp16",
).to("cuda")
to
pipe = StableDiffusionXLFillPipeline.from_pretrained(
"SG161222/RealVisXL_V5.0_Lightning",
vae=vae,
controlnet=model,
variant="fp16",
).to("cuda", torch.float16)
This should fix the problem. I haven't investigated the problem, but it seems the dtype argument for the pipeline isn't properly casting the model weights (this might be a diffusers issue). So directly allowing pytorch to handle the casting fixed the problem for me.
thanks a lot! I've seen this problem and it seems it's because of a recent update with transformers that introduced this breaking change, this space is safe because the version is locked but I'll try to test it and fix it when I have the time.
I really appreciate your feedback.
Thank you very much. With your help, my code has started running.
I have another question, can only support 512 * 512 images now? Can you support rectangular diagrams?
this space by default uses 1024x1024 images, this space was a PoC (Proof of Concept) I did for a diffusers guide so it has the minimum code to make it work. Of course this technique can work with any resolution the SDXL model works with, also you can just crop the part you need and the paste it back but these functions are all out of the scope of the guide or what I have time to do right now.
If you want to work with different resolutions and hard code them, you can add the width and heigth arguments here like for example:
for image in pipe(
prompt_embeds=prompt_embeds,
negative_prompt_embeds=negative_prompt_embeds,
pooled_prompt_embeds=pooled_prompt_embeds,
negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,
image=cnet_image,
width=1280,
height=768,
):
and you need to do the same for the gradio mask image here