Lyell Hintz patreon

ControlNet + More Tutorial [ StreamDiffusionTD 0.2.2 ]

🕑 Added 2024-09-05 16:03:01 +0000 UTC

Comments

Jonatan Ewald

Is there any tutorial or post somewhere on how to install other checkpoints or LoRAS to use with Stream diffusion?

Shio Visconti

...Loading sd15 ControlNet model: control_v11p_sd21_depth.safetensors Failed to load ControlNet model https://huggingface.co/thibaud/controlnet-sd21/blob/main/control_v11p_sd21_depth.safetensors due to: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect. Traceback (most recent call last): File "D:\AI TOUCH\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 637, in _load_model controlnet = ControlNetModel.from_single_file(controlnet_file, config_file=config_file, use_safetensors=True, local_files_only=clip_exists, torch_dtype=torch.float16).to(device=self.device, dtype=self.dtype) File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\loaders\single_file.py", line 617, in from_single_file controlnet = download_controlnet_from_original_ckpt( File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1846, in download_controlnet_from_original_ckpt controlnet = convert_controlnet_checkpoint( File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1125, in convert_controlnet_checkpoint set_module_tensor_to_device(controlnet, param_name, "cpu", value=param) File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\accelerate\utils\modeling.py", line 292, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect.

Ignacia .Hurtado

issue, when i start streaming with controlnet ON, show me this error: "ValueError: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect. " help pleaseee

Dji

Hello, when start streaming it says in the console : " V2V + ControlNet are not supported and will be disabled. A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton'. " Do you have an idea where this issue comes from and how I can resolve it ?

Kasım Açıkbaş

hey, can you tell me how to do the pose controller? from your video I saw depth and edge but couldn't find any pose related object in TD

Raja Singa

I can't do it on streaming. cmd doesn't load data. can you please help me

Gus

yeah Im having problems with this too - earlier versions could

dotsimulate

Did it ever finish loading? maybe restart whole process. Depending on the amount of vram on your gpu, it could be having trouble fitting on it, I would recommend 8gigs at least.

dotsimulate

You can negative prompt certain things with negative weight on LoRA models, which is a bit more limiting but could work for certain things.

dotsimulate

negative prompts don't work with lcm / 1 step processes such as streamdiffusion unfortunately.

Andrea Bucci

[issue] ...Loading sd-turbo (2.1 distilled) model pipeline... ...Loading sd-turbo (2.1 distilled) model: sd-turbo Loading pipeline components...: 0%| | 0/5 [00:00

Edmond Yang

how do I enable the additional settings for negative prompts?

Dan Fong

Thank you so much for the controlnet tutorial. Would you also share how to use openpose in stream diffusion ? How to get the colorful skeleton form video source? Thanks in advance.

Ismael Abrego

Hi all, I'm enjoying this tool, thanks DotSimulate. But I'd like to know if you have found models better than SD-TURBO; do you have any advice?

Frédéric Bussière

What would be the equivalent of img2img in comfy? I get good results when I send a video thru stream diffusion but I would like to increase the quality and fps, and I'm looking for the equivalent with stable diffusion / comfyui. Thanks !

jeevanpillay

same issue. fixed with pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 thanks dot!

TITO FERRARI

same

Diskorsia

I have the same error when I run the stream, Did you manage to solve it?

Julii Haid

lo voy a ver, Gracias Dot!!

Julii Haid

Traceback (most recent call last): File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 683, in _load_model pipe = pipeline_class.from_pretrained( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1090, in from_pretrained cached_folder = cls.download( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1856, in download cached_folder = snapshot_download( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\huggingface_hub\_snapshot_download.py", line 216, in snapshot_download raise LocalEntryNotFoundError( huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass 'local_files_only=False' as input. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 687, in _load_model except huggingface_hub.utils._errors.LocalEntryNotFoundError: AttributeError: module 'huggingface_hub.utils' has no attribute '_errors' Failed to load model due to: module 'huggingface_hub.utils' has no attribute '_errors'

Johao Mikaël Marin Gagné

this issue is caused by hugging face having a newer version. Click on .tox file on Manual Activation VENV then type this to downgrade huggingface to the right version : pip install huggingface-hub==0.24.6 Then launch it and it should download dependencies (I had to relaunch twotimes but now its working)

屈

Why prompt No internet connection detected。

Mariana Kamke

is there only 512x512 ? or is there a way to have a better resolution?

dotsimulate

# StreamDiffusionTD huggingface_hub.utils._errors troubleshooting guide 1. Hit "Manual Activate" (install page) pulse to open command window. 2. Check package versions: ``` pip list ``` Look for: - torch: 2.1.0 - huggingface-hub: 0.24.6 - numpy: 1.26.4 3. Check CUDA: ``` python -c "import torch; print(torch.cuda.is_available())" ``` Should print "True". 4. If versions wrong or CUDA unavailable, check CUDA version: ``` nvidia-smi ``` 5. Install correct packages ( AND replace cu121 with cu118 if you have cuda11.8): ``` pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 ``` 6. Recheck CUDA (step 3). 7. If you continue to have an issue, please join the discord as it is way easier to troubleshoot on there. And post in the troubleshooting forum the details from above.

dotsimulate

see above. Also follow up on discord. It is impossible to get notified or keep up with this via patreon comments.

dotsimulate

# StreamDiffusionTD huggingface_hub.utils._errors troubleshooting guide 1. Hit "Manual Activate" (install page) pulse to open command window. 2. Check package versions: ``` pip list ``` Look for: - torch: 2.1.0 - huggingface-hub: 0.24.6 - numpy: 1.26.4 3. Check CUDA: ``` python -c "import torch; print(torch.cuda.is_available())" ``` Should print "True". 4. If versions wrong or CUDA unavailable, check CUDA version: ``` nvidia-smi ``` 5. Install correct packages (replace cu121 if needed): ``` pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 ``` 6. Recheck CUDA (step 3). 7. If you continue to have an issue, please join the discord as it is way easier to troubleshoot on there. And post in the troubleshooting forum the details from above.

dotsimulate

Yes. I am looking at this. It seems to not be installing all of the pip packages it should. The issue started about 5 days ago or so. still tracking down what is causing it but there are a few fixes that have worked for people in the discord. I will summarize below.

Sasha Bédard

got the same thing trying to use any model other than sdturbo

Jack Kalish

I'm having the same issue today...

Luca Corsi

hi! i have this issue: ...Loading sd-turbo (2.1 distilled) model pipeline... ...Loading sd-turbo (2.1 distilled) model: sd-turbo Traceback (most recent call last): File "D:\touchAI\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 683, in _load_model pipe = pipeline_class.from_pretrained( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1090, in from_pretrained cached_folder = cls.download( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1856, in download cached_folder = snapshot_download( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\huggingface_hub\_snapshot_download.py", line 216, in snapshot_download raise LocalEntryNotFoundError( huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass 'local_files_only=False' as input. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\touchAI\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 687, in _load_model except huggingface_hub.utils._errors.LocalEntryNotFoundError: AttributeError: module 'huggingface_hub.utils' has no attribute '_errors' Failed to load model due to: module 'huggingface_hub.utils' has no attribute '_errors'

Victor E Morales

Hello. Thank you for the amazing work! does 0.2.2 work on mac? Oh I just saw in changelog that you will update Mac-support. I wanna test that :)

Stuart Smith

Mac install guide?

Q 小

Why can't I download streamDiffusion? The message "no internet connection detected "is always displayed+1

WDH2021

Why can't I download streamDiffusion? The message "no internet connection detected "is always displayed

fan ze

Can you record a tutorial for a plug-in to load the civitai Lora model library? I downloaded several model libraries and loaded them in the models in the plug-in, but the effect I got was similar to the effect of not binding the plug-in. I'm not sure whether my model library is not running now. Is there any way to judge that my model library is being used normally?

Kevin Friel

Amaaazing work man! Very cool stuff. I’m curious, as I have a couple of GPU’s…. would there be a way for 2 frames to be processed at once on say a 4090 and 4080 setup? I’ve seen some chatter on this being the way to leverage them, in lieu of actual multi-gpu crunching on each frame. Appreciate it!

Yanandra Ardi Putra P

hello, im newbie, i already install the streamdiffusion 0.2.2, after install the step 1 and step 2, i try to run start stream, the command prompt stuck in last log here Loading pipeline components...: 0%| | 0/5 [00:00