StreamDiffusionTD 0.1.11 🚀🎮🖼️ [ Changelog]

help me solve this problem, I'm quite frustrated looking for this problem : from huggingface_hub import HfFolder, cached_download, hf_hub_download, model_info ImportError: cannot import name 'cached_download' from 'huggingface_hub' (D:\StreamDiffusion\venv\lib\site-packages\huggingface_hub\__init__.py) [process exited with code 1 (0x00000001)] You can now close this terminal with Ctrl+D, or press Enter to restart.

Raja Singa

Hey! Couldnt get the TensorRT to work, and can't find the Discord for support :)

Rotem Mor

Hi, I've been experimenting with this this Tox for StreamDiffusion and really like it; except that the images that I'm getting are far from great (txt to img), no matter how well defined and detailed is the prompt that I am using. SD Turbo model works best that others but is still far from good images. Any advice?

Ismael Abrego

正常工作了，谢谢。

wang haiqiang

Its running perfectly, can anyone guide me how to use it on GPU. thank you

Pixadoo Visuals

absolutely STOKED to have controlnet added thank you so much!

fergus laidlaw

Hello DotSimulate. Thanks for the GREAT tool. Sadly I cannot run it. I have a Razer laptop with an RTX3080 Ti . Do not know if it's powerfull enough . Anyways the error I get when pulsing the "Start Stream" is ModuleNotFoundError: No module named 'diffuser' in the cmd window. (ps: it opens and closes quickly so I did a screen recording to read the error. ) Any clue on what this is ? I've searched this thread but not found any info. Thanks in advance

Philippe Hughes

Same thing for me

Leon

hey! I have a trouble running it. I followed the step-by-step tutorial, but it doesn't seem to work. Where can I tell you about the error? First it's "A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton'" and then while running the model ESRGAN (that work normally fine in automatic 1111), it shows me this exception 'OSError: Error no file named model_index.json found in directory D:\ARTISTIC PROJECTS\2024\Stable Diffusion AI\stable-diffusion-webui\models\ESRGAN.' & "UnboundLocalError: local variable 'pipe' referenced before assignment" Do you have any idea how to fix it? Thank you

Youssef El Idrissi

working great but "ModuleNotFoundError: No module named 'polygraphy' Acceleration has failed. Falling back to normal mode" what did I not install ?.

Gus

I'm currently having the same problem as you, and it worked normally before.

Wu shuyang

Hi, your works are awesome! Can I get the discord link? Thank you very much

Cheng Yuan Wang

I need a tutorial on how to change the model ID using the Hugging Face or Civitai website

삭 이

Such a great tool. Are there any resources on how to run completely local, no internet? I have everything running ok I think, max 16fps on 3090 with a single Step Schedule. Should be more, ya? Very much appreciate any leads on these.

Yones R

Hey loving this tool thankyou for making. I'm having trouble loading a controlnet model. I get the following error \Users\markw\OneDrive\Documents\Touch\StreamDiffusion\streamdiffusionTD\pipeline_td.py", line 499, in predict_x0_batch control_image = torch.cat((control_image, self.control_image_buffer), dim=0) TypeError: expected Tensor as element 1 in argument 0, but got NoneType

Mark

My issue was with the controlnet model and SD1.5. The lllyasviel/control_v11f1p_sd15_depth, didn't work for me. Instead I used https://huggingface.co/webui/ControlNet-modules-safetensors/blob/main/control_depth-fp16.safetensors and now its streaming.

arunr6600 .

Stuck in the same place, I was using sd-Turbo seems to be based on 2.1 so I thought to try the correct canny processor but no success. Is there anything I am missing?https://www.patreon.com/posts/streamdiffusiont-104339324

publicoprivado

你好，我点击start stream pulse之后，一直显示Loading pipeline components...: 20%|██████████▍ | 1/5 [00:00<00:02, 1.75it/s]和RuntimeError: Numpy is not available和UnboundLocalError: local variable 'pipe' referenced before assignment，请问怎么解决？

CICI

Hello, Thank you for making this tool <3 In terms of processing power and hardware, do you know what hardware we should buy to make this more efficient? Does a gaming GPU outperform a Quadro? (OpenCL vs OpenGL) Or would an AI card be better? Would having more CPU power and ram vs GPU help? We're getting about 15-20 fps on a 4090 and I'd love to buy something that would give us better fps, but we don't know WHAT we need to buy for better performance? Any insight would be greatly appreciated. :)

Arvol Dingess

could beto do with egg-info deprecation ?

Benlloyd Goldstein

i have cuda 11.8 , but when i run install i get this error : ------------------------------------- 991.4/991.4 kB 30.7 MB/s eta 0:00:00 Collecting nvidia-cudnn-cu116 (from nvidia-cudnn-cu11->tensorrt_libs==9.0.1.post11.dev4) Downloading nvidia-cudnn-cu116-0.0.1.dev5.tar.gz (4.4 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'error' error: subprocess-exited-with-error python setup.py egg_info did not run successfully. exit code: 1 [17 lines of output] Traceback (most recent call last): File "", line 2, in File "", line 34, in File "C:\Users\benll\AppData\Local\Temp\pip-install-ugh06s25\nvidia-cudnn-cu116_5b65ba7b510e46268f7994b13fc547a8\setup.py", line 152, in raise RuntimeError(open("ERROR.txt", "r").read()) RuntimeError: ########################################################################################### The package you are trying to install is only a placeholder project on PyPI.org repository. This package is hosted on NVIDIA Python Package Index. This package can be installed as: ``` $ pip install nvidia-pyindex $ pip install nvidia-cudnn-cu116 ``` ########################################################################################### [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed Encountered error while generating package metadata. See above for output.

Benlloyd Goldstein

Dot it could be that the Guidance Scale parameter of Stream Setting 1, when it exceeds the value 1 in this version, it breaks?

Matias Gomez De Falcis

可以运行了。谢谢。

wang haiqiang

Hey. Try again with the operator in this post (I made a tiny change and reuploaded just now.)

dotsimulate

我在TD里运行 view installlation guide ,出现黑屏。

wang haiqiang

FYI when you have the correct versions of cuda 11.8, this sorta just works using the install tensorrt button, then selecting tensorrt from the acceleration dropdown on Stream Settings 1.

Scott Hedstrom

Ok after a bit of debugging and experiments, it looks like cuda 11.8 is the required vs 12. Got everything working at 80fps! TensorRT is the juice. Thanks so much @dotsimulate !

Scott Hedstrom

ok, so after a bit of digging, looks like tensorrt.icudaengine version 10.0.1 doesnt have num_bindings, it has num_io_tensors... 8.6.1 seemed to have it, but I can't get it to let me install via pip

Scott Hedstrom

Basically the error is saying python doesnt have access to anything called pytorch_lightning and the software is trying to access it. I would try this: Go to the venv folder in terminal, type `source Scripts/activate` (without the `) This will make sure you are on the venv for the project... Then type `pip install pytorch_lighting` Try running it again...this will potentially lead down a list of similar errors and solutions, but will get ya started.

Scott Hedstrom

ok, finally got the venv installs working, had to add a few things but now when I run w/ tensorrt it says: tensorrt_bindings.tensorrt.ICudaEngine object has no attribute 'num_bindings' not sure where else to dig here as there isnt much coming back when I search for all that

Scott Hedstrom

a video on correct tensort implementation would be amazing!

Benlloyd Goldstein

Thanks for the awesome work and goodies!

Scott Hedstrom

For some reason I can't get tensorRT to boot...it says "no module named 'polygraphy'" in streamdiffusionTD/wrapper_td.py -> _load_model tried installing polygraphy myself in the venv, but no luck.

Scott Hedstrom

Hi, I am getting this error. Any fix? ...Loading ControlNet model: controlnet_sd15_depth_f16.safetensors Failed to load ControlNet model E:/Stable Diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/models/controlnet_sd15_depth_f16.safetensors using from Huggingface (from_pretrained) due to: It looks like the config file at 'E:/Stable Diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/models/controlnet_sd15_depth_f16.safetensors' is not a valid JSON file. Failed to load ControlNet model from local directory (from_single_file) E:/Stable Diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/models/controlnet_sd15_depth_f16.safetensors due to: 'time_embed.0.weight' Thank you

arunr6600 .

how would you go about applying motion loras to this ?

Benlloyd Goldstein

i havent had luck with XL either but control net should work dotsimulate was demoing it in his livestream interview for civitai which is on youtube, probably a matter of loading the right models.

Gordon kummel

Believe it only works with 1.5 models for now but please correct me if I'm wrong - I would also love to get XL working

Jasmine Rose Federer

Hello there! I tried installing/reinstalling/deleting venv folder/ installing different version of python/... but I keep getting the following error: Failed to load SD model from both local and Hugging Face due to: No module named 'pytorch_lightning' Any idea on how to resolve this? Thanks a lot!

Lucas

I had to delete the venv folder but after that updating python and installing the dependencies from scratch it work!

Latent_

hey i need help i can only use openai/turbo-sd model any other 1.5 or xl turbo xl etc. cant work? also non of my control net models are being accepted

aviv oz

Thank you for your reply. Why give the possibility to load a specific ControlNet model? I thought loading a specific controlNet model (eg depth, canny, normal, segmentation etc) in the controlNet tab of streamDiffusion would control the image generation like in Stable Diffusion 1111. But is the preprocessed image that affects the overall output. I get it now. Is there a way to generate the depth from a video in source in touchdesigner at 30fps? Is it even possible? I get generating the edge or normal in touchdeisgner but the depth comes only from the render node if the scene is 3D, right?

Stella Speziali

The result from the model is always going to be the output from the diffusion process. The controlnet input must go in already preprocessed (in depth map form if using depth, or with edge lines if using softedge, etc).

dotsimulate

Hello! Is there a way to see the result of the control net? as an output

Stella Speziali

Link is available via any of the operators I’ve released in the about page (about me>discord)

dotsimulate

heyy - could you please reshare the discord link? thanks

Jasmine Rose Federer

hey! I´m havin issues running it, i know CUDA is necessary but I have a nvidia 330 so i don´t know if the issues come from it

JuanJo Casallas Contreras

"Failed to build tensorrt ERROR: Could not build wheels for tensorrt, which is required to install pyproject.toml-based projects", seems like a common error, would you have any workaround?

Arthur Deleu

Hey man, having issues with tensorrt...

Arthur Deleu

Hello, Thank you for providing amazing TOX! I would like to know how to change the output resolution? I try to get into [StreamDiffusion TD component] and change the [ndiout] but still doesn't work.

kit lee

Thank you!

Lucas Chen

Hello and thank you! I've installed everything successfully and get a stream info output, but my StreamDiffusionTD is blank. The only error I see is: StreamDiffusion\streamdiffusionTD\main_sdtd.py:393: RuntimeWarning: invalid value encountered in cast processed_np = (processed_np * 255).astype(np.uint8)

Joseph Armen

Hello Would you help me install LORA on your latest Tox from streamdiffusion? I get the following error " ERROR loading LoRA Models: Cannot load because down.weight expected shape tensor(..., device='meta', size=(32, 320)), but got torch.Size([32, 320, 1, 1]). If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example. "

Luis Guillermo Helfer

i'm using it with without any problem so i'd say you messed up something during the installation :)

Riccardo Fasoli

thanks amazing work

We_ Make

orphée Cat

I just downloaded the tox and followed the installation but when I try to start the stream the console opens and closes immediately. I see an error "no module named torchvision"

Raul Leal

Any ideas on when are we getting NDI support back?

Dilip Ramírez

it is the about section of each of my operators and I think in about me as well.

dotsimulate

it worked !!!! Thank you!!

Noam F.

this is because of model mismatch. you will want to use 1.5 controlnet model and 1.5 sd main moderl

dotsimulate

I should have a bit clearer info on how to dm or contact for support. I am planning on a video soon for a controlnet update. sd-turbo won't work with controlnet currently. but hoping to get that working soon as well. You need to use 1.5 main sd model and a 1.5 controlnet model at the moment.

dotsimulate

silly question - where is the link for discord? Am i blind?

Anton Thomsen

I solved the problem by installing Python 3.10 64-bit instead of 32-bit. https://www.python.org/ftp/python/3.10.9/python-3.10.9-amd64.exe

GMM

Same her

Daniel Lozano

same issue here

Jeremy Sanders

I have exactly the same issue! Did you manage to solve it yet?

Felix Reiterer

Hi ! I saw your civitai stream .. great stuff thanks! Could you share with me the discord link?

Aratz Hernandez

I often touch 40fps but it doesn't seem to be using the gpu a 20 percent is using it parsec to remote the machine. when i install tensorcore it gives error but then if out of venv i install the correct version it seems to go. it gives me an error if i turn on cfg type it doesn't seem to work any controlnet so I wonder, did you put the page in without explaining which ones you can use because they don't work yet? I would need help but I didn't understand if I have to send a private message to Dotsimulate, if I have to go to discord, if I have to post every question publicly, I would really need feedback but I am not familiar with the rules. https://imgur.com/a/dotsimulate-streamdiffusion-krd-test-v1-1-4xw3Xvr

Guglielmo Guarnera

I thought when I subscribed to your Patreon I was going to be in and out just to download the operator, but I've stayed for the constant updates. You really work hard on this and I love the new features in each update!

Alex England

Hi! I have this issue when i try to install tensorT: RuntimeError: ########################################################################################### The package you are trying to install is only a placeholder project on PyPI.org repository. This package is hosted on NVIDIA Python Package Index. This package can be installed as: ``` $ pip install nvidia-pyindex $ pip install nvidia-cudnn-cu116 ``` ###########################################################################################

Andrea Tamburrini

Hi! First of all, thanks for your work!! I have a few questions. The first one: When I press the "Install (venv + all req)" button, I encounter a series of errors throughout the installation: Building wheel for n-vidia-pyindex (project.toml) ... error error: subprocess-exited-with-error Building wheel for n-vidia-pyindex (project.toml) did not run successfully. exit code: 1 /88 lines of output RuntimeError Traceback (most recent call last): File "" , line 1, in ModuleNotFoundError: No module named 'pip' ModuleNotFoundError: No module named 'setuptools' note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for nvidia-pyindex Failed to build nvidia-pyindex ERROR: Could not build wheels for nvidia-pyindex, which is required to install pyproject.toml-based projects Installing dependencies with pip from the activated virtual environment... Requirement already satisfied: pip in c:\users\flore\downloads\fran\x\streamdiffusion\venv\lib\site-packages (24.0) Looking in indexes: https://download.pytorch.org/whl/cu118 ERROR: Could not find a version that satisfies the requirement torch==2.1.0 (from versions: 2.2.0+cu118, 2.2.1+cu118, 2.2.2+cu118, 2.3.0+cu118) ERROR: No matching distribution found for torch==2.1.0 The second is that (despite the errors I get at the beginning) when I click on "Start Stream", a window opens with some text that I can't read because it closes immediately. In "Log Message" it says "started streaming using CMD" and in "stream Info" it says "not streaming". I hope you can help me. best regards!

Noam F.

Hi, I managed to load a Controlnet model: "Successfully loaded ControlNet model from local directory". But creating a model dropdown does not work - only a single file. When starting I get many error messages about specific files, under the title Process-3. And ultimately also a RuntimeError: "mat1 and mat2 shapes cannot by multiplied (154x2048 and 768x320)". I don't know exactly what to do with input 2 either. Can you show a working example? Kind regards, Pieter

Pieter van Dijk

This is fantastic! Where can I learn about installing Controlnet specifically for StreamDiffusionTD?

Kelly Warner

Ouuu, this is a big one here! Appreciate your hard work my dude 🙏🏽

JJ JV

Everything works fine, the fps count its around 0,5 and by pc specs are a Intel Core i5-10300H, have 8g of ram and have a rtx 1650, but i have a problem in wich the screen becomes black or rainbow color and the model does not load up, Does someone knows how to fix it?

João Silva

I am using cuda 12.1 . I had a cuda version error which is fixed with dotsimulate's pull request on the streamdiffusion GitHub page. I also had a no module found error for cuda and onnxruntime , which can be fixed by activating the venv ( pulse in streamdiffusiontd didn't work for me ) and pip install cuda-python and onnxruntime. I don't think I had other errors on rttensors install. Thanks dotsimulate btw , nice work.

Rapid Carrot

send me some screenshots on discord. there are a few things that could be happening

dotsimulate

Has anyone else had any issues with building TensorRT? It keeps giving me errors and failing to build.

Daniel Potes

this is huge! thank you!

James Clarke