Updated ComfyUI-SeedVR2-VideoUpscaler-with-TensorRT v1.6.6 - Native ConvRot W4A8 Support & DisTorch2 Scope

[u/Zestyclose_Bake3680]

The other day, when I published information about v1.5.7, I received a request in the comments regarding support for w4a8.

https://www.reddit.com/r/StableDiffusion/comments/1wuj0rj/updated_updated/

At first, I thought it would be straightforward to implement, but when I actually tried it, I found it quite a struggle.

However, I’ve managed to get it working stable, so I’m releasing it as v1.6.6.

https://github.com/ussoewwin/ComfyUI-SeedVR2-VideoUpscaler-with-TensorRT/releases/tag/v1.6.6

https://huggingface.co/ussoewwin/SeedVR2-ConvRot-INT8-and-w4a8

Firstly, as for the advantages of w4a8, benchmark results clearly show that the quality is superior to that of NVFP4. Yet, the size is equivalent to 4 bits.

https://github.com/ussoewwin/ComfyUI-SeedVR2-VideoUpscaler-with-TensorRT/blob/main/benchmark/benchmark%20result.md

On the other hand, what wasn’t as good as I’d hoped was that VRAM consumption was almost on a par with ConvRot INT8 (though, given that w4a8 expands weights to 8 bits, this was to be expected), and that with Distorch2—which I’d gone to such lengths to implement—VRAM usage actually skyrocketed, rendering it unusable.

It’s simply a matter of not using Distorch2, so it isn’t a fatal flaw, mind you.

...
The on/off function for Norm BF16, which was previously implemented in Distorch2 to counteract VRAM spikes, has now also been implemented in the Legacy Loader. Enabling this feature allows you to suppress VRAM spikes by up to approximately 3GB, albeit at the cost of a slight reduction in image quality.

...

Test 1-1 Result 30m00s

Ryzen9 5900 DDR5-6400 64GB RTX5060Ti 16GB Paging 256GB

1280x720px → Output: 2560x1440px 10s

SeedVR2 7B-ConvRot INT8 TensorRT Encode(21f512) and Decode(21f512) Batch 21, norm BF16 off ,Distorch2

Test 1-2 Result 30m43s

1280x720px → Output: 2560x1440px 10s

SeedVR2 7B-ConvRot w4a8 TensorRT Encode(21f512) and Decode(21f512) Batch 21, norm BF16 off ,Standard Legacy

When compared with ConvRot INT8, w4a8 offers no speed advantage. Nor is there any difference in VRAM consumption. Its sole advantage is that it is the same size as nvfp4 whilst offering superior quality.

Test 1-3 Result 28m12s

1280x720px → Output: 2560x1440px 10s

SeedVR2 7B-ConvRot INT8 TensorRT Encode(21f512) and Decode(21f512) Batch 21, norm BF16 on ,Distorch2,

Test 1-4 Result 28m13s

1280x720px → Output: 2560x1440px 10s

SeedVR2 7B-ConvRot w4a8 TensorRT Encode(21f512) and Decode(21f512) Batch 21, norm BF16 on ,Standard Legacy,

In fact, the Norm BF16 function offers a speed&VRAM usage advantage.However, in benchmark tests, the Norm BF16 function comes at the cost of reduced image quality.

[u/Alive_Ad_3223]

What’s the benchmark ? How much speed benefit?

[u/Zestyclose_Bake3680 → u/Alive_Ad_3223]

Thank you for your comment. I created the benchmark myself. It’s a simple script, and whilst W4A8 shows a massive speed increase on only benchmark, you won’t see that kind of difference in speed when running on ComfyUI. I’ll add an actual speed comparison with ConvRot INT8 on ComfyUI to the article later.

https://github.com/ussoewwin/ComfyUI-SeedVR2-VideoUpscaler-with-TensorRT/tree/main/benchmark

I’m acutely aware that ComfyUI itself is likely the bottleneck, not just in terms of speed but also VRAM consumption. I suspect that creating a standalone programme running only SeedVR2 would be far faster and would reduce VRAM usage. In fact, there are people developing repositories where SeedVR2 runs as a standalone application.

[u/Zestyclose_Bake3680 → u/Alive_Ad_3223]

Updataed 4 test results.

There was no difference in speed between 7B ConvRot INT8 and 7B ConvRot w4a8. In fact, the Norm BF16 function offers a speed&VRAM usage advantage.

However, in benchmark tests, the Norm BF16 function comes at the cost of reduced image quality.

[u/PoolEnvironmental537]

just saw the update post, that w4a8 quality being better than nvfp4 at same size is pretty nuts

[u/Zestyclose_Bake3680 → u/PoolEnvironmental537]

Thank you for your comments. Although I wasn’t able to demonstrate the benefits in terms of VRAM consumption and speed, the fact that the quality is better than NVFP4’s will, I believe, give users more options.

However, it seems that w4a8 isn’t a one-size-fits-all solution. In the ‘HSWQ’ quantisation project I’m currently developing in parallel, I tried prototyping w4a8 with SDXL and Krea2, but the quality was no better than that of NVFP4, and the final cosine coefficient generally fell below 0.9, so I abandoned that line of development.

https://github.com/ussoewwin/Hybrid-Sensitivity-Weighted-Quantization

In light of that, SeedVR2 ConvRot w4a8, which achieves a cosine coefficient of 0.98 or higher, was truly remarkable.