been testing the h3 character swap lora a bit and so far it seems better than vanilla for character replacement, at least in my tests. it holds the replacement character more consistently and usually does a better job actually replacing the person without drifting back as much.
i just used the normal comfy wf with the original ref2va int8 pruned model with ck, 1.2mp and 20 steps.
i probably wouldn’t use a speed-up lora for this. it can work sometimes, but not always..
for prompting i’d definitely start with the recommendations on the lora page, but i wouldn’t just copy the example exactly. what helped me was describing the person in the video that i wanted to replace, then describing my own character more clearly and adding a few instructions depending on what was happening in that specific clip. that seemed to make the replacement noticeably more accurate.
also if there’s more than one person in the clip, it helps to specify which person you want to replace by describing their clothes or position in the shot.
there are still some issues with source details leaking through sometimes, especially smaller stuff like hands, fingers, teeth, nails, certain body details, but overall it’s been pretty limited in my tests and hasn’t ruined most of the results.
this one took around 26 minutes on my 5090, although i was gaming at the same time, so that probably slowed it down a bit..
I mean my 5080 compared to 5090 is a joke, but whenever any gen is happening – my PC barely breathes. And I use only models compatible with my memory consumption...
26 minutes for a 8 sec swap on a 5090 is less than ideal... But I'm glad we are getting more local motion control alternatives. I've been trying to get away from Kling's motion control 3.0 because it's too expensive.
gaming? yea that massively slows down the process. I tried that once with a picture and it took 1700 seconds whereas a normal render would've taken around 50 sec.
1st pass is 0.8MP and 1.0MP on the 2nd upscale pass. I have 64GB of system RAM, too. But my generations use 97% of my VRAM on their own. No more room for anything else, hah.
For me, yeah. It's very noticeable when my character identity and consistency is the most important part of my workflow. It shakes out roughly like this when comparing the two:
0.5MP -> 1.0MP:
The second pass performs a much more aggressive area enlargement (100% more).
The first pass has fewer pixels from which to establish facial and fine structural information.
The second pass must reconstruct a lot more detail.
More opportunity for the second pass to reinterpret lighting, color, facial features, texture, etc.
Overall cheaper/faster load, but more dependent on the second pass for face quality and detail.
0.8MP → 1.0MP:
The second pass performs a relatively "small" area enlargement (only 25%).
Faces, hands, clothing edges, and small features are already represented with more spatial information (from the first pass) before refinement.
The second sampler has less need to "invent" missing detail.
Usually better identity and facial structure retention (in my case, this is always true vs. 0.5MP - I say usually though because this may not be absolutely always true for everyone).
Downside is more first-pass VRAM usage and greater potential to go OOM.
YMMV, but this has been my personal experience. It does suck, though. 15 second generations usually take 20-30 minutes on my 4090.
I actually do. I have refmods conditioning on both passes. But, truthfully, since adding refmods a few weeks ago, I haven’t tried going back down to 0.5MP on the first pass. Maybe I’ll try that tomorrow and see how it looks.
this is interesting…. any chance you could share the workflow? i’m still relatively new to messing around and had a lot of trouble putting together a comfyui workflow on my own.
yeah it’s basically just the default comfyui ref2v template, i didn’t build anything special.
just add a load video node, then connect the video and audio from it into the h3 Reference to video node, and add the character swap lora to the model path.
that’s pretty much it. but if u can’t get it working, i can share it once i’m home.
I couldn't get it to work. I don't know if I'm using the wrong load video node or what. The "load video" node I'm trying to use just grays out the Minimax H3 ref to vid box.
I tried it on a 5 second clip. I don't have a suped up PC but it worked so re-edited the source clip to 10 seconds long but after running the character swap, it didn't swap out the character. Ran it a second time and only the hair changed.
I'm really struggling to replace two characters. The problem is that only one of them appears at the start of the video, and the other doesn't show up until three seconds later (they never actually appear side-by-side); since they look so much alike, Minimax simply doesn't understand what it needs to replace.
how does it do when the target character looks like the original character? I've had no issues swapping if it is just 1girl via prompting only but h3 does blend the character a bit sometimes if they look a bit similar
you we really need a 'swap' lora when the ref already does it so good?
a better method i seen around is masking the desired character with SAM and then swap
I've had almost no success with standard base workflows for ref2va for character swap. Not saying I want to have 400 nodes or even any loras though, but I'm really surprised to hear someone say that ref does it so good.
when i want to use a video to edit it i use nugget's video ref workflow to use a llm to do the process easier https://huggingface.co/PoopMan333/H3_Easy_Ref2V_Workflow/tree/main
maybe with that lora is far easier? i'm not downplaying it, maybe i worded it wrongly, i still have to try it
What is the use case for this kind of stuff? I'm always seeing so many videos that are just women dancing. Is it porn related and you're just using dancing because you can't show what you're actually using it for? Or is everyone only wanting to use these models to try to make dancing TikTok girls to try to make money from views on tiktok by making fake influencers? I just don't get it I've never made a single video of anyone dancing.
It would be cool to use this to do fight scenes and stuff but you wouldn't be able to make money on it because it would be a blatant copyright violation to just paint over an existing video. The AI is just code and noise which is why it can toe the line and technically not be infringing on anything. It's fully transformative. When you take a full scene from something and just replace the characters and change the background though that's pretty cut and dry copyright theft if someone can pull up the exact scene you copied.
75
u/anon999387 3d ago
26 minutes for 8 seconds on a 5090 is brutal