ComfyUI for Developers Who are Good at Stuff

While I pine for the days of spending weeks in zbrush sculpting and painting every last detail for the next great AAA game, my suspicion is that those days are gone... or at least they're gone for anyone living in the US in the foreseeable future.

So what to do? Let Europe and Asia, with their superior government support and cost of living, grab this entire industry? Suppose that's one option. I'd rather find a path through if I can.

I think genAI, if I can figure out how to actually wrangle it, could be part of that path through all of this. Hire exceptional artists, and multiply their efforts through new ways of working. We have much of the best talent here. We just can't afford to throw bodies at problems like they can.

The Specificity Problem

Problem with current AI workflows for my uses, is specificity. Only the best products make their money back. And the best stuff is the best because, often, it's considered and polished to hell, and every detail is made to work elegantly with every other, by hand. You can't just throw an anime LoRA from Civatai (see below) and expect players to connect with it because it looks plausibly like JJK. That's not how valuable IP works.

That said? Hitting a style is HARD. Particularly on a larger team. So much time is wasted trying to get other artists to hit the same quality and style as the reference art. Maybe we can democratize that. Speed it up. Make it easier, but still make it so the artist can jump in as much as they need. Do more with less.

Style LoRAs as a First Step

Got to start somewhere. Here's my first thought is style tuning.

Often the first thing you do as an Art Director and Concept Artist, is:

• Gather real life references
• Gather style references based on existing films, paintings and games
• ... and then try to synthesize those two references into a new thing.

Style LoRA's do just that, and a power user can make their own to train it exactly to their needs. So that's what I did here as an initial test.

• I grabbed my own ref by hand.
• I decided what is style ref and what is subject matter ref.
• I trained my own LoRA specifically from that reference (see below)
• I then overpaint, using the LoRA again as an inpaint helper (trying that later)

Note how most of the work is still a person making decisions.

Initial Results and Next Steps

Training took about 5 hours initially, but once you know what you're doing, it's more like an hour or so. Here's initial results. First image is a photo reference:

... and second one here is the result directly from the custom LoRA I created. Obviously needs paintover, but a solid head start.

From here, my expectation is that the artist can inpaint using traditional DCC tools (Photoshop, Krita, etc), and can optionally use this same LoRA to steer loose inpaints back into the target style.

Promising early results... will keep going and see how viable this is in practice.