1
Is there a way to create art for a book with consistent theme and characters using AI? (lemmy.dbzer0.com)
submitted 11 months ago by to c/stable_diffusion@lemmy.dbzer0.com
14 comments fedilink hide all child comments
 

I'm interested in creating art for a book, like art that goes alongside the text.

One of the difficult parts of this is that I want characters in the book to have a stable look and not change from image to image.

Is there a way to do this? I have experimented with different localized models and often there were artifacts and I couldn't get consistent results. I am mildly intelligent with running local models, but I am neither an expert nor a computer genius. I was able to do something like "character is pretty and tall with black hair" but each time that anything was generated, the character would look different.

It's been about a year since I last tried anything and since then technology has progressed. If I can't get the characters to look consistent from picture to picture, I'd rather not have images, as I can't afford an illustrator.

all 14 comments

sorted by: hot top controversial new old
[–] 5 points 11 months ago (4 children)

As far as I know, one way to do it is to use an image2image model / image editing model. You'd need to generate one reference image of your character. And then feed the reference image into it and tell it to draw that character in a different scenario. I haven't done it in a while, not sure what people use these days to do it locally.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] [S] 2 points 11 months ago (3 children)

    That sounds so hard to do locally. I don't even know where I would begin. Is that expert level stable diffusion?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 4 points 11 months ago* (2 children)

    You would probably need to use something like Visual Novel Character Creation Suite to generate a character and a dataset for LoRA training, and then train a LoRA for each character. LoRA are adapters you plug into larger models to help it generate whatever concept you want, in this case it would be a character. With a well-trained LoRA you shouldn't have any problem with character consistency.

    If you're trying to get started with generating locally, here's a tutorial for Comfy UI.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 11 months ago (1 child)

    A big note that it's nigh impossible to put two separate characters in one image, as the model has no idea who you are referencing. You can see this in all image models how they usually only have one central person, or two people who look oddly alike. Only way to get two or more characters would be with complex workflows to inpaint certain characters.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
    • [–] 4 points 11 months ago (4 children)

    My limited experience is that stable characters across a number of images are a weakness today, and I wouldn't be confident that genAI is a great way to go about it. If you want to try it, here's what I'd go with:

    • If you can get images with consistent outlines via some other route, you can try using ControlNet to do the rest of the image.

    • If you just need slight variations on a particular image, you can use inpainting to regenerate the relevant portions (e.g. an image with a series of different expressions).

    • If you want to work from a prompt, try picking a real-life person or character as a starting point, that may help, as models have been trained on them. Best is if you can get them at once point in time (e.g. "actor in popularmovie"). If you have a character description that you're slapping into each prompt, only describe elements that are actually visible in a given image.

    I've found that a consistent theme is something that is much more achievable, in that you can add "by " to your prompt terms for any artist that the model has been trained on a number of images from. If you're using a model that supports prompt term weighting (e.g. Stable Diffusion), you can increase the weight here to increase the strength of the effect. Flux doesn't support prompt term weighting (though it's really aimed at photographic images anyway). It's possible to blend multiple artists or genres as prompt terms.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] 3 points 11 months ago* (last edited 11 months ago) (3 children)

    Eyeballing here. I'm a learner and have hardly used this.

    Would a checkpoint model (right term?) achieve consistency, built from a specific set of pictures?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 4 points 11 months ago*

    If you have or can create a LoRA trained on images of the character you're presenting, that may be helpful. Or if you have a checkpoint model trained on that character. Would be like having a character that the base model is trained on.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 7 points 11 months ago* (last edited 11 months ago) (3 children)

    Qwen edit can take an image as a sample, and work with that. “The character in a victorious pose” would get whatever character you have, and reproduce it in a victorious pose. Couple of examples:

    And a little janky because it IS generative AI after all…

    Edit: and a bonus screenshot showing how little effort I had to put into this lol

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] [S] 3 points 11 months ago (2 children)

    Is there a way to run this in a Linux-based system?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 11 months ago

    Yes-ish. The base is Draw Things and the relevant bits are https://github.com/drawthingsai/draw-things-community?tab=readme-ov-file#cuda-capable-linux that isn’t too difficult to setup. The app with the pretty interface is Apple only (the developer one day decided to cram the full 1.5 on his iPhone and that was the start of this. The app has feature parity between the iOS, iPad and Mac versions, the gRPC server is “just” the generation parts decoupled from the app) but there’s a Comfy plugin to use the server.

    BTW on Apple’s hardware Comfy is poorly optimized, while Draw Things is optimized. The iPhone XR is the oldest hardware capable of on device generation, and (with the right settings) could do a SDXL 1024x1024 generation. 13 minutes mind you for 8 steps, but also 3gb of total system memory. On the other hand, the iPhone 17 Pro is a third of the speed of my RTX 3060. There’s also a friendly Discord, and the dev clearly enjoys adding support for new, cool models because he’s quick at it but doesn’t share roadmaps of any kind.

    Yeah. I really, really like that thing.

  • permalink
  • fedilink
  • source
  • parent