Thinking on this more, said llama freeware would probably also need to be able to understand images too.
GPT-4 can actually understand and caption images pretty accurately, which is probably why Dalle-3 can generate well. So it would need to have image recognition capabilities to, it would also allow higher img2img capabilities.
Extending this, if this second "Prompt AI" can understand both text and images, to be able to caption it too. You can probably use it to train itself too. Similar to CLIP or BLIP.
Another thing is perhaps adding a GAN to it. For example, Style-GAN-Human based off the research is way better at creating humans than diffusion models.
StyleGAN-Human: A Data-Centric Odyssey of Human Generation
stylegan-human.github.io
Mainly hands look like hands, unlike Diffusion models.
So I'm wondering if you can connect it together in a pipeline like Prompter AI(understands prompt)->Diffusion(generates broad conceptual image based on prompt)->GAN(draws over with precision based on prompt and SD image)
This way if you want a realistic human, it can do it with good hands.
However problem is that GAN's unlike diffusion models are very specific, Diffusion models can take a concept and apply it to a lot of things (like john wick is a cartoon style). While GAN's can only generate a specific thing, a GAN trained on realistic faces can't make a cartoon face.
So maybe a pipeline where we switch GAN and Diffusion would work.
Prompter AI(understands prompt)->GAN(draws precise image based on prompt)->Diffusion(draws over precise image with stylized art based off prompt and GAN image)
However there are base GAN's capable of generating different concepts. So perhaps its possible to use StyleGAN3 maybe that can work together with the Diffusion model.
only problem is hardware space, and how much ram it would take to run what's basically 3 dnn's at the same time lol.