You can get wonderful photographs with the exact composition and scene you want for all sorts of use cases.
Second hack.
Model the scene you want from low resolution geometry (boxes, spheres etc) for the composition. Then use that to ground the generated image (ie give it as reference) and ask the model to re-imagine it in the style you want.
You can use AI to transfer content between mediums and manufacture designs.
Succesfull artists don’t necessarily develop the pieces beyond conceptualization and editorial control.
See for instance Damien Hirsch. Or the academic painters with their own workshops.
AI can handle large amounts of routine tasks. Like converting an image to a 3D mesh. I don’t frankly see the difference between that and Damien Hirsch giving his craftsmen a task to deliver a piece.
If you talk to modern art critica you will end up with argumentation where the concept of the piece is what matters not the execution or it’s quality.
If original concept is all that matters very hard to see how AI generated steps could not be part of that pipeline.
Nobody is claiming getting random stuff from generic non-specific prompting counts as art.
But what if you do an original painting or drawing; then have AI convert it to 3D mesh and print it - and pass it off as art? That is a non-obvious situation.
Nice article. There's another option for pixel art that's recently been coming into vogue as tool usage improves in LLMs which is procedural graphics.
I actually experimented with creating a set of skills around rendering early AGI style EGA graphical scenes, and it worked surprisingly well.
So rather than storing full bitmaps, you instead store sequences of simple drawing instructions designed to recreate some specific scene - like you would an SVG.
As a result you end up sidestepping a lot of the problems with larger generative models, which can have issues with grid alignment, palette choices, and high-frequency noise.
This is very cool! I will try your method for background and tile creation in the asset-generation tool I've built [1]. Would your procedural method work for animations as well, like sprites?
How so? Judging from the examples the model does seem to take advantage of the language (creates and uses reusable elements) and AFAICT the language itself compiles down to "real" Sierra PIC commands so in theory it can be used with a Sierra PIC viewer/engine.
Yeah compiling to Sierra PIC is a unique advantage over asking the model for an image. But the claim im attacking is different. The claim in attacking is that you cant just tell it to produce images like this directly.
>As a result you end up sidestepping a lot of the problems with larger generative models, which can have issues with grid alignment, palette choices, and high-frequency noise.
All of these can be avoided with simple prompts and don’t require all this ceremony.
Convo then the artifact from the convo (sorry, apparently no way to share the pics but they are just captured from the artifact).
Thanks! Yeah, I started building it because I read an interview with the original creators of King’s Quest, Ken and Roberta Williams where they mentioned in passing that it was too expensive (in the 80s) from a storage perspective to store entire bitmaps for each scene, so instead they'd just store a series of very simple drawing instructions and render on demand.
I’m super excited to see more LLM-driven procedural graphics stuff in the future!
These models always create pixel art that looks like garbage because the people using it don't know enough about the medium to properly judge the output. This article is no exception. If you want pixel art for your project then learn or hire someone with that expertise
Are you saying people unfamiliar with the 'pixel arts' can't enjoy it their own way?
I don't think the author is posing as a professional. These models are like having a 3D printer at home. Very handy in a pinch, but what they make doesn't replace the real thing.
I am just a layperson when it comes to pixel art, but to me the example in the article is clearly garbage/low quality.
- the magenta background is still visible along the edges
- the color palette is all over the place
- random noise pixels of totally different colors are visible scattered around
- the lighting and shading doesn't look that good
There's plenty of real food that is as heavy and as salted, and as such no healthier, than junk food. But it actually tastes good. (I can't exactly put the finger on what makes an Italian pizza better than a US 7/11 one, but it's not about the former being healthier).
A substantial percentage of AI criticism can boil down to some form of gatekeeping. Maybe AI can't draw a true Scotsman but many will still enjoy the result.
But if the user thinks it looks fine, then won't most players think the same?
I just don't get the complaints about "use proper art" when it comes to games. Most of the time the game has some other, far more glaring, visual issues (anti-aliasing, clipping, icons-not-matching-items).
> But if the user thinks it looks fine, then won't most players think the same?
I have a suspicion that when you're looking at (and generating, because you then look at the results) AI generated images and text more you get desensitized and are more likely to say that something looks okay when to other people it won't.
Really? I'm noticing the opposite. First round of imagegen assets seem basically ok, but as time goes on you start to notice the similarities that are properties of the current crop of models and they start to annoy you.
People aren't using it wrong (although it's true most users have no taste), it's just the model devs don't care and generally have no idea about pixel art at all. The model just needs to be trained by someone knowledgeable. For example Retro Diffusion worked pretty well for its era - no mixels, proper palettes, etc.
Isn't that a good thing? Since these models generate garbage output of very low quality, they will never compete with the high quality generated only by human artists. Therefore this is a passing fad and human artists will always dominate.
The worst result would be if these models were any good vs being absolute slop. Then human artists would need to worry. But in fact there is no need to worry at all, as surely people (aside from a few people with bad taste) will clearly see the difference.
> Use the same style suffix and seed for every asset in a category so the batch doesn’t drift.
This part's a little confusing to me. What does "suffix" mean in this context? (Is it referring to just like the style-instructions boilerplate text you might include with each image?)
And then also: does locking a set seed really have an effect one way or other for consistency? Wouldn't changing the prompt text even if only slightly have just as much a randomization effect as changing the seed, making fixing it or using a random seed both functionally the same? Or is that not how the latest models work?
Great article -- and maybe this is just me -- but I do find the primacy of the online services interesting. Me and my 16gb vram card can get most of this done at home for "free."
Are control nets not being used much anymore? I haven't seen anything that provides a similar level of composition control, and yet I do not see them being referenced much at all.
They are used a lot, and modern edit models kind of support base cases - depth, canny, openpose - as a reference image. I mean they understand "use image 2 as depth map", result indeed have same depth, etc. But interpretation sometimes arbitrary to model.
For the tasks where several control nets should be adhered (depth and canny and segmentation) dreaded SDXL is still unmatched. SDXL also the only one handling influence weight for each control net normally, this ability completely vanished on modern models
Control nets in early post-SDXL models like ZIT or Flux kind of work too, but still much worse in quality when combined. I suspect pure diffusion architecture of SDXL is really favorable to mixing control signals with precision, DITs probably can be trained to similar level for mixing signals - but it is much harder and never seen in practice. At least every post-SDXL model I tried is trained with single one in mind, judging from my attempts. This rant is from personal expirience.
All of that still reads as a non-professional messing about at ten different things.
If we want to use a pixel art generator commercially we need 'practical guide' to making a small change to one animation, two months after creating the original character.
We need images generated in psd layers, with actual text using a font.
My guess would be that "professional" essentially translates to consistent, reliable and structured in the fashion a given industry expects. In this sense, everything AI/LLMs do is going to be "non-professional". Even when they solve frontier level math problems, the mathematicians agree "this is a terrible proof" (this goes against how the field does things). These things is always inconsistent/unreliable/unstructured but sometimes not too inconsistent etc to be useful. "How do use" tip sheets are also going to a list of ad-hoc observations, many of which its easy to get after X many tries with the generator - but that OK, essentially what you want is ways to save yourself time screwing around.
These thing may reach your standard consistent at some point, maybe. It's not important since before and after this point, they will require "messing around" to produce something useful but they can produce something if you need "good enough" but if you demand actually good, they won't arrive ever.
Related question: Whenever I try to run Image generation locally, be it with Flux or Qwen, I seem to be getting results that are about 1-2 years behind what ChatGPT can produce today - and take up to 5 minutes for a single image. Is this a hardware problem (running it on a decent windows gaming PC), a configuration problem or just the reality of what local models can deliver at the moment on consumer hardware?
If you're on a GPU and take five minutes for an image, something is wrong: either you're not actually running on GPU, or you're generating at way too high resolution. Open models work best at not far above 1024x1024.
> Whenever I try to run Image generation locally, be it with Flux or Qwen, I seem to be getting results that are about 1-2 years behind what ChatGPT can produce today.
At the risk of stating the obvious, you can't really compare local models to proprietary frontier models. For all we know, those models are heavily subsidized and have trillions more parameters than the smaller, modest ones you can run on a single GPU.
I've been running a prompt-adherence benchmark site [1] all the way back since the days of SDXL. If you look at the progress dropdown, I do think the gap is closing.
June of this year saw the release of some of the biggest open-weight bangers (Krea 2, Ideogram 4, and the kind of slept-on Boogu-Image 0.1), though the license terms can be kind of meh.
The best locally hosted model scores 8 out of 15, while the proprietary ones score 12 out of 15 on the text-to-image metric. If you want something that can reasonably decent for local gen while still being relatively small for modest GPUs, take a look at the recently released Moss-Image model [2]. But if you're expecting results at a level of something like GPT-Image-2.5 Sunburst, you're going to be disappointed.
Thank you for this, I'm a bit behind the times now on local generations and tend to use nano banana pro now when i generate although i used to use stable diffusion and automatic1111 way back, dabbled in everything around and Midjourney 3 onwards.
The website shows me clearly where things are at.
I expect I'd still like playing with Midjourney for more artistic styled work but it was the cost that drove me away.
Thanks, I've done that, but with mixed results. I've also tried comfyui wrappers and alternatives that try to hide the complexity and just set sensible defaults, but I'm beginning to suspect its ultimately a consumer hardware vs million dollar NVidia server question.
What are you trying to generate? I've also noticed that local models aren't as good as cloud models due to hardware but there are still ways of optimizing the process.
You can get results a billion times better than the horrible, disgusting look of ChatGPT in less than a couple of seconds with a model like Krea or MOSS, and ComfyUI. With just a 8GB VRAM GPU or so...
Just visit Civitai, look at the pages for the base models, filter for images with the ComfyUI workflow in them, then you can just download and drag and drop them onto your local ComfyUI app.
I've been using it to generate animated svgs. I made a breakthrough in asking it to create bones/rigging and movement with a python library and apply the result to the svg.
How much aesthetic freedom are you giving to opus?
The technical ability of anthropic models continue to steadily improve, but I still find the creative and compositional choices to fall far short of anything I would ever use.
I wrote this after a few weeks of experimenting with image models through APIs, mostly to work out what’s actually cheap and reliable. Every image in it came from those experiments and cost about a penny each. The bits I found most interesting were why “describe the camera” works better than “describe the view”, and how poorly general models handle exact structure like sprite sheets. Happy to answer questions.
Learn to draw.
Then ask the model to restyle the image.
You can get wonderful photographs with the exact composition and scene you want for all sorts of use cases.
Second hack.
Model the scene you want from low resolution geometry (boxes, spheres etc) for the composition. Then use that to ground the generated image (ie give it as reference) and ask the model to re-imagine it in the style you want.
Succesfull artists don’t necessarily develop the pieces beyond conceptualization and editorial control.
See for instance Damien Hirsch. Or the academic painters with their own workshops.
AI can handle large amounts of routine tasks. Like converting an image to a 3D mesh. I don’t frankly see the difference between that and Damien Hirsch giving his craftsmen a task to deliver a piece.
If you talk to modern art critica you will end up with argumentation where the concept of the piece is what matters not the execution or it’s quality.
If original concept is all that matters very hard to see how AI generated steps could not be part of that pipeline.
Nobody is claiming getting random stuff from generic non-specific prompting counts as art.
But what if you do an original painting or drawing; then have AI convert it to 3D mesh and print it - and pass it off as art? That is a non-obvious situation.
I actually experimented with creating a set of skills around rendering early AGI style EGA graphical scenes, and it worked surprisingly well.
So rather than storing full bitmaps, you instead store sequences of simple drawing instructions designed to recreate some specific scene - like you would an SVG.
As a result you end up sidestepping a lot of the problems with larger generative models, which can have issues with grid alignment, palette choices, and high-frequency noise.
https://github.com/scpedicini/agi-stylist
https://kq-styles.specr.net
[1] https://github.com/acatovic/ai-game-studio
>As a result you end up sidestepping a lot of the problems with larger generative models, which can have issues with grid alignment, palette choices, and high-frequency noise.
All of these can be avoided with simple prompts and don’t require all this ceremony.
Convo then the artifact from the convo (sorry, apparently no way to share the pics but they are just captured from the artifact).
Convo: https://claude.ai/share/78cd63c8-ffdd-4569-8de2-ac8ab5c5cfe0
Artifact: https://claude.ai/artifact/3M6xiKJKz1oDxTKGWi8FUi
The difference i see is the shading. Could easily tell it to do flat fills instead.
Regardless it has none of the supposed problems of AI generated pixel art.
I’m super excited to see more LLM-driven procedural graphics stuff in the future!
I don't think the author is posing as a professional. These models are like having a 3D printer at home. Very handy in a pinch, but what they make doesn't replace the real thing.
The problem isn't that they look shitty to pixel artists or experts, they look shitty for anyone with a bit of observation.
It's junk art the same way Mc Donald's is junk food.
I think we could do a bit better here with the explanation of what looks bad to you..
- the magenta background is still visible along the edges - the color palette is all over the place - random noise pixels of totally different colors are visible scattered around - the lighting and shading doesn't look that good
There's plenty of real food that is as heavy and as salted, and as such no healthier, than junk food. But it actually tastes good. (I can't exactly put the finger on what makes an Italian pizza better than a US 7/11 one, but it's not about the former being healthier).
McDonalds', over time, will make you less healthy.
AI slop over time will perhaps change aesthetics, but hard to chalk that up to anything other than personal preference.
I just don't get the complaints about "use proper art" when it comes to games. Most of the time the game has some other, far more glaring, visual issues (anti-aliasing, clipping, icons-not-matching-items).
I have a suspicion that when you're looking at (and generating, because you then look at the results) AI generated images and text more you get desensitized and are more likely to say that something looks okay when to other people it won't.
The worst result would be if these models were any good vs being absolute slop. Then human artists would need to worry. But in fact there is no need to worry at all, as surely people (aside from a few people with bad taste) will clearly see the difference.
This part's a little confusing to me. What does "suffix" mean in this context? (Is it referring to just like the style-instructions boilerplate text you might include with each image?)
And then also: does locking a set seed really have an effect one way or other for consistency? Wouldn't changing the prompt text even if only slightly have just as much a randomization effect as changing the seed, making fixing it or using a random seed both functionally the same? Or is that not how the latest models work?
For the tasks where several control nets should be adhered (depth and canny and segmentation) dreaded SDXL is still unmatched. SDXL also the only one handling influence weight for each control net normally, this ability completely vanished on modern models
Control nets in early post-SDXL models like ZIT or Flux kind of work too, but still much worse in quality when combined. I suspect pure diffusion architecture of SDXL is really favorable to mixing control signals with precision, DITs probably can be trained to similar level for mixing signals - but it is much harder and never seen in practice. At least every post-SDXL model I tried is trained with single one in mind, judging from my attempts. This rant is from personal expirience.
If we want to use a pixel art generator commercially we need 'practical guide' to making a small change to one animation, two months after creating the original character.
We need images generated in psd layers, with actual text using a font.
Sticking to the brand palette religiously.
These thing may reach your standard consistent at some point, maybe. It's not important since before and after this point, they will require "messing around" to produce something useful but they can produce something if you need "good enough" but if you demand actually good, they won't arrive ever.
It won't bridge the gap to gpt-image-2.5 though, and it's going to slow down the process further.
With a 4090, Qwen Image 2.1 Turbo takes around 15s per image, depending on the resolution. That's without the prompt enhancer.
At the risk of stating the obvious, you can't really compare local models to proprietary frontier models. For all we know, those models are heavily subsidized and have trillions more parameters than the smaller, modest ones you can run on a single GPU.
I've been running a prompt-adherence benchmark site [1] all the way back since the days of SDXL. If you look at the progress dropdown, I do think the gap is closing.
June of this year saw the release of some of the biggest open-weight bangers (Krea 2, Ideogram 4, and the kind of slept-on Boogu-Image 0.1), though the license terms can be kind of meh.
The best locally hosted model scores 8 out of 15, while the proprietary ones score 12 out of 15 on the text-to-image metric. If you want something that can reasonably decent for local gen while still being relatively small for modest GPUs, take a look at the recently released Moss-Image model [2]. But if you're expecting results at a level of something like GPT-Image-2.5 Sunburst, you're going to be disappointed.
[1] - https://genai-showdown.specr.net
[2] - https://huggingface.co/Comfy-Org/MOSS-Image
Just visit Civitai, look at the pages for the base models, filter for images with the ComfyUI workflow in them, then you can just download and drag and drop them onto your local ComfyUI app.
The technical ability of anthropic models continue to steadily improve, but I still find the creative and compositional choices to fall far short of anything I would ever use.