Text-to-image AI

T2i AI (text-to-image artificial intelligence) systems such as DALL-E, Midjourney, Motionleap sit around waiting for the input of prompts to get to work. When they receive the text prompt that you send (from an online console or within an app) e.g. ‘a red rose floating on the surface of a lake with mountains in the background‘, they dig into their knowledge base to start constructing an image. It’s important to understand this process does not work by finding images in the database to match your prompt. In fact, there are no images in the database as such; there are generalised ‘notions’ of things. If AI had to construct images from existing images, it would take a great of computing. Instead, results usually come in a few seconds. 

Imagine I ask you to paint a red rose. If you had first to find a suitable picture to copy, that would take you some time. Instead, as soon as you hear the words ‘a red rose’, some general notion of a red rose came to your mind. You could immediately start painting the rose. As you do so, you will be critical of what you’re doing: it may not look right, so you might scrub it and start again, this time the picture will look closer to your idea of a red rose and if you’re not an experienced artists, you may need a few tries and with each try you get closer to what is acceptable. That is very much how T2I works.

There are two parts (they call them ‘neural networks’) to the AI: the generator i.e. the bit that makes the image, and the discriminator, the bit that judges how well the ‘painting’ matches your prompt. So the generator will offer various examples of ‘rose’ building on its ‘knowledge’ of roses, having been trained on dozens of images of roses to ‘learn’ the general characteristics that also separate them from other flowers like dandelions, lilies or orchids. The discriminator then compares what the generator delivers with the prompt, asking for new options if not ‘satisfied’; or else it accepts the image and sends it to you, the prompter.

Now, what makes AI smart is that modern systems ‘know’ how things should look even if you’ve not been specific. For example, a rose floating on water will be reflected in it: the image will show a reflection even if you don’t specifically ask for reflections. In fact, if you do ask for a reflection, the words you use will push the next part of the prompt towards that back, and that could mean that part of the prompt is given lower prominence or even ignored. This can cause errors because, in fact, having reflections can be taken for granted but while mountains in the background is important to you, it is in fact an optional featuree. So a prompt asking for reflections and mountains may deliver the reflection but no mountains (see left image below).

There’s a whole lot more to writing prompts or seeds and different systems respond differently e.g. to spelling mistakes. Thus arises a fancy new skill ‘prompt engineering’. Furthermore, much depends on how well trained the AI is. I found, for instance, that DALL-E ‘knows’ the difference between the painting styles of Sonia Delaunay and Robert Delaunay. Some systems can’t distinguish between digital art and cartoon. We’re just at the beginning of learning what they can and cannot do. You see below two attempts at ‘a red rose floating on the surface of a rippling lake with fujiyama in the background’ from DALL-E and from DeepAI. Neither is better than the other; you could say they differ stylistically.