A Tiny Diffusion Model That Draws ASCII Art

Yadong Xie · November 12, 2026

A language model writes left to right, one token after the last. A text diffusion model doesn't. It starts with every position blank and fills them in over a few dozen passes, anywhere on the page, and on each pass it can see everything decided so far.

For writing, that is an odd fit. A sentence has an order, and a model that ignores it is working against the grain. A picture has none. ASCII art is text that has none either: a grid of characters that form an image. Filling it in everywhere at once is simply how you would draw it, and a few things follow:

• It is fast. A grid of 512 cells takes 32 passes, where writing it cell by cell takes 512.
• Every cell sees in all directions. A left-to-right writer has to commit to the top of a head before it knows where the feet will go. Here a cell can wait for what is below and to the right of it.
• Repainting is free. Filling in a hidden region, given the rest, is the training task itself.

ASCII art is also very little data. A model for it can be small enough to train on one graphics card in a couple of hours and to run in a browser tab.

So I trained one to draw it: 10 million parameters, a prompt in, a picture out on a grid of 32 × 16 characters, in about half a second. To give it a job I put it in a game. Diffusion Dungeon is a small ASCII roguelike, and when you meet a goblin, pick up a potion or walk into a crypt, the picture on the left of the screen is drawn then and there. None of them are stored.

It took eight versions of the model to get there, and the first seven were bad. This post goes through how the model works, how it is trained in two stages, and the one change that took it from bad to usable, which was to the data rather than the model.

A picture is 512 multiple-choice questions

The canvas is 16 rows of 32 characters. Every cell holds one of 19 characters: a space, the full block █, the half blocks ▀▄▌▐, the ten quadrant characters such as ▖▟▚, and three shades ░▒▓. A character cell is twice as tall as it is wide, so the grid is square on screen, and because a quadrant character splits its cell into 2 × 2, the picture is really 64 × 32 small pixels.

To the model, drawing is answering 512 questions with 19 choices each. There is a twentieth value, a mask, meaning "not decided yet".

The text goes through all-MiniLM-L6-v2, a frozen sentence encoder, which turns a prompt into up to 24 vectors of 384 numbers. A linear layer maps them to the model's width. Those 24 positions and the 512 cells then go into one bidirectional transformer as a single sequence: 8 layers, 8 heads, 320 wide, 10 million parameters. Each cell gets a row embedding and a column embedding, so the model knows where on the canvas it is. The output is 19 scores for every cell.

The first version had no pretrained encoder. It read the prompt as letters and had to learn English from a few thousand captions, and it never did. Swapping in MiniLM was the one architecture change that made a visible difference.

The objective: hide cells, guess them

Training follows MDLM, a masked diffusion model for discrete data. One training step goes like this:

1. Take a batch of 128 pictures and one caption for each.
2. For each picture pick a masking rate t between 0 and 1. The rates are spread evenly across the batch rather than drawn independently, which keeps the loss from jumping around.
3. Replace each cell with the mask with probability t.
4. Run the model, and compute cross-entropy on the masked cells only.
5. Divide each picture's loss by its t.

The division is the part that makes it a diffusion objective rather than plain masked-language-model training. Without it, a picture with 500 cells hidden contributes a hundred times more than one with 5, and the model only ever practices the start of a drawing.

The rate decides what is being practiced. Near 100% the model has nothing but the prompt, and has to learn what a spider looks like. Around 50% it has to complete shapes. Near 5% it sees almost everything and learns which exact character finishes an edge. Generation walks through all of these in order, so training has to cover all of them.

Two additions. For 30% of the pictures the mask is a rectangle rather than scattered cells, which teaches repainting a region while leaving the rest alone. And 10% of the time the caption is replaced by nothing, which is what classifier-free guidance needs later.

Drawing: reveal it in 32 steps

Generation starts with every cell masked. At each step the model predicts all of them, and a few are fixed in place. The number fixed follows a cosine schedule, slow at first and fast at the end, and a cell that has been fixed never changes.

The same picture at steps 0, 12, 22 and 32.

Which cells get fixed is random. My first sampler fixed the cells the model was most confident about, which is what most papers do. Here it collapsed: the model is always most confident about background, so it filled the canvas with background and had nothing left to hang a subject on.

Each step runs the model twice, with and without the prompt, and pushes the prediction away from the unprompted one (guidance 2, temperature 0.7). In the game, a mask that evolves by Conway's Game of Life is laid over the top, so a new picture grows out of the old one.

Where the pictures come from

There is no dataset of 32 × 16 block art, so I had a language model draw one, through the Codex CLI. I tried three ways of asking, and they fail differently.

The same three subjects from each method.

Any character. The first prompts allowed anything: box-drawing lines, slashes, shading. A model writes text one row at a time and never sees the result. Buildings and bottles, made of horizontals and verticals, come out fine. Anything with diagonals or curves drifts a little on every row, and by the bottom a spider's legs no longer meet its body. When it works it is the richest of the three, with tone and a setting. It works about two times in three.

Block characters only. Limiting it to blocks removed the drift, since there is no line to keep straight, only cells to fill. But the model wanted square pixels, and a cell is twice as tall as wide, so it typed everything as pairs: ██. Seventy-one percent of its runs of full blocks had even length. The pictures were accurate, at a fraction of the resolution the grid allows.

A pixel map. In the end the model doesn't write the final characters. It writes a 64 × 32 map using #, =, - and . for bright, mid, dim and background, and a program turns every 2 × 2 pixels into the matching character. For a symmetric subject it writes only the left half and the program mirrors it. The model decides what goes where, and the program picks the glyphs.

Stage one: pretraining on emoji

Drawing with a language model is slow, so the model first learns from pictures that cost nothing. Three open emoji sets and two icon sets give 19,399 small images that already come with names: an emoji's Unicode name and keywords, an icon's file name and tags.

Each one is cropped, scaled to between 70% and 94% of the frame, placed near the middle, and reduced to 64 × 32 pixels. Every 2 × 2 block becomes a character, the same conversion the pixel maps go through. Bright areas turn into █, darker ones into shades, transparency into space. Each image is converted several times at different sizes and positions, and mirrored unless its name says "left", "right" or "arrow". That makes 74,144 pictures.

Pretraining pictures, converted from emoji and icons.

This stage runs for 40,000 steps: AdamW at a learning rate of 3e-4, 500 steps of warm-up and a cosine decay to a tenth, weight decay 0.1, gradients clipped at 1. On one RTX 5080 in 16-bit precision that is 5.3 steps a second, a little over two hours.

Alongside the weights being trained I keep an exponential moving average of them, with a decay of 0.9995. The averaged copy smooths out the step-to-step jitter, and it is the one that gets evaluated, saved and shipped.

After pretraining the model can draw a heart, a smiling face and an umbrella. Ask it for a lighthouse and you get a rounded blob. It knows how block characters join into shapes, and very little about any particular thing.

Stage two: fine-tuning on the drawings

Fine-tuning starts from the averaged pretraining weights and uses the same code. Three things change: the data, the learning rate, which is halved so the new pictures adjust the model rather than overwrite it, and the length.

Loss on the training pictures and on held-out ones, for both stages.

The length is set by the held-out loss. A tenth of the drawings are kept out of training, and the model is scored on them every 500 steps. In pretraining that score was still falling at the end. In fine-tuning, with only 2,346 pictures, it bottoms out at step 3,000 and creeps back up while the training loss keeps dropping. From there on the model is memorizing its training pictures rather than getting better at drawing, so the checkpoint in the game is the one from step 3,000, ten minutes in.

How the held-out set is chosen matters more than its size. Each call to the drawing model returns four pictures that look alike. Split at random, every held-out picture would have three near-twins in training, and the held-out loss would keep rewarding memorization. I hold out whole calls.

Splitting the work this way is what made the rest of the project practical. Pretraining takes two hours and rarely needs repeating. A fine-tuning run takes ten to forty minutes, so I could change the data and see what it did within the hour.

What decided the quality: thousands of subjects, a few pictures each

Version 8 had what I thought was good data: 13,942 pictures of 4,673 subjects, drawn with the pixel-map method. I reviewed 100 of its outputs blind, mixed with other versions. Eleven were good. An apple came out as a blob, a key as a ring, a star as a smear.

My first guess was that the model was too small. The training log said otherwise. The loss on subjects the model had trained on was 0.292. The loss on subjects it had never seen was 0.300. If having seen "frog" barely helps you draw a frog, you haven't learned what a frog is. You've learned what pictures in general look like.

Each subject had two to four pictures, and each was a different view. That isn't enough to learn any of them. Small generative models that work have hundreds of examples of each thing.

So I ran the cheapest test I could. Twenty subjects, about sixty pictures each, 1,183 in all. Same model, same starting weights, same training code.

Same prompts, same seeds. Only the data differs.

In a blind review of 100 pictures from each, the share I rated good went from 7% to 74%. The new model wasn't copying its training set either: its outputs matched their nearest training picture in 69% of cells, and training pictures match each other in 71%.

This also changes how to read the first drawing method. I had dropped it after a model trained on it scored worst of all. That model had the same few pictures per subject, so I never learned whether the method or the sparsity was to blame.

One caption, one kind of picture

In the old data, each subject was drawn as a front view, another pose, a close-up and an action shot, and all four had the same caption. Given "a frog" and four unrelated targets, the model averages them.

Now the view is part of the caption. Most pictures of a subject are its most typical view, under the plain name. The rest are labelled.

Training pictures for one subject. The game only ever uses the first row.

The same reasoning applies to anything that varies between pictures. An early pretraining set gave every emoji a random background pattern. No caption mentioned backgrounds, so all the model could learn from that is that backgrounds are random. I regenerated the set on a plain background before training on it.

Measuring

For most of the project I compared versions by held-out loss. Every time the data changed, the held-out set changed with it, and the numbers meant nothing across versions. Version 6 scored 0.220 and version 8 scored 0.300, and version 8 was the better one on the game's prompts. Held-out loss is the right tool for deciding when to stop one run. It can't compare two.

Two measures replaced it.

The first is a blind review: the same prompts and the same seeds for every model, the pictures shuffled together so I can't tell which model drew which, and three grades. It takes ten minutes.

The second is a classifier. With sixty pictures of each subject, a small convolutional network can learn to name the subject of a picture: 94% on held-out drawings. Run on a model's outputs, it says how often the picture is recognizably the thing that was asked for, with nobody looking. I tried CLIP for this first. It has never seen art this small and ranked every close-up as a failure.

The final model

The game needs 36 subjects: ten monsters, seven items, and nineteen rooms and scenes. Each was drawn about sixty times, 2,144 pictures in forty minutes. For places, the prompt asks for a few key objects on an empty background rather than a room full of texture.

Two of the sixty or so training pictures for each subject.

Before and after, same prompts and seeds.

Blind, on 108 pictures from each model, the earlier version got 1 good, 35 passable and 72 bad. The new one got 101, 6 and 1. The classifier agreed: it recognized 14 of the earlier model's 108 pictures and all 108 of the new one's.

In the browser

The model is exported to ONNX and runs on onnxruntime-web with WebGPU: 41 MB of weights, about 15 ms a step, 32 steps a picture. The text encoder would be the larger download, at 90 MB, and the game doesn't need it. Its prompts are fixed, so their encodings are computed ahead of time and shipped as an 880 KB file, along with the sentence vectors the narration uses. The encoder is only fetched, through transformers.js, when you type a prompt of your own into the demo.

Two things I tried and dropped. Quantizing to int8 cut the file to 12 MB, but WebGPU has no kernel for the integer matmul, the session fell back to CPU, and a picture took 15 to 50 seconds. A 64 × 32 grid looked much better in tests and took 5.3 seconds a picture instead of 0.5, because attention over four times the cells costs far more than four times as much.

The narration under the map isn't generated. It's 240 lines written ahead of time, and the game picks one by comparing MiniLM sentence vectors of the situation and of each line, and taking a near match it hasn't used lately. The dungeon, field of view and pathfinding are rot.js.

What it can't do

It knows 36 things. Type anything else into the demo above and you get the nearest of the 36, or a blob. Open-vocabulary drawing is a different order of data, and a model this size won't get there by being trained longer.

Pictures of the same subject look alike. The sixty training pictures of a goblin are sixty similar goblins, because each drawing call starts from nothing and lands near the same idea. More variety means asking for it on purpose, pose by pose.

Adding a subject is a known cost now, though: write one line of art direction, wait for sixty drawings, fine-tune for ten minutes.

You can play Diffusion Dungeon in the lab.