Yadong Xie / 谢亚东
ProjectsLabWritingSaved|GithubLinkedInX

ImgPilot

Creator · Lepton AI · 2023

ImgPilot screenshot

There's a canvas on the left and a painting on the right. Outline a jar and scribble two green stems into it, and the right pane fills in with a watercolor of flowers in a glass vase while your pencil is still moving. You don't press a button.

I created and built it at Lepton AI. The main constraint was latency. Image models were already good enough, but too slow for a feedback loop. A latent consistency model produces an image in four denoising steps instead of fifty, and four steps on a single A10 can keep up with a hand drawing on a canvas. So instead of writing a long prompt and hoping, you steer something you can already see.

A hand drawing a vase of leaves with an Apple Pencil on an iPad running ImgPilot in the browser, the right-hand pane repainting it as a watercolor as each stroke lands
Shot on an iPad in Safari, on the public site. The right pane re-renders on every stroke.

The canvas is Excalidraw. Every change exports it to a bitmap that goes to the model as the image-to-image input at strength 0.7, high enough to repaint everything and low enough to keep your composition. The prompt has two fields: what you're drawing and what it should be painted like. People can't come up with the second one on the spot, so it's a menu of five media and nine movements, and "oil painting, impressionism" takes two clicks.

The back end is a Lepton photon, which puts the model, its dependencies, and its resource shape in one Python file. Like Elmo Chat and Lepton Search, ImgPilot was made to show what could be deployed on Lepton.

The hosted service was shut down in June 2024 and imgpilot.com is offline. The repository, front end and photon both, is still up and runs on your own GPU.

GitHub — leptonai/imgpilot

←Lepton SearchtryEmoji→