From GUI to NUI: How LLMs Change the Way We Interact

Yadong Xie · January 11, 2023

For years our phones could hear us almost perfectly and still understand very little. A voice assistant would transcribe "I'd like to pay for what's in my basket" word for word, then fail because nobody had programmed that sentence. The microphone was never the problem. What was missing was the step from words to what the user means, and that is the step large language models are good at.

My argument here is that LLMs complete the natural user interface, and that the way to use them is gradual: put the model where its uncertainty is affordable, and let it into existing products one layer at a time.

Three ways to talk to a computer

Every era of computing has had its own way for people to address machines.

The command line asks you to learn the machine's language exactly. The graphical interface maps the machine's functions onto things you can see, so you recognize instead of remember. A natural user interface goes further. Joshua Blake defines it as

an interface that is designed to reuse existing skills for interacting appropriately with content.

The important word is reuse. A natural interface asks you to learn nothing: you talk, point, touch and look the way you already do with people and things.

Hearing is not understanding

It helps to split any interaction into two halves.

Device interaction is how your input reaches the machine: speech recognition, multitouch, eye tracking, gesture recognition. Data interaction is how you get from that input to the information or function you want: a command, a search, a recommendation.

The two halves have moved at different speeds. Device interaction kept improving year after year: touchscreens became universal, speech recognition became good enough for dictation, and gaze and gesture tracking moved into consumer hardware. Data interaction is what defines each generation of dominant products.

Office and Windows organized the world into menus and commands. Yahoo and Google made search the way into information. TikTok and WeChat feeds made recommendation the way content finds you. Each shift changed what users had to know: first the command, then the right keywords, then nothing at all, as long as you were happy to receive rather than ask.

The voice assistant's failure sits right here. It had a modern device half attached to a command-era data half: you could say anything, but it could only act on sentences someone had anticipated. A language model removes that ceiling. It takes an open-ended sentence and turns it into intent, structure or an action, which is exactly the data interaction a natural interface needs. And language increasingly connects the other modalities too: the same models move from text to images, audio, video and instructions.

So the natural user interface was never blocked by sensors. It was blocked by the lack of a general way to understand what people mean. LLMs are that way.

Put the model where uncertainty is affordable

The catch is that a language model is probabilistic. Ask twice and you may get two answers, and some answers will be wrong. Products differ enormously in how much of that they can absorb.

A brainstorming tool benefits from variety. A summary or a search result tolerates imperfection, because a mostly-right answer is still useful and the reader can check it. A payment, a tax return or a dosage cannot tolerate it at all: a plausible wrong answer is worse than no answer. LLMs reach products in that order, from the forgiving to the strict, and the strict ones only work when the model is kept away from the part that has to be exact.

A tax product shows how to arrange that.

The model does the conversation, where its flexibility is the whole point. It then turns a messy conversation into structured inputs, where small errors can be caught by validation and by the user. The calculation, where the answer must be right, stays in an ordinary deterministic engine. The natural interface wraps the product; it doesn't replace its core.

This is the design question for any AI feature: not "can the model do this?" but "what happens when the model is wrong here, and who notices?"

Upgrade in place

The second lesson is that nobody should rebuild their product as a chat box. People already know the interfaces they use, and the cheapest learning curve is none. The move toward a natural interface works better as a series of upgrades, each one handing a little more of the step from intent to action to the model.

Keep the existing interaction and add to it. Google Colab's AI coding features put a "Generate" button beside "+ Code" and "+ Text". You describe what you want and the code lands in an ordinary cell. The notebook is unchanged; there is simply a new way to fill it.

Let the model understand the interface that already exists. Google Research showed how to enable conversational interaction on mobile by handing an LLM the screen's view hierarchy with a few examples. With no change to the app, the model can answer questions about a screen, summarize it, or work out which element an instruction refers to.

Generate on demand. At the far end, what the user needs is built from the request itself. Early agent demos already turn plain step-by-step instructions into a working backend endpoint, with code, environment variables and deployment. Applied to the front end, the same idea means an interface assembled for each request.

Each step is useful on its own, and none of them asks users to give up what they know.

A web page you can talk to

To see how little it takes, I built a small shop page you can speak to. You press the microphone and say what you want; the page decides what that means and shows it.

"I want to check out" brings up the payment view. "I want to buy iPhone" brings up the cart. Neither sentence is a command the page was told to expect word for word. The page only knows a short list of actions and what each one is for; the model decides which one the user meant.

The interesting part is how ordinary the code is. In sketch form, the page describes its actions as data:

const actions = [
  { name: "checkout", description: "when user want to checkout" },
  { name: "buy", description: "when user want to buy something" },
  // one entry per thing the page can do
];

function intentPrompt(utterance: string) {
  return `Pick the action that matches the shopper's request.
Answer with that action as JSON and nothing else.

Actions:
${JSON.stringify(actions, null, 2)}

Request: ${utterance}`;
}

and the rest is plumbing: speech in, model in the middle, a view out.

const recognition = new webkitSpeechRecognition();
recognition.onresult = async (event) => {
  const utterance = event.results[0][0].transcript;
  const reply = await llm.generate(intentPrompt(utterance)); // the in-browser model
  const action = actions.find((a) => a.name === JSON.parse(reply).name);
  render(action ?? "ask-again");
};
recognition.start();

Two details matter more than they look. First, the model's answer is checked against the list before anything happens. A small model will sometimes invent an action or return malformed JSON, and the page should fall back to asking again rather than doing something unexpected. That is the "where is uncertainty affordable" rule applied at the scale of one function.

Second, the two halves of the interaction run in different places. Speech recognition uses the Web Speech API, which in Chrome sends the audio to a server-based engine. The language model runs on the user's own GPU. Chrome shipped WebGPU in 2023, and Web LLM, from the MLC LLM project, uses Apache TVM to compile models for it. A quantized 7-billion-parameter Vicuna model loaded into the page generated about 20 tokens a second on the machine I tested with. The first visit downloads the weights, which takes a few minutes; after that they are cached, and the text of each request never leaves the browser.

Where this leaves us

The demo is a toy, but the shape is general: a product lists what it can do, a model maps what people say onto that list, and ordinary code does the rest. Three things follow from it.

The model is a tool, not the goal. Users want to finish a task, and the model earns its place only where it makes that easier; deterministic code stays wherever correctness matters.

Natural doesn't always mean faster. A sentence lowers what people must learn, but a familiar button can still beat it, so the best products will offer both and let people choose.

And we are not at a true natural user interface yet. The device half has been maturing for years, and LLMs finally give us the data half. The work now is putting the two together carefully.