Building a Chrome Extension with LLMs

Yadong Xie · June 1, 2024

Elmo is a Chrome extension we built at Lepton AI. Open it on a web page, a YouTube video, a PDF or a Google Doc, and it summarizes what you're looking at and answers questions about it, in your language, with no account.

The model behind a product like this is the part you control least. You can pick it, but you can't change how it thinks, and in a few months something better will replace it. Building Elmo taught me that answer quality mostly depends on what sits around the model: what you feed it, how you ask, what you retrieve for it, and where it runs. This post is about those parts.

One pipeline behind every feature

Summaries, questions and translations all go through the same path.

The page is turned into clean, LLM-friendly content in the browser, sent to the model with a prompt, optionally enriched with search results, and the answer is rendered next to the page. Each box is a place where quality is won or lost, and only one of them is the model.

Don't fight the model

Here are two summaries of the same Wikipedia article, produced by two different models with the same prompt.

The card on the left gets the start date wrong, garbles "Yemen" into "emen", and leaves a phrase in English in the middle of a Chinese sentence. The card on the right has none of these problems. Nothing about our code changed between them; only the model did.

That is the most useful thing I can say about choosing a model: many problems you might spend a week engineering around simply disappear with a stronger model. Language mixing, ignored formatting instructions, short context windows and shaky facts all got better with each generation. Workarounds for one model's weaknesses age fast.

So treat the model as a replaceable part. Watch the public rankings — the LMSYS Chatbot Arena ranks models by blind human votes between pairs of answers, and the Open LLM Leaderboard scores open-weight models on standard benchmarks — but don't trust either for your product. Keep a few dozen real pages and questions from your own users, run every candidate model against them, and switch when one wins.

Most of the quality is in the input

A model can only summarize what it receives, and a raw web page is mostly not content: navigation, ads, cookie banners, related links and scripts. Feed that in and you pay for tokens that make the answer worse.

Elmo extracts the content in the browser before anything goes to the model: the article text, a video's transcript, the text of a PDF. Doing it on the client has three advantages. It's cheap, because the request shrinks to what matters. It works on pages a server could never fetch, behind the user's login. And it gives you a chance to keep structure the model needs — a table extracted as a table can be read; the same table flattened into a run of numbers can't.

Prompts are product code

Write the prompt in the user's language

An English prompt ending in "answer in Japanese" is easy to write, but on a long input the model tends to drift into the language of the page. It is more reliable to translate the instructions themselves, keep one prompt per language, and pick it from the user's locale:

function summaryPrompt(language: string, context: string) {
  switch (language) {
    case "ja":
    case "ja-JP":
      return `一歩ずつ進めてみましょう。私が提供する内容を読んで、次の作業を慎重に実行してください。
ステップ 1: このコンテキストを 1 つの文に要約します。
ステップ 2: コンテキストの内容を要約し、要約を作成します。
ステップ 3: コンテキストの要点。
...
${context}`;
    default:
      return `Go step by step, read the content I provide, and perform the following actions:
Step 1: Summarize this context in one sentence;
Step 2: Expand the summarization and write an abstract of a few sentences;
Step 3: Additional bullet points highlighting the opinions conveyed in the context.

# Summary
<one sentence summary here>

# Abstract
<abstract here>

# Opinions
<additional bullet points highlighting the opinions here>

${context}`;
  }
}

The output template is translated too, so the headings come back in the reader's language along with everything else. That is how the cards above have Chinese section headings on top of an English source.

Make the model cite

When the answer comes from search results, the prompt should do more than ask a question: it should make every claim traceable. Lepton Search, the open-source search demo we built on the same stack, numbers each result before it goes in and asks the model to tag sentences with those numbers.

The whole mechanism is in the prompt:

You are a large language AI assistant built by Lepton AI. You are given a user
question, and please write clean, concise and accurate answer to the question.
You will be given a set of related contexts to the question, each starting with
a reference number like [[citation:x]], where x is a number. Please use the
context and cite the context at the end of each sentence if applicable.

Your answer must be correct, accurate and written by an expert using an unbiased
and professional tone. Please limit to 1024 tokens. Do not give any information
that is not related to the question, and do not repeat. Say "information is
missing on" followed by the related topic, if the given context do not provide
sufficient information.

Please cite the contexts with the reference numbers, in the format [citation:x].
If a sentence comes from multiple contexts, please list all applicable
citations, like [citation:3][citation:5]. Other than code and specific names and
citations, your answer must be written in the same language as the question.

Here are the set of contexts:

{context}

Remember, don't blindly repeat the contexts verbatim. And here is the user question:

The front end turns each [citation:x] into a link like the ones in the screenshot. Readers can check every sentence, and the "information is missing on" line gives the model a sanctioned way to say it doesn't know, which is cheaper than any hallucination filter.

You don't have to invent these patterns yourself. Google's Gemini prompt gallery, Anthropic's prompt engineering guide and the community Prompt Engineering Guide collect the ones that work.

Retrieval fails in known ways

Retrieval-augmented generation puts a lookup in front of the model: the request is used to find relevant documents, and the model answers from them.

In Elmo the page is the main document, and search fills in the background the page takes for granted. The idea is simple, but RAG systems fail in a small number of predictable ways. The paper Seven Failure Points When Engineering a Retrieval Augmented Generation System maps them, and Wenqi Glantz extended the list to twelve pain points with proposed fixes. Six of them came up again and again for us.

Missing content. The answer isn't in what was retrieved, and the model invents a plausible one. The citation prompt's escape hatch is one defense; cleaner input is the other.

Too much context. The right passage is there but buried, and the model overlooks it. Reranking and trimming help more than a longer context window.

The wrong source format. The answer was in a table, a list or an image, and extraction flattened it. This is why the parser matters as much as the prompt.

The wrong output format. You asked for JSON or a fixed template and got something else. Use JSON mode where the API has one, validate against a schema, and tolerate small deviations.

Instability. Rate limits, timeouts and outages are normal. Keep a fallback model behind a router such as OpenRouter.

Security. A web page can contain text written to hijack the model. Treat page content as data, never as instructions, sanitize what you render, and consider a guard model such as Llama Guard or a framework like NeMo Guardrails.

Some of it moves into the browser

Everything so far assumes the model runs on a server. At Google I/O 2024 the Chrome team announced built-in AI: Gemini Nano running inside Chrome, exposed to pages and extensions through JavaScript, with an early preview program for developers.

The exploratory Prompt API sends free-form prompts to Gemini Nano. Task APIs, such as translation, are built for one job and backed by models tuned for it. Here is the Prompt API in the preview build, asked to translate a sentence from the built-in AI proposal itself into Chinese:

Three lines of code, no server, no API key, and the page content never leaves the machine. The output is also a fair picture of what a small on-device model can do. The sentence is translated, but "the Chrome built-in AI team" comes out as something like "Chrome's internal AI team members", and "ship in Chrome" is read literally, as in shipping goods. Good enough to get the gist; not good enough to publish.

That sets the division of labor for an extension like Elmo. Short, forgiving tasks — a quick translation, classifying a page, rewriting a sentence — can move on-device, where they cost nothing and keep data local. Long summaries and questions that need search stay with a large model on a server. The pipeline doesn't change; what changes is where each box runs.

The API names in the preview have changed since, so check the current documentation before building on them.

What to take away

The model will keep getting replaced, by you or by the field. The parts that last are the ones around it: clean input, prompts written as carefully as code, retrieval designed for the ways it fails, and a deliberate choice of where each step runs. Spend your effort there.