How to serve your pages as Markdown for AI agents

An AI assistant that reads your website gets HTML: menus, scripts, footer, a lot of noise around the text it cares about. A Markdown version of each article gives it only what matters: the title, the summary and the content. On this site, adding .md to the URL of an article or project returns it. This is how it is done.

If you want to see the result, try it with this very article: /en/serve-your-pages-as-markdown-for-ai-agents.md.

The three pieces

1. A rewrite in the proxy: any URL ending in .md is sent to an internal route.
2. A route handler that finds the content and answers with text/markdown.
3. A function that converts the content to Markdown.

Piece 1: the proxy

In proxy.ts, before the language logic:

// Markdown version for agents: /<article>.md and /projects/<slug>.md
if (pathname.endsWith('.md')) {
  const name = rest.at(-1)?.slice(0, -3);
  const section =
    rest.length === 1 ? 'blog' : rest.length === 2 ? matchSegment(locale, rest[0])?.internal : undefined;
  if (!name || (section !== 'blog' && section !== 'projects')) return NextResponse.next();
  return NextResponse.rewrite(new URL(join('md', locale, section, name), req.url));
}

Articles live at the root (/my-article), so a URL with a single segment is an article (blog). With two segments, the first indicates the section, for example projects. Whatever doesn't fit goes on its normal way.

Piece 2: the route handler

In app/md/[locale]/[section]/[slug]/route.ts:

const markdown = (body: string) =>
  new Response(body, {
    headers: {
      'Content-Type': 'text/markdown; charset=utf-8',
      'X-Robots-Tag': 'noindex',
      'Cache-Control': 'public, s-maxage=3600, stale-while-revalidate=86400',
    },
  });

export async function GET(_req: NextRequest, { params }: Ctx) {
  const { locale, section, slug } = await params;
  if (!isLocale(locale)) return new Response('Not found', { status: 404 });

  if (section === 'blog') {
    const post = await getPost({ slug, locale });
    return post ? markdown(postMarkdown(post, locale)) : new Response('Not found', { status: 404 });
  }
  // …the same for projects
}

Two decisions:

• X-Robots-Tag: noindex. The Markdown version shouldn't compete with the HTML page on Google. It is for machines, not for the search engine.
• One-hour public cache. The text rarely changes, and an agent requesting many pages shouldn't hit the database every time.

Piece 3: from Lexical to Markdown

Our articles' content lives in Payload, which uses the Lexical editor and stores it as a JSON tree. You have to walk it and write Markdown. Our postMarkdown function builds the document:

export function postMarkdown(post: Post, locale: Locale): string {
  return join([
    `# ${post.title}`,
    `> ${post.excerpt}`,
    [
      `URL: ${SITE_URL}${postPath(locale, post.slug)}`,
      `${category.title} · ${post.publishedAt?.slice(0, 10)} · ${author.name}`,
    ].join('\n'),
    lexicalToMarkdown(post.content.root),
  ]);
}

Title, summary as a quote, a line with the original URL, the category, date and author, and then the body. The original URL lets whoever reads the Markdown version know where the canonical one is.

lexicalToMarkdown walks the nodes and returns headings, quotes, lists, tables, images, links and code blocks with their language. If your content is in Markdown from the start (for example, a Markdown-based CMS), you skip this piece and return the file.

A complement: llms.txt

Besides the .md versions, we serve llms.txt and llms-full.txt at the root: text indexes with the site description and links to services, projects and articles. It is a recent convention, and there is no guarantee every agent reads it. It costs little to maintain, because it is generated from the same database.

What it doesn't do

This doesn't guarantee any model will cite you or improve your position on Google. What it does is make reading easier for whoever decides to read you, and it is a small job: a proxy, a route and a converter.

If you are preparing a site to be read by machines, a technical SEO audit in 30 minutes will also help.