Our Articles Were Invisible to Half the Internet

We built an articles system, published to it, and looked at the result in a browser. It was fine. It was also, to a large share of the machines that matter, completely empty.

The defect

The page shipped as a shell. The browser loaded it, ran a script, fetched the article from an API, and injected the text into the page. Standard practice, works perfectly, and the article appeared in about eighty milliseconds.

But that is what happens after JavaScript runs. What the server actually sent was a document containing the word "Loading…" and an empty container.

Anything that reads the HTML without executing scripts — and that is a lot of things — saw a blank page with a spinner. Not a partial article. Nothing.

Why this is worse now than it used to be

The usual reassurance is that Google renders JavaScript, which is true. It is also no longer the whole question.

A growing share of the traffic that matters comes from systems summarising and citing pages rather than ranking them, and many of them fetch HTML and parse it — no browser engine, no JavaScript execution. Link preview generators behave the same way: paste a client-rendered article into a chat app and the preview shows whatever was in the shell.

So the failure mode is specific and modern: the page is fine for humans, invisible to the systems increasingly deciding what humans get shown.

How we found it

Not through a tool. By running curl against our own article URL and reading what came back.

That is the whole technique, and it is worth making a habit: look at what the server sends, not at what the browser shows you. View-source in a browser will happily show you the rendered DOM and hide the problem entirely.

The fix

The article page is now rendered on the server. A request produces a complete document with the body text already in it — no fetch, no injection, no spinner.

The complication is that our marketing site is static and our content lives behind an authenticated API in a different project. We did not want to rebuild the site to publish an article, and we did not want articles on a different domain.

The answer was a proxy rewrite: the marketing domain forwards those paths to a function in the other project, which renders the article and returns HTML. To a visitor and to a crawler, the article is served from the main domain as ordinary HTML. Publishing still takes seconds and requires no deployment.

What we got as a side effect

Server rendering forced per-article metadata, which we had also got wrong.

Client-rendered pages share one set of tags, because the tags exist before the content is known. Every article had the same title, the same description and the same preview image. Sharing any of them produced an identical, generic card.

Rendering on the server means the title, description, canonical URL, preview image and structured data are all built from the article being served. That was not the bug we set out to fix; it was arguably the more damaging one.

When client rendering is fine

This is not an argument against it. Most of what we build is client-rendered and should be.

The dividing line we now use: is this content, or is it interface?

Interface — a cart, a filter, a form, a dashboard — should be client-rendered. Nobody needs to cite your cart, and it is behind a login anyway.

Content — anything you want quoted, summarised, linked or found — must be in the initial HTML response. If a machine that does not run JavaScript would see nothing, then for a meaningful and growing share of the internet, there is nothing there.

The check

One command, on any page you care about being found:

curl -s https://your-site/some-article | grep "a sentence from the body"

If that returns nothing, neither does the crawler.