How to Convert Any Webpage to Clean Markdown for LLMs

To convert a webpage to clean Markdown for an LLM, fetch the fully rendered HTML, extract the main content block, delete everything else (navigation, ads, scripts, cookie banners, footers), and then convert what's left to Markdown. Done right, it regularly shrinks token counts by 90% or more. Your model reads the article, not the menu. In short: - Raw HTML is mostly markup, scripts and boilerplate, not content. - Use a readability-style extractor to find the main article block. - Strip `script`, `style`, `nav`, `footer`, `aside` and `form` tags before converting. - Convert to Markdown so headings, lists, links and tables survive in a compact form. - On JavaScript-heavy sites, render the page in a headless browser before extracting anything. ## Why is raw HTML so expensive for a model? Open the source of almost any modern page and scroll. You'll find inline styles, tracking scripts, SVG icons, JSON blobs for analytics, and dozens of nested `div` tags before you hit a single paragraph. The text a person actually reads is frequently under 5% of the file. A language model can't skip that noise. Every class name and script tag gets tokenized, billed, and squeezed into your context window. Even worse, all that boilerplate dilutes the model's attention. Ask a model to summarize a product page and it may quote the "Related products" carousel rather than the spec sheet. Markdown fixes both problems. It keeps structure, like headings, lists, links and tables, while dropping almost all of the markup weight. Models also encounter plenty of Markdown during training, so they read it reliably. ## What do the token counts look like before and after? Here's a representative example: a typical 1,500-word news article, measured with OpenAI's `tiktoken` tokenizer. Your numbers will vary by site, but the pattern holds. - Raw HTML as downloaded: roughly 45,000 tokens - HTML after removing scripts and styles: about 14,000 tokens - Main content block only, still as HTML: roughly 4,500 tokens - Main content converted to Markdown: roughly 2,100 tokens That's a drop of more than 95%. Feed 1,000 pages a day into a RAG pipeline and that's the difference between a reasonable bill and a painful one. It also lets you fit twenty clean articles into the context space one raw page would fill. ## What are the steps to clean up a webpage? A solid url to markdown for llm workflow follows five steps. 1. Fetch the page with a realistic user agent, a sensible timeout, and redirect handling. 2. Render if needed. If the content loads through JavaScript, you'll need a headless browser like Playwright. 3. Extract the main block. Readability algorithms score elements by text density and pick the likely article. 4. Clean up stragglers. Remove scripts, styles, forms, share buttons and newsletter boxes that got past extraction. 5. Convert and polish. Transform the HTML into Markdown, collapse extra whitespace, and store the title and URL as metadata. ## Can you show a working example? Here's a minimal Python version using `requests`, `readability-lxml`, `beautifulsoup4` and `markdownify`. ```python import requests from bs4 import BeautifulSoup from readability import Document from markdownify import markdownify as md url = "https://example.com/article" # Fetch the raw HTML html = requests.get(url, timeout=10).text # Keep only the main article block main_html = Document(html).summary() # Remove anything that is not content soup = BeautifulSoup(main_html, "html.parser") for tag in soup(["script", "style", "nav", "footer", "aside", "form"]): tag.decompose() # Convert to Markdown with # style headings markdown = md(str(soup), heading_style="ATX") print(markdown) ``` This works well for static blogs and news sites. It won't help with pages that build their content in the browser, which brings us to the harder cases. ## How do you handle single-page apps and blocked requests? Single-page apps frequently send back a nearly empty HTML shell. Fetch one with `requests` and you'll get a loading spinner and a bundle of scripts, nothing your model can use. You have to render the page, wait for the content to load, and only then extract it. At volume, you'll also run into rate limits, bot detection and region-locked content. Running headless browsers and rotating proxies on your own is genuine infrastructure work. If that's not where you want to spend your time, you can hand the rendering, proxies and parsing to [the link.sc fetch and search API](https://link.sc), which returns clean Markdown or structured JSON from a URL. It includes an MCP server too, so agents in tools like Claude or Cursor can pull pages on their own. ## What mistakes should you avoid? The biggest mistake is stripping too much. Overly aggressive filters can wipe out tables, code blocks or captions that hold important facts. Spot-check a few outputs from each site type before you trust the pipeline. Other pitfalls worth watching: - Losing link URLs. Keep them when the model has to cite sources or follow links. - Leaving relative paths alone. Make them absolute so they work outside the original page. - Throwing away metadata. Title, author and date help the model weigh freshness and trust. - Never measuring. Count tokens before and after so you know your cleanup is actually paying off. ## Is it worth building this yourself? For a side project or a fixed list of static sites, the script above is plenty. Once you're dealing with thousands of unpredictable URLs, the edge cases pile up fast. Compare the engineering time with the per-request cost of a managed url to markdown for llm service. ## What to remember A model can only work with what you give it. Cut the page down to its content, preserve the structure in Markdown, and track the token savings. Clean input gets you lower costs and better answers.

Learn more: MCP server for Claude and Cursor

• https://link.sc

Link.sc is one Api to fetch and search the web, returning clean markdown and Json for Llms, Rag pipelines, and AI agents. link.sc