How to Convert a Webpage to Clean Markdown

Decide what the Markdown must preserve

Markdown is effective for documentation, knowledge bases, migration, search, and RAG because it keeps hierarchy without browser styling. It is not a replacement for HTML when layout, forms, or interactive behavior matter.

Choose between the complete page and readable main content. Most content workflows need the latter.

Fetch and validate the source

Follow redirects, record the final URL, restrict protocols and content types, and set response limits. Verify that the response contains expected content rather than an access or consent screen.

Use browser rendering only when JavaScript creates essential text. Static HTML is the faster default.

Remove page chrome before conversion

Discard scripts, styles, navigation, repeated headers and footers, ads, cookie controls, and unrelated recommendations before mapping tags. Otherwise valid Markdown remains semantically noisy.

For article pages, follow the main article extraction workflow.

Map semantic HTML deliberately

Convert headings by document hierarchy, paragraphs to separated blocks, lists to list syntax, links to descriptive Markdown links, and code to inline or fenced code. Preserve block quotes and meaningful tables.

Do not infer heading rank from font size or normalize whitespace inside code examples.

Handle tables, images, and links

Convert simple data tables; retain HTML or structured data for complex tables that Markdown would misrepresent. Keep useful images with accurate alt text and omit decorative pixels.

Resolve relative links against the final URL and reject unsafe schemes. Use the Link Extractor for a dedicated link inventory.

Validate Markdown quality in production

Check heading order, list indentation, closed code fences, resolved links, and source coverage. Sanitize retained HTML before rendering it in a browser.

Version the converter, cache unchanged pages with content hashes, and test representative templates. Follow the API quickstart for implementation guidance.

Convert HTML to Markdown in Python

Two libraries do most of this work, and the choice matters more than it looks.

markdownify converts whatever HTML you hand it, so the quality depends entirely on removing navigation and footers first.

import requests
from bs4 import BeautifulSoup
from markdownify import markdownify

html = requests.get("https://example.com/article", timeout=20).text
soup = BeautifulSoup(html, "lxml")

for tag in soup.select("nav, header, footer, aside, script, style"):
    tag.decompose()

main = soup.select_one("article, main") or soup.body
markdown = markdownify(str(main), heading_style="ATX")

trafilatura identifies the main content itself, which is usually the better default for pages you have not seen.

import trafilatura

downloaded = trafilatura.fetch_url("https://example.com/article")
markdown = trafilatura.extract(downloaded, output_format="markdown")

Neither runs JavaScript. On a client-rendered page both return an empty shell, which is the point at which you need a browser or an API that renders before extracting.

The free Web Page Scraper handles the rendering decision for you and returns Markdown directly.

Review a Markdown conversion example

A documentation page may contain a breadcrumb, sidebar, article heading, paragraphs, a numbered procedure, code, a comparison table, and footer navigation. A clean conversion should preserve the article hierarchy and code exactly while excluding repeated navigation. Inspect the output as a reader would, not only as valid syntax.

• Heading levels should reflect the document hierarchy.
• List nesting must remain unambiguous.
• Code fences must close and preserve whitespace.
• Relative links should resolve against the final page URL.
• Complex tables should not be flattened into misleading rows.

For a source-text-only workflow, first isolate the article with the Article Text Extractor.

Make conversion reliable at scale

Store the requested URL, final URL, collection time, converter version, content hash, and warnings with the Markdown. Normalize predictable whitespace, but do not silently rewrite quotations, code, numbers, or link destinations.

Measure malformed output, unresolved links, empty results, conversion size, latency, and template-specific regressions. Sanitize any retained HTML before displaying it. When conversion supports a knowledge system, version documents so stale chunks can be removed rather than accumulated beside newer text.

Common failure patterns and how to diagnose them

Start with the earliest failing layer. If a request does not return the expected page, inspect DNS, redirects, HTTP status, content type, firewall behavior, and access controls before changing extraction or metadata rules. If the correct document arrives but the result is empty, compare initial HTML with the rendered page and determine whether JavaScript supplies the missing information.

When only some fields are wrong, inspect the page source and identify which signal produced each value. A stale canonical, duplicated title, malformed JSON-LD block, relative URL, or CMS fallback should be corrected at its source. Avoid adding a special case for one URL when the same template defect affects a wider section of the site.

Finally, distinguish a deterministic failure from a recommendation. Invalid JSON, an unreachable URL, or a conflicting index directive can be proven. Content usefulness, ideal wording, and business priority still require human judgment. A professional report should show the evidence, explain the consequence, and avoid presenting a heuristic as a universal rule.

Production quality checklist

Validate the exact production URL, not only a CMS preview, local fixture, or isolated code sample. Confirm the requested URL, final URL, HTTP status, content type, and visible result agree with the page you intended to process. Test at least one normal case, one sparse page, one redirected URL, and one expected failure so the interface communicates limitations clearly.

• Keep source URLs and observation times with exported results.
• Use descriptive labels, headings, and error messages.
• Specify image dimensions and keep media files lightweight.
• Test keyboard access and narrow mobile layouts.
• Separate automatic checks from recommendations requiring judgment.
• Respect access controls, publisher policies, privacy, and applicable law.

After deployment, rerun the workflow against representative URLs and monitor for changes in output size, missing fields, status codes, response time, and template behavior. Keep a known-good result for comparison. When a check fails, fix the source template or data pipeline rather than hiding the warning in the interface.