How to Extract Links from a Webpage

Fetch the correct page version

Record requested and final URLs after redirects. Relative links must resolve against the final document base. Verify the response is HTML and contains expected content.

Render JavaScript only when it creates links essential to the task.

Collect anchors and useful context

Capture href, visible anchor text, image alt text inside linked images, relationship values, and location when useful. Treat fragment, email, telephone, JavaScript, and data URLs separately.

Do not assume every clickable element is a crawlable link; real destinations should use anchors.

Resolve and normalize URLs

Convert relative paths to absolute URLs, lowercase hostnames, remove default ports, and handle fragments consistently. Do not strip query parameters blindly.

Use a standards-based parser following the WHATWG URL Standard.

Classify links and relationship attributes

Define internal links by canonical host policy, including deliberate subdomains. Classify external, same-page, download, pagination, language, and utility links separately.

Capture nofollow, sponsored, ugc, noopener, and noreferrer rather than collapsing them into one flag.

Deduplicate without losing anchor context

Keep one destination-level view for inventory and placement-level records for anchor analysis. Store occurrence count and meaningful anchor variants.

Flag empty, generic, and misleading anchors for review rather than treating every repetition as an error.

Audit and export the link dataset

Find broken destinations, redirect chains, accidental nofollow values, weak anchors, and important pages receiving too few contextual links. Continue with the SEO audit checklist.

Export source URL, destination, host, class, anchor, rel values, count, and extraction time. CSV supports review; JSON preserves richer context.

Extract links with Python or JavaScript

Three routes, depending on whether the page needs a browser.

Python, static HTML. Fine when the links are in the served markup.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse

url = "https://example.com/"
html = requests.get(url, timeout=20).text
soup = BeautifulSoup(html, "lxml")

links = []
for a in soup.select("a[href]"):
    absolute = urljoin(url, a["href"])
    if urlparse(absolute).scheme in ("http", "https"):
        links.append({"url": absolute, "text": a.get_text(" ", strip=True),
                      "rel": a.get("rel", [])})

JavaScript, in the browser console. Useful for a one-off on a page you are already looking at, and it sees links added by scripts.

[...document.querySelectorAll("a[href]")].map(a => ({
  url: a.href,           // already absolute
  text: a.textContent.trim(),
  rel: a.rel,
}));

Without code. The free Link Extractor does the same work, including rel attributes and internal or external classification, and renders the page first where that is needed.

Whichever route you take, resolve relative URLs against the page URL rather than the document root: href="../about" means something different depending on where it appears.

Classify links for the actual analysis job

Define internal and external using the effective site boundary, not a simple string prefix. Decide how subdomains, alternate ports, international domains, CDN hosts, fragments, query parameters, and redirected destinations should be treated. Preserve the original href beside the resolved URL so normalization remains auditable.

• Group repeated destinations and count their occurrences.
• Keep distinct anchor text even when URLs match.
• Record rel values such as nofollow, sponsored, and ugc.
• Separate navigation, editorial, footer, image, and action links when location is available.

For SEO work, compare the inventory with the page purpose and identify missing contextual routes, generic anchors, broken destinations, and isolated pages.

Validate and monitor link inventories

Manually inspect samples from internal, external, fragment, redirect, and excluded groups. Check that URL resolution uses the final document base and that unsafe schemes never enter a crawler queue. If JavaScript creates essential navigation, compare static and rendered results before escalating every page to a browser.

Track total links, unique destinations, broken links, redirect chains, generic anchors, nofollow distribution, and newly introduced external domains. Use the Link Extraction API for repeatable inventories and connect findings to the broader on-page SEO audit workflow.

Common failure patterns and how to diagnose them

Start with the earliest failing layer. If a request does not return the expected page, inspect DNS, redirects, HTTP status, content type, firewall behavior, and access controls before changing extraction or metadata rules. If the correct document arrives but the result is empty, compare initial HTML with the rendered page and determine whether JavaScript supplies the missing information.

When only some fields are wrong, inspect the page source and identify which signal produced each value. A stale canonical, duplicated title, malformed JSON-LD block, relative URL, or CMS fallback should be corrected at its source. Avoid adding a special case for one URL when the same template defect affects a wider section of the site.

Finally, distinguish a deterministic failure from a recommendation. Invalid JSON, an unreachable URL, or a conflicting index directive can be proven. Content usefulness, ideal wording, and business priority still require human judgment. A professional report should show the evidence, explain the consequence, and avoid presenting a heuristic as a universal rule.

Production quality checklist

Validate the exact production URL, not only a CMS preview, local fixture, or isolated code sample. Confirm the requested URL, final URL, HTTP status, content type, and visible result agree with the page you intended to process. Test at least one normal case, one sparse page, one redirected URL, and one expected failure so the interface communicates limitations clearly.

• Keep source URLs and observation times with exported results.
• Use descriptive labels, headings, and error messages.
• Specify image dimensions and keep media files lightweight.
• Test keyboard access and narrow mobile layouts.
• Separate automatic checks from recommendations requiring judgment.
• Respect access controls, publisher policies, privacy, and applicable law.

After deployment, rerun the workflow against representative URLs and monitor for changes in output size, missing fields, status codes, response time, and template behavior. Keep a known-good result for comparison. When a check fails, fix the source template or data pipeline rather than hiding the warning in the interface.