seo-auditRuns a technical SEO audit of a live website and its codebase from the terminal and reports findings with evidence, ranked by how much each one keeps pages out of search results. Use when a user asks for an "SEO audit", "technical SEO check", "why did our traffic drop after the migration", "why isn't Google indexing these pages", "check robots.txt, sitemap …
Independent GitHub snapshot · Automated classification · Original documentation governs
A page reaches search results only if a crawler can fetch it, it answers with HTTP 200, it carries content in a form the crawler reads, and no signal tells the engine to leave it out or prefer another URL. Most traffic losses come from one of those four breaking, usually after a redesign, a framework change or a configuration slip, and each one can be tested from a terminal.
This skill audits in that order: site-wide access, then a sample of pages, then rendering, speed and the source code that produced the fault. The output is a findings table in which every row has the command that showed the problem, so the user can rerun it after the fix. Rules quoted here follow Google Search Central as of October 2026; Bing and other engines read the same signals.
Find out or ask: the canonical host (www or bare domain), what prompted the audit (a drop with a date, pages missing from the index, a pre-launch check), which page types earn the traffic, whether the repository is available, and whether the user can export data from Search Console. Do not sign in to anything: ask for the "Page indexing" export and the Performance report for the affected dates.
Work in a scratch directory and keep every output file; they are the evidence.
SITE=https://larkfieldcycles.com
for u in http://larkfieldcycles.com/ https://larkfieldcycles.com/ http://www.larkfieldcycles.com/ https://www.larkfieldcycles.com/; do
curl -sS -o /dev/null -L --max-time 20 -w '%{http_code} hops=%{num_redirects} %{url_effective}\n' "$u"
done
curl -sS -o robots.txt -w 'robots %{http_code} %{content_type} %{size_download} bytes\n' "$SITE/robots.txt"
curl -sS --compressed -o sitemap.xml -w 'sitemap %{http_code} %{content_type}\n' "$SITE/sitemap.xml"
grep -o '<loc>[^<]*' sitemap.xml | sed 's/<loc>//' > urls.txt && wc -l < urls.txt
grep -o '<lastmod>[^<]*' sitemap.xml | cut -c10-19 | sort | uniq -c | sort -rn | head -3
How to read it:
| Result | Conclusion |
|---|---|
| The four host variants do not all end on one URL with 200, or one fails TLS | Duplicate or unreachable host. One hop to the canonical host is the target |
robots.txt returns 5xx | Google stops crawling the site for up to 12 hours, then falls back to the last copy it fetched. Treat as an outage |
robots.txt returns 404 or another 4xx (except 429) | Read as "no restrictions". Fine unless something was meant to be blocked |
robots.txt is HTML or larger than 500 KiB | Rules are ignored or cut off. A catch-all route is probably serving the app shell |
Disallow covers a path that should rank | Blocked from crawling. The longest matching rule wins and Allow wins a tie; paths are case-sensitive |
A blocked path also carries noindex | The noindex is never seen, because the page is not fetched. Pick one: unblock and noindex, or block |
Sitemap is missing from robots.txt, or the file is a sitemap index | Read the Sitemap: lines; for an index, fetch each child and concatenate the loc values |
| More than 50,000 URLs or 50 MB uncompressed in one file | Over the limit; split it |
Every lastmod is the same date | Google only trusts lastmod when it matches real changes; priority and changefreq are ignored either way |
Save this as seo_page_check.py. It fetches the HTML the server sends, without running scripts.
#!/usr/bin/env python3
"""Read URLs on stdin, print one JSON line of indexability signals per URL."""
import json, re, sys, urllib.error, urllib.request
from html.parser import HTMLParser
class Signals(HTMLParser):
def __init__(self):
super().__init__()
self.out, self.grab, self.buf = {"h1": 0, "hreflang": 0, "jsonld": []}, None, ""
def handle_starttag(self, tag, attrs):
a = {k: v or "" for k, v in attrs}
name, rel = a.get("name", "").lower(), a.get("rel", "").lower()
if tag == "title" and "title" not in self.out: self.grab, self.buf = "title", ""
elif tag == "script" and a.get("type") == "application/ld+json": self.grab, self.buf = "jsonld", ""
elif tag == "h1": self.out["h1"] += 1
elif tag == "meta" and name in ("robots", "googlebot", "description"): self.out[name] = a.get("content", "")
elif tag == "link" and rel == "canonical": self.out["canonical"] = a.get("href", "")
elif tag == "link" and a.get("hreflang"): self.out["hreflang"] += 1
def handle_data(self, data):
if self.grab: self.buf += data
def handle_endtag(self, tag):
if self.grab == "title" and tag == "title": self.out["title"] = " ".join(self.buf.split())
elif self.grab == "jsonld" and tag == "script":
try:
doc = json.loads(self.buf)
nodes = doc.get("@graph", [doc]) if isinstance(doc, dict) else doc
self.out["jsonld"] += [str(n.get("@type")) for n in nodes if isinstance(n, dict)]
except ValueError:
self.out["jsonld"].append("UNPARSABLE")
else: return
self.grab = None
for url in filter(None, map(str.strip, sys.stdin)):
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (compatible; site-audit)"})
try:
with urllib.request.urlopen(req, timeout=30) as res:
status, final, x_robots, raw = res.status, res.url, res.headers.get("X-Robots-Tag"), res.read()
except urllib.error.HTTPError as err:
print(json.dumps({"url": url, "status": err.code})); continue
except OSError as err:
print(json.dumps({"url": url, "status": 0, "error": str(err)})); continue
html = raw.decode("utf-8", "replace")
page = Signals(); page.feed(html)
words = len(re.sub(r"(?is)<(script|style)\b.*?</\1>|<[^>]+>", " ", html).split())
print(json.dumps({"url": url, "status": status, "final": final, "x_robots": x_robots,
"kb": len(raw) // 1024, "words": words, **page.out}, ensure_ascii=False))
Sample every template rather than every URL: the home page, a few of each page type, and a random slice of the sitemap. Add one invented path to test the error page.
{ head -5 urls.txt; shuf -n 40 urls.txt; echo "$SITE/audit-missing-page-check"; } | python3 seo_page_check.py > pages.jsonl
# URLs whose status, redirect, robots rule or canonical needs a look
jq -r 'select(.status != 200 or .final != .url or ((.robots // "") + (.x_robots // "") | test("noindex"; "i")) or (.canonical // "") != .url)
| [.status, .url, .final // "-", .robots // "-", .canonical // "-"] | @tsv' pages.jsonl
jq -r '.title // empty' pages.jsonl | sort | uniq -d # titles shared by several pages
jq -r 'select(.status == 200) | [.url, .h1, .words, .kb, (.jsonld | join(","))] | @tsv' pages.jsonl
| Signal | Conclusion |
|---|---|
A sitemap URL is not 200, or final differs from url | The sitemap advertises dead or redirecting URLs; list only final, canonical ones |
| The invented path returns 200 | Soft 404: every mistyped or deleted URL looks like a real page. Return a real 404 status |
noindex in robots, googlebot or x_robots | The page is excluded on purpose or by accident; the most restrictive rule found wins |
canonical missing | Acceptable, but add a self-referencing absolute canonical on indexable pages |
canonical names another URL | You are asking for that other URL to be indexed instead. Check it is 200, indexable and truly the same content |
canonical differs only by host, protocol or trailing slash | Conflicting signals; align links, redirects, sitemap and canonical on one form |
words far below what the browser shows | Content arrives through JavaScript; go to step 4 |
kb above 2000 | Googlebot reads the first 2 MB of an HTML file and of each script or stylesheet; content past that is not indexed |
h1 is 0, title empty or duplicated, description missing | Search engines pick their own title and snippet. There is no character limit to enforce; aim for a distinct, descriptive title per page |
hreflang above 0 | Each language version must list itself and every other version, with absolute URLs, and the listed pages must link back |
jsonld empty or UNPARSABLE | Hand over to structured data work; it affects appearance, not indexing |
For a migration, also take the old URLs with traffic (from the user's export or an old sitemap) and test each: curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' "$old". Every old URL should answer 301 or 308 straight to its closest equivalent. A 404, a 302, a chain of several hops, or a blanket redirect to the home page all lose the old page's standing.
chrome=$(command -v google-chrome || command -v chromium || command -v chromium-browser)
"$chrome" --headless --user-data-dir="$PWD/chrome-profile" --virtual-time-budget=8000 --dump-dom "$SITE/pricing" > rendered.html
curl -sS "$SITE/pricing" > raw.html
for f in raw.html rendered.html; do echo "$f words=$(wc -w < "$f") links=$(grep -o '<a [^>]*href=' "$f" | wc -l)"; done
If text, links, the canonical or the robots meta exist only in rendered.html, indexing depends on Google's render queue: it works, but later and less reliably, and other crawlers that do not run scripts see an empty page. Recommend server rendering or prerendering for pages that must rank. Two rules are strict: a noindex in the raw HTML may stop the page being rendered at all, so removing it with script does not work; and a canonical changed by script conflicts with the one in the raw HTML. Navigation must be real a elements with href; click handlers on other elements are not followed.
Good means, at the 75th percentile of real visits: LCP within 2.5 s, INP at or under 200 ms, CLS at or under 0.1. Field data decides; lab runs explain.
# Field data: 28 days of real Chrome users. Needs a free API key in CRUX_API_KEY.
curl -sS -X POST "https://chromeuxreport.googleapis.com/v1/records:queryRecord?key=$CRUX_API_KEY" \
-H 'Content-Type: application/json' -d "{\"origin\": \"$SITE\", \"formFactor\": \"PHONE\"}" \
| jq '.record.metrics | {lcp: .largest_contentful_paint.percentiles.p75, inp: .interaction_to_next_paint.percentiles.p75, cls: .cumulative_layout_shift.percentiles.p75}'
# Lab data: one simulated mobile load, no key needed
npx --yes lighthouse@13 "$SITE/" --only-categories=performance,seo --output=json --output-path=lh.json --quiet --chrome-flags="--headless=new"
jq '{perf: .categories.performance.score, seo: .categories.seo.score, lcp_ms: .audits["largest-contentful-paint"].numericValue,
cls: .audits["cumulative-layout-shift"].numericValue, tbt_ms: .audits["total-blocking-time"].numericValue}' lh.json
A 404 "data not found" from CrUX means too few visits for a record: say so and report lab numbers as indicative only. Lighthouse cannot measure INP; total blocking time is its nearest proxy. Speed is rarely the cause of a sudden drop; treat it as a tiebreaker.
grep -rnE 'noindex|X-Robots-Tag|canonical|Disallow' --exclude-dir={node_modules,.next,dist,build,.git} . | head -40
| Stack | Where the signals are produced |
|---|---|
| Next.js App Router | app/robots.ts, app/sitemap.ts, metadata or generateMetadata (alternates.canonical, robots), redirects() and headers() in next.config, middleware.ts |
| Astro, Eleventy, Hugo | site or base URL in the config, the sitemap integration, the head partial of the base layout, public/robots.txt |
| Hosting layer | vercel.json, netlify.toml, _redirects, _headers, nginx or Caddy config; staging rules and environment checks often leak into production here |
| WordPress | Settings, Reading, "Discourage search engines"; then the SEO plugin's canonical and noindex settings |
Lead with a three-line summary, then one table: severity | finding | evidence (command and output) | affected URLs | fix (file and change). Severity: Blocker means pages cannot be crawled or indexed; Major means the wrong URL is indexed or signals are split; Minor affects only how a result looks; Info is context. Close with what could not be checked and what the user should confirm in Search Console (URL Inspection on two fixed URLs, then the Page indexing report after a few weeks).
Request: "Organic clicks on larkfieldcycles.com fell about 40% since we moved from WooCommerce to a Next.js storefront on 8 September. Find out why." The user supplies the 200 top landing pages from before the move.
$ while read old; do curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' "$old"; done < old-top200.txt | cut -d' ' -f1 | sort | uniq -c
17 200
52 308
131 404
$ jq -r 'select((.canonical // "") != .url) | .canonical' pages.jsonl | cut -d/ -f3 | sort | uniq -c
41 www.larkfieldcycles.com
| Severity | Finding | Evidence | Fix |
|---|---|---|---|
| Blocker | 131 of 200 former landing pages return 404; /product/... moved to /bikes/... with no mapping | First command above | Add one-hop permanent redirects in next.config.js redirects() from each old path to its new equivalent; keep them at least a year |
| Major | All 52 redirects point to the home page | redirect_url is / for each | Redirect to the matching product or category instead |
| Major | Canonicals name www. while the site resolves to the bare domain | Second command; metadataBase in app/layout.tsx | Set metadataBase to the bare domain so canonical, sitemap and redirects agree |
| Minor | 38 product pages share one title | uniq -d on titles | Build the title from the product name in generateMetadata |
Request: "Search Console says most of kestrelbooks.io/guides is 'Excluded by noindex tag', but there is no noindex anywhere in our HTML. What is wrong?" The site is a Vite single-page app on Vercel.
$ for f in raw.html rendered.html; do echo "$f words=$(wc -w < "$f") links=$(grep -o '<a [^>]*href=' "$f" | wc -l)"; done
raw.html words=61 links=0
rendered.html words=1340 links=47
$ printf '%s\n' https://kestrelbooks.io/guides/vat-returns https://kestrelbooks.io/audit-missing-page-check | python3 seo_page_check.py | jq -c '[.status, .x_robots, .words, .canonical]'
[200,"noindex",38,null]
[200,"noindex",38,null]
| Severity | Finding | Evidence | Fix |
|---|---|---|---|
| Blocker | Every response carries X-Robots-Tag: noindex | x_robots above; a headers rule in vercel.json written for preview deployments has no host condition | Limit the rule to preview hosts or delete it |
| Blocker | Guides carry 38 words of text and no links before scripts run | words and the raw/rendered comparison above | Prerender /guides/* at build time so the article text is in the HTML |
| Major | Unknown paths answer 200 with the app shell | Second URL above | Serve a 404 status for unmatched routes |
| Info | No canonical on any page | canonical is null | Add a self-referencing canonical once pages are prerendered |
The report notes that the status label in Search Console will only change after Google recrawls, and asks the user to run URL Inspection on two guides after deploying.
robots.txt controls crawling, not indexing: a disallowed URL can still be listed without a snippet if others link to it. Use noindex on a crawlable page to keep it out.priority and changefreq do nothing, the site: operator is not a count of indexed pages, and appearing in AI Overviews needs no special file or markup beyond being indexed and eligible for a snippet.