Post-Redesign Technical SEO Cleanup: Fix Crawl Errors, Redirects & Indexing (Step-by-Step)

Every redesign feels like a win the day it launches — new look, faster feel, happier stakeholders. Then, a week later, organic traffic quietly drops. Pages that ranked for years vanish from Search Console. Support tickets mention “page not found.” This is the single most common (and most fixable) technical SEO emergency agencies see. Below is the exact audit-and-repair sequence we run on every post-redesign cleanup, in the order that actually stops the bleeding fastest.

Quick Answer (TL;DR) A redesign breaks rankings when the new site changes URLs, templates, or code without carrying over the old site’s crawl paths and signals. The fix, in order: (1) audit crawl errors, (2) close redirect loops and chains, (3) repair canonical tags, (4) clear indexation issues, (5) rebuild the XML sitemap, (6) restore page speed, (7) repair internal linking, (8) sanity-check robots.txt. Confirm each fix inside Google Search Console before moving to the next — not after.

Before You Start: What to Gather

This cleanup goes faster — and is far less stressful — if you collect these five things first:

• Google Search Console access to the live property (and the old one, if it’s separate)
• A full crawl of the OLD site pre-redesign (Screaming Frog export, or the last cached crawl) — this is your map of every URL that used to exist
• A crawl of the NEW site (Screaming Frog, Sitebulb, or similar)
• The redirect map the developer used (even a rough one) so you can check it against reality
• PageSpeed Insights / Core Web Vitals baseline from before the redesign, if you have it

Step 1: Audit Crawl Errors

 PROBLEM

Redesigns almost always change URL structure — folder paths, trailing slashes, file extensions, or entire taxonomies. Every old URL that isn’t accounted for becomes a 404, and Google keeps trying to crawl those dead URLs for weeks, wasting crawl budget that should go to your new pages.

 RECOMMENDED SOLUTION

1. Crawl the pre-redesign URL list (from your saved crawl or Wayback Machine) against the live site using Screaming Frog’s “List Mode.”
2. Export every URL returning a 4xx or 5xx status code.
3. For each dead URL, find the closest-matching live page and add a 301 redirect — never redirect everything to the homepage.
4. Re-crawl the same list after redirects are live to confirm zero unhandled 404s remain.

 CONFIRM IT WORKED

• In Search Console, go to Pages > Not found (404) under the indexing report — the count should trend down over the following 1–2 weeks, not stay flat.
• Use the URL Inspection tool on 5–10 of the previously-broken URLs and confirm they now show “Page is not indexed: Page redirected” with the correct destination.

Step 2: Fix Redirect Loops & Redirect Chains

 PROBLEM

In the rush to launch, developers often add a new redirect rule on top of an old one instead of replacing it — old-URL → interim-URL → new-URL, or worse, old-URL → new-URL → old-URL (a loop). Chains slow crawling and dilute link equity at every hop; loops stop Google from reaching the page at all.

 RECOMMENDED SOLUTION

• Crawl your full redirect map with Screaming Frog’s redirect chain report (Reports > Redirects > All Redirects).
• Flag any URL with more than one hop, and any URL that returns to its own starting point.
• Update the server or .htaccess/redirect rules so every old URL points directly (one hop) to its final destination.
• Re-crawl to confirm each redirect now resolves in a single 301 hop.

 CONFIRM IT WORKED

• Screaming Frog’s redirect chain report should show 0 chains over 1 hop.
• In GSC, spot-check the same URLs with URL Inspection — the “Google-selected canonical” should match the final destination page, not an intermediate one.

Step 3: Repair Broken Canonical Tags

 PROBLEM

New CMS templates or staging-to-production migrations frequently leave canonical tags pointing to the wrong place — a staging domain, a stripped-down AMP version, a parameter’d URL, or simply the wrong page entirely. When the canonical disagrees with reality, Google may choose to index the wrong URL, or nothing at all.

 RECOMMENDED SOLUTION

• Crawl the live site and export the canonical tag on every page (Screaming Frog: Directives tab).
• Filter for canonicals that don’t self-reference correctly: pointing to staging/dev domains, to HTTP instead of HTTPS, or to a different page.
• Correct the canonical tag at the template level so it’s generated dynamically and self-referencing by default, rather than hardcoded.
• Re-crawl and confirm every priority page’s canonical now points to itself on the live domain.

 CONFIRM IT WORKED

• In GSC, open URL Inspection on your top 10–20 pages by traffic and compare “User-declared canonical” to “Google-selected canonical” — they should match.
• Check the Page Indexing report for a drop in the “Alternate page with proper canonical tag” and “Duplicate, Google chose different canonical than user” categories over the next 1–2 weeks.

Step 4: Clear Indexation Issues

 PROBLEM

The most damaging redesign mistake is also the simplest: the staging site’s “noindex” meta tag or a blanket robots.txt Disallow rule ships to production. Whole sections — sometimes the whole domain — can quietly drop out of Google’s index within days.

 RECOMMENDED SOLUTION

1. View source (or use Screaming Frog) on a sample of pages across every template type and check the <meta name=”robots”> tag for a stray “noindex.”
2. Check robots.txt directly at yourdomain.com/robots.txt for leftover Disallow rules meant for staging.
3. Remove any unintended noindex/Disallow rules and confirm the change is live (not cached by a CDN).
4. For any pages Google has already dropped, use URL Inspection > Request Indexing once the fix is confirmed live.

 CONFIRM IT WORKED

• GSC’s Page Indexing report: the “Excluded by ‘noindex’ tag” count should fall, and “Indexed” count should recover toward your pre-redesign baseline over 2–4 weeks.
• robots.txt Tester (in GSC or via the robots.txt report) should show your priority URLs as “Allowed.”

Step 5: Rebuild a Clean XML Sitemap

 PROBLEM

Sitemaps generated during development often still reference old URLs, staging pages, or paginated/parameter URLs that shouldn’t be indexed — and if the new sitemap was never resubmitted, Google is still working from a stale map of the old site.

 RECOMMENDED SOLUTION

1. Generate a fresh XML sitemap from the live CMS (Yoast, RankMath, or your CMS’s native generator) so it reflects current URLs only.
2. Confirm the sitemap excludes noindexed pages, redirected URLs, and any parameter/duplicate variants.
3. Submit the new sitemap URL in Search Console under Indexing > Sitemaps, and remove or replace any old sitemap entries that 404.

 CONFIRM IT WORKED

• The Sitemaps report in GSC should show “Success” status with a “Discovered URLs” count matching your live page count.
• Open the sitemap URL directly in a browser and spot-check 5–10 links — every one should return a 200 status, not a redirect or 404.

Step 6: Restore Load Speed & Core Web Vitals

 PROBLEM

New designs frequently ship heavier: unoptimized hero images, additional JavaScript frameworks, web fonts, and third-party embeds (chat widgets, marketing pixels) all commonly added in the redesign process without a performance budget.

 RECOMMENDED SOLUTION

• Run PageSpeed Insights (and Chrome UX Report data if available) on your 5 highest-traffic templates — homepage, category, product/article page, etc.
• Compress and lazy-load images, especially the largest hero/banner images above the fold.
• Defer or remove non-critical JavaScript (chat widgets, unused sliders, redundant analytics snippets).
• Set explicit width/height on images and embeds to stop layout shift as the page loads.

 CONFIRM IT WORKED

• PageSpeed Insights should show Core Web Vitals in the “Good” range (green) for LCP, INP, and CLS on both mobile and desktop.
• GSC’s Core Web Vitals report (Experience section) should show URLs moving from “Poor”/“Needs improvement” into “Good” over the following 28-day rolling window — this report lags real-world fixes by design.

Step 7: Repair Internal Link Structure

 PROBLEM

New navigation menus, footers, and related-content modules often break the internal linking that used to connect your pages, orphaning content that once ranked well. Orphaned pages are harder for Google to find, crawl, and trust.

 RECOMMENDED SOLUTION

• Crawl the site and pull an “Orphan Pages” report (Screaming Frog, combined with a GSC/GA4 URL list of pages that get traffic but no longer receive internal links).
• Rebuild links to important orphaned pages from relevant category pages, navigation, or contextual in-content links.
• Confirm the new navigation and footer link to all key sections at a reasonable click depth (ideally 3 clicks or fewer from the homepage).

 CONFIRM IT WORKED

• Re-crawl and confirm previously orphaned priority pages now show at least one internal inlink.
• In GSC’s Links report, check that “Top linked pages” still includes your priority content — a sudden drop signals the internal link structure hasn’t been fully restored.

Step 8: Sanity-Check robots.txt

 PROBLEM

robots.txt is small enough to be an afterthought, but a single leftover Disallow rule from staging — or a new rule blocking CSS/JS the redesign now depends on — can stop Google from crawling or even rendering pages correctly.

 RECOMMENDED SOLUTION

• Open yourdomain.com/robots.txt and read it line by line — don’t assume it’s unchanged from before the redesign.
• Confirm it does not block /wp-content/, /assets/, or any CSS/JS folder the new theme depends on for rendering.
• Confirm the sitemap directive at the bottom of robots.txt points to the new, live sitemap URL.

 CONFIRM IT WORKED

• Use the URL Inspection tool’s “View Crawled Page” / rendered screenshot on a sample page — it should render visually the same as it does in a browser (not a broken, unstyled page).
• GSC’s robots.txt report (under Settings) should show the file was fetched successfully with no unexpected Disallow rules against key sections.

Step 9: Update Your llms.txt File

PROBLEM
Most redesign checklists stop at robots.txt and sitemaps — but if your site publishes an llms.txt file, that file almost never survives a redesign untouched. It typically still lists old URLs, references pages that were merged or removed, and describes a site structure that no longer matches reality. It’s not confirmed that AI crawlers currently act on llms.txt at all — no major platform has publicly stated it factors into crawling or citation behavior — but if it ever does get used, a stale file would be handing out the wrong map of your site. Keeping it accurate costs little and hedges against that possibility.

RECOMMENDED SOLUTION

1. Locate (or create) your llms.txt file at yourdomain.com/llms.txt.
2. Cross-check every URL listed against the new site structure — update, redirect-target, or remove entries for any page that moved, merged, or was deleted.
3. Update the plain-language site/section descriptions if the redesign changed navigation, categories, or core offerings.
4. If you maintain an llms-full.txt (expanded version with more content detail), regenerate it from the current live pages rather than patching the old one.
5. Add or update the reference to llms.txt in robots.txt if your setup calls for it.

CONFIRM IT WORKED

• Open yourdomain.com/llms.txt directly in a browser and manually click through 5–10 listed links — every one should resolve to a live, correct page (200 status, not a 404 or redirect chain).
• Compare the file against your current live sitemap — the two should describe the same site, not two different versions of it.
• There’s no confirmed way to verify an AI crawler has “read” or acted on the file, so treat this step as good hygiene rather than something with a measurable before/after — unlike the GSC-based confirmations in the other steps.

The Confirmation Timeline: What to Watch in GSC, Week by Week

Technical fixes are only half the job — confirming Google has actually seen and processed them is what closes the loop. Recovery isn’t instant; use this timeline to know what “normal” looks like at each stage.

Timeline	What to check in GSC	Healthy signal
Day 1–3	Coverage/Page Indexing report, sitemap status	Sitemap shows “Success”; no spike in Not Found (404)
Day 4–7	URL Inspection on 10–15 priority URLs	Priority URLs show “URL is on Google”
Day 7–14	Core Web Vitals report, Crawl Stats report	CWV trending to “Good”; crawl requests normalizing
Day 14–30	Performance report (clicks & impressions)	Impressions recovering toward pre-redesign baseline

Common Mistakes That Undo All of the Above

Mistake	Why it hurts	Quick fix
Redirecting everything to the homepage	Kills page-level relevance and link equity	Map old URL → closest matching new URL, 1:1
Leaving staging “noindex” on after launch	Whole site can drop from the index within days	Check the robots meta tag and robots.txt the day of launch
Forgetting to resubmit the sitemap	Google keeps crawling the old URL map for weeks	Submit the new sitemap; remove the stale one from GSC
Canonicals still pointing to the staging domain	Google may index staging instead of the live site	Search-and-replace the canonical domain sitewide, then verify in page source
No redirect testing before DNS cutover	Loops and chains surface only after go-live, at peak traffic	Crawl the full redirect map on staging with a crawler first

Frequently Asked Questions

How long does it take to recover rankings after a redesign?

Most sites see crawl and indexing signals stabilize within 2–4 weeks of the fixes above, provided redirects, canonicals, and indexation are corrected in the first few days. Ranking and traffic recovery typically follows 4–8 weeks after that, depending on the size of the site and how much URL structure changed.

Do I need to redirect every single old URL?

Yes, ideally. Any old URL that isn’t redirected returns a 404, which both frustrates users following old links or bookmarks and wastes crawl budget. A small number of genuinely obsolete pages can be left to 404 naturally, but every URL that carried traffic, links, or rankings should get a deliberate 1:1 redirect.

What’s the difference between a redirect chain and a redirect loop?

A chain is multiple hops to reach the final page (A → B → C); it works, but slows crawling and dilutes signal at each hop. A loop is when the chain circles back on itself (A → B → A), which never resolves and blocks Google — and users — from reaching any page at all.

Can a redesign cause indexing problems even if the URLs didn’t change?

Yes. Template-level issues — a stray noindex tag, a canonical pointing to the wrong version of a page, heavier JavaScript rendering that hides content from crawlers, or robots.txt rules blocking CSS/JS — can all cause indexing drops even when the URL structure itself stayed identical.

Does llms.txt need to be resubmitted anywhere, like a sitemap?

No — there’s no submission or “resubmit” step the way there is with GSC and XML sitemaps. AI crawlers simply re-fetch it periodically, so the fix is just making sure the file at /llms.txt is accurate and live; there’s nothing further to trigger.

Sources & Source Grading

In line with our editorial standard, every technical claim above is graded by source reliability:

 VERIFIED   Google Search Central documentation — URL Inspection tool, Sitemaps report, robots.txt specification, and Core Web Vitals thresholds (LCP/INP/CLS “Good” ranges).

 VERIFIED   Google Search Console Help Center — Page Indexing report categories (“Not found (404)”, “Excluded by noindex tag”, “Duplicate, Google chose different canonical”).

 VENDOR CLAIM   Screaming Frog documentation — List Mode crawling, redirect chain reporting, and orphan page detection methodology.

 DISPUTED   Exact recovery timelines (“2–4 weeks,” “4–8 weeks”) are practitioner heuristics based on typical case patterns, not a documented Google SLA — actual recovery speed varies by site size, crawl budget, and how quickly fixes are shipped.

DISPUTED — llms.txt is a proposal introduced by Jeremy Howard (Answer.AI, September 2024, llmstxt.org), not a web standard, and not an official crawling directive. No major AI platform (OpenAI, Anthropic, Google, Perplexity) has publicly confirmed their crawlers read or prioritize it. Maintaining it is a reasonable low-cost hygiene practice, but there’s no verified evidence yet that it changes AI citation or crawling behavior.

Don’t Want to Run This Audit Yourself? Alneeko Technologies runs this exact post-redesign cleanup for e-commerce and content sites as a fixed-scope technical audit — crawl diagnosis, redirect mapping, canonical repair, and a week-by-week Search Console confirmation report. Book a free audit call to get a prioritized fix list for your site within 48 hours.