Technical SEO Services: What They Should Include for Google and AI Crawlers

For most of the last decade, technical SEO meant making a site easy for Googlebot. That is still the job. What has changed is the number of bots that matter. Buyers now ask ChatGPT, Claude, Perplexity, and Google's AI features to recommend providers and explain options, and the crawlers behind those tools behave differently from Google's.

This guide explains what technical SEO services should include, how AI crawlers change the scope, what an audit should deliver, and how to compare the proposals you get. It is written for marketing leaders who need to buy this work, not for developers who already do it. If you want the short definition first, see our technical SEO glossary entry.

What Do Technical SEO Services Include?

Technical SEO services cover six areas: crawl access, rendering, indexation, site structure, structured data, and page experience, plus ongoing monitoring. Each one is a gate. If a page fails any of them, the work above it (content, links, brand) cannot do its job.

We use a simple model for this at marketFX, which we call the marketFX Crawl-to-Citation Chain. A page has to pass six links before it can rank in Google or be cited in an AI answer, and it is only as visible as its weakest link.

The marketFX Crawl-to-Citation Chain. A page is only as visible as its weakest link.

Here is what each link covers in practice.

1. Access. Can Googlebot, Bingbot, GPTBot, ClaudeBot, and PerplexityBot fetch the page at all? This covers robots.txt rules, server errors, and status codes. It also covers bot rules in your CDN or firewall, which often block AI crawlers without anyone on the marketing team knowing. Google's own guidance on its AI features lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" as a basic. See our glossary entry on AI crawlers for the main user agents.

2. Render. Is the main content, and every important link, in the HTML the server sends? Or does it only appear after JavaScript runs in a browser? This is the link most likely to be broken on modern sites, and the one where Google and AI crawlers differ most. We cover it in detail below.

3. Index. When several URLs show the same content, which one counts? A canonical URL setup, clean redirects, sensible noindex rules, and an accurate XML sitemap tell engines which version to keep. Without them, signals split across duplicates and the wrong pages show up.

4. Structure. Can bots find every page that matters, in a few clicks, through normal links? Site architecture, internal linking, crawl depth, and orphan pages all sit here. On large sites this is also where crawl budget becomes a real constraint.

5. Meaning. Once a bot has the page, can it tell what the page is about and who it is from? Clean titles and headings, schema markup that matches the visible content, and clear entities (your organization, services, people, and locations) all help machines interpret a page correctly.

6. Experience. Does the page load quickly and stay stable on a phone? Core Web Vitals measure loading, responsiveness, and visual stability. Google lists a great page experience among its SEO fundamentals for both classic results and AI features.

Monitoring sits across all six. Technical SEO is not a one time project. Releases, CMS changes, new plugins, and firewall updates can break any link in the chain overnight. A good service includes server log review, Search Console checks, and a short regression check after each release.

Why Do AI Crawlers Change Technical SEO?

AI crawlers change technical SEO because most of them read only the HTML your server sends. Google can render JavaScript, but GPTBot, ClaudeBot, PerplexityBot, and several others do not, so content and links that appear only after JavaScript runs are invisible to them.

This is not a theory. Vercel and the consultancy MERJ analyzed crawler traffic across Vercel's network and published the results in The rise of the AI crawler in December 2024. They found that none of the major AI crawlers they measured rendered JavaScript, including OpenAI's crawlers, Anthropic's ClaudeBot, Meta's crawler, ByteDance's Bytespider, and PerplexityBot. The exceptions were Google's Gemini, which uses Googlebot's infrastructure, and AppleBot.

A more recent test confirmed the pattern for links. In August 2026, Search Engine Land published a 41 day experiment by Vinicius Stanula. He split one site's link hierarchy into two groups: one with links hard coded in HTML, one with links injected by JavaScript. GPTBot and ClaudeBot crawled every HTML-linked page and none of the JavaScript-linked ones. Amazonbot and Meta's crawler also reached none of the JavaScript-linked pages.

Phase 1 of a 41 day crawl test on one site. Source: Vinicius Stanula, Search Engine Land, August 19, 2026. Shares computed from the published page counts.

Two caveats matter. This was one site, on a new domain, in one niche, over one 41 day window, so the low Googlebot numbers say more about crawl budget on a fresh site than about Google in general. The author notes these limits himself. But the AI crawler result matches what Vercel saw across a much larger network.

The same experiment showed the fix works. When the JavaScript links were converted to plain HTML, GPTBot picked up 250 new pages within about two days. The page content did not change. Only the way the links were delivered changed.

Vercel's data also showed AI crawlers waste far more requests than Googlebot. ChatGPT's crawler spent 34.82% of its fetches on 404 pages and 14.36% on redirects, against 8.22% and 1.49% for Googlebot. Broken links, stale assets, and redirect chains cost you more with these bots.

What changes in a technical audit when AI crawlers matter. Sources: Vercel and MERJ (December 2024), Search Engine Land (August 2026), Google Search Central.

What about Google's own AI features? Google says the same SEO best practices apply to AI Overviews and AI Mode. In its guide to AI features and your website, Google says a page must be indexed and eligible to show with a snippet, and that there are no additional technical requirements. It also says you do not need new machine readable files, AI text files, or special schema to appear. So for Google, strong technical SEO is the AI search work. For other assistants, the rendering gap is the big addition.

That is also why we treat files like llms.txt as optional extras rather than a fix. They can help some tools read a site. They do not replace content and links in the HTML, and they do not guarantee citations.

What Does a Technical SEO Audit Service Include?

A technical SEO audit service should include a full crawl, a raw HTML versus rendered page comparison, server log analysis for search and AI crawlers, an indexation review, a structured data check, Core Web Vitals data, and a prioritized fix list with owners. A list of errors from a tool is not an audit.

Here is what we would expect a credible audit to deliver:

• A full site crawl with status codes, redirect chains, canonicals, noindex tags, duplicate titles, and orphan pages.
• A raw HTML check on key templates, comparing what the server sends with what a browser renders. Disable JavaScript and click through the paths to your revenue pages. If navigation or main content disappears, AI crawlers cannot see it.
• Server log analysis showing which bots actually visit, how often, and where their requests go. This is the only way to know whether GPTBot or ClaudeBot reach your service pages or spend their time on 404s.
• Bot access review covering robots.txt, CDN and firewall rules, and any rate limits that block crawlers you want.
• Indexation review in Google Search Console and Bing Webmaster Tools: which pages are indexed, which are excluded, and why.
• Structured data review checking that schema is valid and matches the visible page.
• Core Web Vitals data from real users, by template, on mobile and desktop.
• A prioritized fix list with effort, impact, and a named owner for each item, written so a developer can act on it.

The last point is where many audits fail. A 200 row spreadsheet sorted by tool severity does not tell a team what to do on Monday. Good audits rank issues by business impact. A rendering problem on a revenue template matters more than 400 missing alt attributes on old blog images.

If you want to see how AI visibility testing fits alongside this, our guide to running an AI search audit covers the prompt testing and citation side. A technical audit and an AI search audit work best as one engagement.

How Much Does Page Speed Matter for Technical SEO?

Page speed matters, but it is rarely the first constraint. Core Web Vitals are one of the signals Google uses for page experience, and most sites still fail at least one threshold on mobile. Fix access, rendering, and indexation first, then improve speed on the templates that drive revenue.

The HTTP Archive's 2025 Web Almanac shows steady progress. In 2025, 48% of mobile websites and 56% of desktop websites had good Core Web Vitals, up from 32% and 41% in 2021. That still means more than half of mobile sites miss at least one threshold.

Share of websites passing all Core Web Vitals thresholds, by device. Source: HTTP Archive Web Almanac 2025, Performance chapter.

Speed work also tends to help the rendering problem. Sites that move key content to server rendering usually get faster first loads at the same time. That is one reason we look at the two together rather than as separate projects.

How Does Technical SEO Change for Ecommerce, Enterprise, and Multi-Location Sites?

The six links stay the same, but the hard part moves. Ecommerce sites fight duplicate URLs from filters and changing inventory. Enterprise sites fight scale, legacy platforms, and slow release cycles. Multi-location brands fight hundreds of near identical location pages that need unique, crawlable content.

Ecommerce technical SEO. Faceted navigation can create far more URLs than a store has products. Crawl control, canonical rules for filter pages, out of stock handling, and product structured data that matches the page are the core of the work. Fixes usually happen once at the template level, so one change can affect thousands of pages. Our guide to ecommerce SEO services covers the wider scope.

Enterprise technical SEO. The issues are often well known. The challenge is getting them shipped across several platforms, business units, and development teams. Good enterprise technical SEO looks like clear tickets, documented standards, and regression checks in the release process. See our guide to choosing an enterprise SEO agency for how that work is usually organized.

Multi-location and franchise technical SEO. Location pages are often built from one template with a city name swapped in. The technical job is to make every location page crawlable from normal links, give each one accurate structured data, and keep details consistent with each Google Business Profile. Our multi-location and franchise marketing work covers this at the brand and location level.

Should You Hire a Technical SEO Agency, Consultant, or Specialist?

Hire a technical SEO consultant when you have strong developers and need diagnosis and direction. Hire an agency when you need diagnosis plus implementation support across several areas. Hire an in-house specialist when the site is large enough to need someone on it every week.

Many companies combine options: an in-house specialist or developer for day to day fixes and an outside partner for audits, log analysis, and AI crawler work that needs specialist tools.

Whatever you choose, technical SEO should not sit apart from the rest of search. The best technical fix in the world will not help a page nobody needs. Pages earn visibility when technical health, useful content, and authority work together.

How Should You Compare Technical SEO Proposals?

Compare technical SEO proposals on what they will check, how they will prioritize, and who will implement the fixes, not on the length of the deliverable list. Ask every provider the same questions so the answers are comparable.

Questions worth asking:

1. Will you analyze our server logs? Without logs, nobody knows what Googlebot and AI crawlers actually do on your site.
2. How will you check rendering? Ask how they compare raw HTML with the rendered page, and which templates they will test.
3. Which AI crawlers will you review, and how? Look for named user agents, robots.txt and firewall review, and a plan to test whether key pages are readable without JavaScript.
4. How will you prioritize? The answer should involve business impact by template, not tool severity scores.
5. Who implements? Some providers only advise. Others work in your CMS or alongside your developers. Know which you are buying.
6. What happens after launch? Ask about monitoring, release checks, and how often they will review logs and Search Console.
7. How will you report results? Expect indexation, crawl, and performance data, plus organic traffic and leads. If AI search is in scope, expect a baseline of AI answer citations too.

Red flags include guaranteed rankings, promises that one file or one schema type will get you into AI answers, and audits that end with a tool export. Google itself says indexing and serving are never guaranteed, even for pages that meet every requirement.

On price, technical SEO services vary widely with site size, number of templates, platform, whether implementation is included, and how much ongoing monitoring you need. Compare scopes line by line before you compare fees.

How marketFX Approaches Technical SEO

At marketFX, technical SEO is the first part of our SEO and AI visibility consulting. Every engagement starts with an audit of technical health, structured data, crawlability and indexation, including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended access, along with content, internal linking, and location signals. The output is a prioritized fix list with effort and impact scoring and a 90 day action plan with named owners.

We work alongside your developers and content team rather than handing over a report and leaving. If your technical foundations are already strong and you want them adapted for AI answers, our AI SEO agency page explains that work. If AI answer citations are the main goal, see our GEO agency practice.

If you want a view of how Google and AI crawlers see your site today, book a strategy call. We will look at access, rendering, and indexation on your most important templates and tell you which link in the chain to fix first.