Runnable, tested artifacts for every production-worthy code block in SEO for Engineers by Nima Jafari. Sitemaps, robots.txt CI gates, Googlebot verification, log analysis, and more, in Python & TypeScript.
Turn raw server/CDN access logs into verified-crawler analytics: parse many log formats, verify Googlebot/Bingbot by IP (not user-agent), enrich with DuckDB, and analyze crawl budget, crawl waste, and crawl traps with SQL.
SEO crawl budget manipulation methodology as a Claude Skill — fast indexing for low-competition long-tail keywords
A framework-agnostic reference for handling faceted navigation without wrecking crawl budget: a five-tier classifier (index / canonicalize / noindex / robots-block) driven by one facet registry, with sitemap, robots.txt, and parameter normalization.
Free, private, 100% client-side log file analyzer: GPTBot vs OAI-SearchBot cost split, crawl budget, Cloudflare/Fastly/AWS edge rules + robots.txt generator. Training vs search-index vs user-triggered AI split (2026).
A fast Python utility to audit XML sitemaps, validate HTTP status codes, and detect non-indexable, redirected, or 404 URLs wasting crawl budget. Designed to streamline search engine indexation. Developed by Rajesh Nitharwal, Founder at Linkoster.com — offering technical SEO audits, high-DA guest posting, and Digital PR outreach.
Find redirect chains & loops, then emit the exact .htaccess/nginx rules that collapse them. Zero deps.
Crawl-budget analysis from server logs — per-bot (incl. GPTBot/ClaudeBot/PerplexityBot) status mix, waste %, top paths. Zero deps.
Which crawlers and AI bots hit your site, per day, verified vs spoofed — from Cloudflare's Free plan. Zero dependencies.
Lightweight WordPress plugin that keeps your last-modified date honest, so sitemap lastmod and schema dateModified only change when the content really changes.
Streaming Apache/Nginx log analyzer for SEO and crawl-budget analysis on multi-GB logs. 70+ bot families incl. modern AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider), reverse-DNS verification, 24 reports, React dashboard. Tested at 16 GB / 50.9M lines.
To associate your repository with the crawl-budget topic, visit your repo's landing page and select "manage topics."