Google Search Console may report that a Blogger URL is blocked by robots.txt. Before changing anything in your blog's SEO settings, check whether the rule really matches that URL and whether the page should be available to search engines.
This tutorial focuses on diagnosing a single Blogger post. It explains where to find the live robots file, how to interpret a matching rule, and why a crawl restriction is not the same as a noindex instruction.
What robots.txt does—and doesn't do
The robots.txt file contains instructions to crawlers about which URL paths they may fetch. On a Blogger blog, it normally appears at the domain root as https://example.com/robots.txt, using your real blog domain.
According to Google's robots.txt guidance, a disallowed page can still be discovered through external links even though its content is not crawled. Robots.txt is therefore not a reliable way to keep private material out of search results. Use the appropriate noindex method or access controls for that goal.
Step 1: Write down the exact URL and status
Copy the full published post URL from Blogger's View link. In Google Search Console, inspect the same URL rather than a vaguely similar homepage or a mobile parameter version.
Read the reported status carefully. “Blocked by robots.txt,” “Excluded by noindex,” “Crawled—currently not indexed,” and “Discovered—currently not indexed” represent different situations. Do not apply a robots.txt fix to an article that is actually excluded for another reason.
Step 2: Open the live robots.txt file
Visit https://YOUR-BLOG-DOMAIN/robots.txt in a browser. If your blog uses a custom domain, use that domain rather than automatically assuming the old Blogspot address shows the same current configuration.
Read the file as text. Look for groups beginning with a user-agent directive and the Allow or Disallow rules that follow. A rule applying to one crawler might not apply to another.
Step 3: Compare the rule with the post path
Suppose the post path is /2026/10/example-guide.html. A disallow rule targeting that path or a parent directory may prevent crawling. A rule for a different path, such as a search-results page, should not automatically be blamed for the post's status.
Robots matching has specific rules for user agents, paths, wildcards, and precedence. Do not assume every string that happens to appear inside the path causes a block. Use Google's documentation when interpreting nontrivial patterns.
Worked example: a search-page rule is not a post block
Imagine the live file contains User-agent: * followed by Disallow: /search. A published post at /2026/10/example-guide.html does not start with /search, so that rule alone is not proof the article is blocked. By contrast, a matching rule targeting the post path or all paths could change crawler access.
This is only an illustration. The real answer depends on the complete live file, the correct user-agent group, and Google's path matching rules. If in doubt, make a copy of the current file and compare the precise URL instead of guessing from one line.
Step 4: Inspect Blogger's custom robots settings
Open Blogger → Settings → Crawlers and indexing and look for the option for a custom robots.txt file. The official Blogger settings guide describes this feature.
If you have enabled a custom file, copy its current text to a safe note before editing. Confirm that the rule actually blocks the post. Avoid pasting robots templates found on random sites; a broad Disallow: / can prevent crawling of the entire site for the affected crawler.
Step 5: Check other indexing signals
Even after fixing an unintended crawl block, a page may remain unindexed for other reasons. Inspect its robots meta tag, canonical URL, HTTP status, and whether it is publicly reachable. A robots.txt change is not an indexing guarantee. If those checks pass but the article itself needs refreshing, use our practical checklist for updating old blog posts to review its content separately.
If the page intentionally should not appear in search, consider whether the correct approach is a supported noindex setting rather than blocking Google from seeing the page. A crawler that cannot fetch the page generally cannot read its noindex tag.
Step 6: Validate the change carefully
- Make the smallest necessary change to the custom robots rules.
- Save the setting and reopen the public robots.txt URL to confirm the live version.
- Use Search Console URL Inspection on the affected post when appropriate.
- Monitor the reported status over time; old crawl data may remain visible for a while.
Do not resubmit the same URL repeatedly because the report has not changed immediately. Crawling and indexing are separate processes.
Common mistakes
- Editing robots.txt when the actual issue is a
noindextag. - Using a broad disallow rule copied from another Blogger template.
- Confusing a normal search-results exclusion with a published post URL.
- Assuming a corrected robots file guarantees ranking or indexing.
- Blocking Googlebot while hoping it will read a canonical or noindex instruction on the blocked page.
Frequently asked questions
Can robots.txt force Google to index my post?
No. It can permit crawling, but whether Google indexes a page depends on other signals and the page's usefulness.
Is “blocked by robots.txt” the same as “excluded by noindex”?
No. Robots.txt controls crawler access; noindex tells a crawler that can read the page not to index it.
Should I block all Blogger search pages?
Decide based on your site and current platform defaults. Do not apply blanket rules without checking which URLs they affect.
What if Blogger isn't using my custom robots file?
Compare the saved setting with the live root file, allow time for the update, and verify the correct blog and domain before making another change.
When the file looks correct but Search Console still says blocked
Confirm that you are inspecting the same host and protocol as the public post, and that Search Console's message reflects the current rule rather than an earlier crawl. Check the exact URL, especially if you recently changed a custom domain. A successful live fetch after a fix does not by itself guarantee indexing or an immediate report update.
Final check
Diagnose the exact URL, find the matching rule, edit only if it is unintended, and separately review indexing and canonical signals. That is safer than replacing the whole robots file every time a post stops appearing in Google.
