Webflow sync, pageviews & more.
NEW

Has anyone experienced the issue of Googlebot being blocked by robots.txt in Webflow, even though access has been allowed in the file?

TL;DR
  • Check and update the robots.txt in Webflow's SEO settings, then re-publish the site to apply changes.
  • Use Google Search Console to verify what Googlebot sees and ensure you're testing the correct domain.
  • Avoid domain conflicts by properly setting up canonical URLs and redirects between custom and webflow.io domains.
  • Account for caching delays and re-submit for indexing in Google Search Console if necessary.
  • Confirm noindex settings aren't enabled in page-level SEO settings.
  • Ensure you're publishing changes to the correct live environment, not just staging.

Yes, some users have experienced Googlebot being blocked by the robots.txt in Webflow, even when it appears correctly configured. This issue is usually related to publishing mismatches or staged changes not being live.

1. Confirm Your Live robots.txt Content

  • Go to Project Settings → SEO tab to check the robots.txt content.
  • Make sure it includes User-agent: * and Allow: / (or your intended allow/disallow rules).
  • Re-publish your site after making any changes in the SEO tab. Webflow does not automatically republish when this file is edited.

2. Validate With Google’s Tools

  • Use Google Search Console’s robots.txt Tester to check what Googlebot is seeing live. This ensures that changes are actually live and correctly formatted.
  • Make sure to test the correct domain (e.g., staging vs. production).

3. Avoid Multiple Domain Conflicts

  • If you're using both the webflow.io subdomain and a custom domain, ensure you're only indexing one version. Googlebot might try to crawl the webflow.io domain if canonical tags or redirects aren’t properly set.
  • Webflow often includes a disallow directive for the webflow.io subdomain (Disallow: /), which is intentional.

4. Clear Cache and Crawl Delays

  • Googlebot cache might delay the detection of a newly updated robots.txt. Allow a short period before rechecking.
  • If urgent, re-submit in Google Search Console and trigger a URL Inspection → Request indexing.

5. Check for Page-Level Meta Robots Tags

  • A page can be blocked by a meta robots noindex tag or header even if it’s allowed in robots.txt.
  • In Webflow, verify in Page Settings that “Hide this page from search engines” is not enabled.

6. Verify with Webflow Hosting

  • Ensure you're checking the correct environment. Edits in Webflow’s robots.txt field apply only to the domains you publish to.
  • If you've only published to staging or didn’t publish changes to the custom domain, Googlebot will still see the old or incorrect file.

Summary

Even with a correct robots.txt in Webflow, Googlebot can be blocked due to publishing delays, caching, or multiple-domain setup. Always verify with Google’s tools and make sure you’ve published the latest SEO setting changes to the production site.

Webflow Resources
Webflow templates
Webflow inspiration
Rate this answer

Other Webflow Questions

Explore Logic tutorials
Where can I adjust the column and gutter widths of the guide overlay in Webflow?
How can I enable scroll in the navbar on my Webflow site so that all navigation links are visible on mobile?
How can I implement smooth scrolling using Locomotive.js in Webflow while ensuring position sticky functionality is working correctly?
How can I lock the size of an image in Webflow so that it remains the same within a circle or line, even when I replace the image?
How do I fix the "403 restricted error" with the "openresty" message when trying to visit a friends' site on a custom domain linked through Webflow?
Can I build a website on Webflow and then import it to a domain purchased on Namecheap?
How can I create social sharing buttons/links on dynamic templates in Webflow using custom code and the method described by ( )?
Is it possible to import a list of usernames and passwords to the Webflow CMS, create a login page with a Webflow form, and grant access to other pages only if the username and corresponding password match the imported list?
Is there a way to hide a download page or landing page behind a form in Webflow and prevent access to the page without form submission?
How can I resolve the "page with redirect" error and eliminate alternative versions of my homepage URL in Google Search Console for my Webflow website?
How can I resolve the Google reCaptcha error on my Webflow form? I have tried adding both the Site and Secret keys, enabling validation, and even using different Google accounts, but the issue persists. Has anyone else experienced this and found a solution? Thank you.
What am I missing in Webflow when the black background appears on the button after clicking it in the top nav menu, even though I am on the current page and trying to change the menu's current state?
Why do users receive a "Passwords cannot be collected" popup when attempting to log in or sign up on our published Webflow site?
How do I connect the forms on my Webflow website, based on the Saasplex X template, to Hubspot without a native integration?
How can I rearrange the order of CMS content in Webflow?
Can Webflow be used to call a CSS or JS file stored on a different server, instead of hosting them on GitHub?
How can I use the Finsweet script to manage two different sliders on my Webflow page, each feeding content from different collections?
How can I prevent a full-screen slider in Webflow from pausing on hover? Additionally, I would like to link two sliders to one control or navigation. Can anyone with JavaScript knowledge help me achieve this?
How can I prevent scroll when my nav is open in Webflow using code? The scroll is disabled correctly when I change the class name of the second 'menu', but it does not get activated when the menu is closed. It doesn't work at all if both class names are the same. Can anyone provide assistance with this issue?
How can I add date filtering to my current list of filters in Webflow using Finsweet CMS Filtering?
How can I change the language of publication dates in Webflow CMS collection from English to Portuguese?
How can I configure Webflow forms to exclude empty or unchecked checkboxes when sending emails? Additionally, is it possible to customize the email text for checked checkboxes, such as using "réservé" instead of "true"?
How can I dynamically redirect to a CMS created after form submission in Webflow?
How can I make the menu close when a link is clicked in Webflow?
How can I hide the "no items found" empty field in Webflow for a second multi-image gallery on certain pages?
How can I implement swipe functionality between tab panes on mobile using custom code in Webflow?
How can I disable right-clicking on images to prevent downloading on my Webflow portfolio website?
Is there a way to fix the issue of values from Textareas with multiple lines not getting parsed correctly in Webflow Logic when posting to an external API?
In Webflow, how can I hide specific elements in the footer (such as "Ramin Khaze The Difference" and "Schedule showing") when the URL contains a certain parameter, such as "?mls"? I am a beginner in coding and would appreciate any help. Thank you!
How can I prevent the "w-current" class in Webflow from overriding the height of my custom class on a link block element?