Webflow sync, pageviews & more.
NEW

Has anyone experienced the issue of Googlebot being blocked by robots.txt in Webflow, even though access has been allowed in the file?

TL;DR
  • Check and update the robots.txt in Webflow's SEO settings, then re-publish the site to apply changes.
  • Use Google Search Console to verify what Googlebot sees and ensure you're testing the correct domain.
  • Avoid domain conflicts by properly setting up canonical URLs and redirects between custom and webflow.io domains.
  • Account for caching delays and re-submit for indexing in Google Search Console if necessary.
  • Confirm noindex settings aren't enabled in page-level SEO settings.
  • Ensure you're publishing changes to the correct live environment, not just staging.

Yes, some users have experienced Googlebot being blocked by the robots.txt in Webflow, even when it appears correctly configured. This issue is usually related to publishing mismatches or staged changes not being live.

1. Confirm Your Live robots.txt Content

  • Go to Project Settings → SEO tab to check the robots.txt content.
  • Make sure it includes User-agent: * and Allow: / (or your intended allow/disallow rules).
  • Re-publish your site after making any changes in the SEO tab. Webflow does not automatically republish when this file is edited.

2. Validate With Google’s Tools

  • Use Google Search Console’s robots.txt Tester to check what Googlebot is seeing live. This ensures that changes are actually live and correctly formatted.
  • Make sure to test the correct domain (e.g., staging vs. production).

3. Avoid Multiple Domain Conflicts

  • If you're using both the webflow.io subdomain and a custom domain, ensure you're only indexing one version. Googlebot might try to crawl the webflow.io domain if canonical tags or redirects aren’t properly set.
  • Webflow often includes a disallow directive for the webflow.io subdomain (Disallow: /), which is intentional.

4. Clear Cache and Crawl Delays

  • Googlebot cache might delay the detection of a newly updated robots.txt. Allow a short period before rechecking.
  • If urgent, re-submit in Google Search Console and trigger a URL Inspection → Request indexing.

5. Check for Page-Level Meta Robots Tags

  • A page can be blocked by a meta robots noindex tag or header even if it’s allowed in robots.txt.
  • In Webflow, verify in Page Settings that “Hide this page from search engines” is not enabled.

6. Verify with Webflow Hosting

  • Ensure you're checking the correct environment. Edits in Webflow’s robots.txt field apply only to the domains you publish to.
  • If you've only published to staging or didn’t publish changes to the custom domain, Googlebot will still see the old or incorrect file.

Summary

Even with a correct robots.txt in Webflow, Googlebot can be blocked due to publishing delays, caching, or multiple-domain setup. Always verify with Google’s tools and make sure you’ve published the latest SEO setting changes to the production site.

Webflow Resources
Webflow templates
Webflow inspiration
Rate this answer

Other Webflow Questions

Explore Logic tutorials
How can I make a Lottie animation fill the entire vertical container and align it to the right in Webflow?
What are some open-source CMS options that offer dynamic functionalities like Webflow, are suitable for large enterprise sites/web apps, secure, scalable, customizable, and extendable, while also having a visual interface and potentially being whitelabel?
How can I design a website in Webflow using the actual preview resolution of my device, while ensuring responsiveness and avoiding any issues caused by different resolutions?
How can I use Webflow forms to handle Mailchimp signups with checkboxes for group opt-ins?
How can I create a sticky navigation in Webflow without custom code or multiple navbars?
How can I create a responsive grid in Webflow, where a 3x3 grid of images on desktop becomes a 2x2 grid on tablet and a 1x1 grid on mobile?
How can I bypass Webflow's limitation for nested collection lists in static single pages and use unlimited nested collections with more than 5 items using a lightweight jQuery with no performance issues?
How can Webflow be used to bind a rich text area for blog posts?
Does Webflow support the backdrop blur filter and are there any workarounds if it's not working on the live site?
How can I make an image fade in smoothly after a page finishes loading in Webflow?
What is the new CMS feature in Webflow that allows users to filter dynamic lists using a defined range instead of just a number of items, and what are some of its benefits and upcoming enhancements?
Is there a more efficient method in Webflow for adding keywords and other Meta tags to my header without duplication? Thank you for any assistance. Mike
Why are my imported items duplicating on the live site in Webflow, but not in the designer or Collections folder?
Is it possible to use a Webflow form to sign up users for specific Mailchimp groups or can custom code be used to achieve this integration?
How can I fix the issue with border radius not displaying properly on iOS when applied to a parent div instead of the image itself in Webflow?
Can Webflow be used to call a CSS or JS file stored on a different server, instead of hosting them on GitHub?
How can I use the Finsweet script to manage two different sliders on my Webflow page, each feeding content from different collections?
How can I prevent a full-screen slider in Webflow from pausing on hover? Additionally, I would like to link two sliders to one control or navigation. Can anyone with JavaScript knowledge help me achieve this?
How can I change the language of publication dates in Webflow CMS collection from English to Portuguese?
How can I dynamically redirect to a CMS created after form submission in Webflow?
How can I hide the "no items found" empty field in Webflow for a second multi-image gallery on certain pages?
How can I implement swipe functionality between tab panes on mobile using custom code in Webflow?
How can I disable right-clicking on images to prevent downloading on my Webflow portfolio website?
In Webflow, how can I hide specific elements in the footer (such as "Ramin Khaze The Difference" and "Schedule showing") when the URL contains a certain parameter, such as "?mls"? I am a beginner in coding and would appreciate any help. Thank you!
How can I prevent the "w-current" class in Webflow from overriding the height of my custom class on a link block element?
Can anyone help me add a digital clock and date to my Webflow site? I've been able to add custom code, but I need assistance with styling the text. Thank you in advance for your help!
Can I capture a screenshot of a Webflow page on click and convert it to a downloadable PDF?
How can I pull the stored email from the first form submitted on our website and input it into a field on a later form during the sign-up process in Webflow? I have tried using the provided script but it only displays "undefined" in the field. I don't have much knowledge of JavaScript and found this script online.
How do I mirror user input from a form field to a non-form text field on the same Webflow page in real-time using JavaScript?
When will Webflow update the "Select Field" component to allow for a placeholder with different colors before and after selection, without the need for a label above the field?