{"id":38137,"date":"2025-09-12T20:18:29","date_gmt":"2025-09-12T20:18:29","guid":{"rendered":"https:\/\/www.wpstaq.com\/?post_type=knowledgebase&#038;p=38137"},"modified":"2025-10-01T05:37:05","modified_gmt":"2025-10-01T05:37:05","slug":"robots-txt-file-managing-the-default-file","status":"publish","type":"knowledgebase","link":"https:\/\/www.wpstaq.com\/hk\/knowledgebase\/robots-txt-file-managing-the-default-file\/","title":{"rendered":"Robots.txt file &#8211; managing the default file"},"content":{"rendered":"<h2>What We Do by Default<\/h2>\n<p>Every site hosted on Staq comes with a pre-configured <code>robots.txt<\/code> file designed to help protect sensitive areas of your WordPress site while still allowing essential assets (like CSS and JS) to be crawled.<\/p>\n<p>Here\u2019s the default content:<\/p>\n<pre><code># Default WPStaq robots.txt for WordPress hosting\r\n# WARNING: This file is automatically managed by WPStaq and will be overwritten on updates.\r\n# To customize: Remove the \"Default WPStaq robots.txt\" comment above to make this file user-managed.\r\n\r\nUser-agent: *\r\nDisallow: \/calendar\/action*\r\nDisallow: \/events\/action*\r\nDisallow: \/cdn-cgi*\r\nAllow: \/*.css\r\nAllow: \/*.js\r\nDisallow: \/*?\r\nCrawl-delay: 10\r\n<\/code><\/pre>\n<h3>Why This Matters<\/h3>\n<ul>\n<li><strong>Blocks dynamic calendar and event actions<\/strong> from being crawled.<\/li>\n<li><strong>Allows static assets<\/strong> like CSS and JS files to be crawled by search engines.<\/li>\n<li><strong>Disallows parameter-based URLs<\/strong> (e.g., <code>?ref=<\/code>, <code>?utm=<\/code>) to avoid crawl budget waste.<\/li>\n<li><strong>Sets a crawl delay<\/strong> of 10 seconds to reduce load on your site from aggressive bots.<\/li>\n<\/ul>\n<h2>How to Override the <code>robots.txt<\/code> File<\/h2>\n<p>If you want to customize the <code>robots.txt<\/code> rules to suit your SEO strategy or specific site setup, you can easily take control of the file.<\/p>\n<h3>Step 1: Connect via sFTP<\/h3>\n<p>To access the file directly, connect to your website\u2019s server using sFTP.<\/p>\n<p>\ud83d\udc49 Follow our sFTP guide here:<br \/>\n<a href=\"https:\/\/www.wpstaq.com\/knowledgebase\/ftp-access\/\" target=\"_blank\" rel=\"noopener\">How to Access Your Site via sFTP<\/a><\/p>\n<h3>Step 2: Locate the File<\/h3>\n<p>Once connected:<\/p>\n<ul>\n<li>Navigate to the root directory of your WordPress site (inside the <code>wordpress<\/code>\u00a0folder<\/li>\n<li>Look for the file named <code>robots.txt<\/code>.<\/li>\n<\/ul>\n<h3>Step 3: Take Ownership<\/h3>\n<p>By default, the file is managed by WPStaq and will be overwritten during system updates.<br \/>\nTo make it user-managed:<\/p>\n<ul>\n<li><strong>Edit the file<\/strong> using a text\/code editor (via sFTP).<\/li>\n<li><strong>Delete the comment:<\/strong><br \/>\n<code># Default WPStaq robots.txt for WordPress hosting<\/code><br \/>\nat the top of the file.<\/li>\n<\/ul>\n<p>Once that line is removed, Staq will stop overriding the file during updates.<\/p>\n<h2>Customizing Your Rules<\/h2>\n<p>After taking control of the file, you can modify it.<\/p>\n<h2>About the Crawl-delay Rule<\/h2>\n<p>By default, Staq includes a <code>Crawl-delay: 10<\/code> directive in your <code>robots.txt<\/code> file. This rule helps reduce server load from aggressive bots that don\u2019t pace their requests.<\/p>\n<h3>Why We Add Crawl-delay<\/h3>\n<ul>\n<li><strong>Protects server resources<\/strong> \u2013 prevents bots from overwhelming your PHP workers with too many simultaneous requests.<\/li>\n<li><strong>Controls bandwidth usage<\/strong> \u2013 reduces spikes that could eat into your monthly bandwidth.<\/li>\n<li><strong>Safe for SEO<\/strong> \u2013 major search engines like Google ignore <code>Crawl-delay<\/code>, so it won\u2019t slow down your indexing.<\/li>\n<\/ul>\n<h2>FAQ<\/h2>\n<ul>\n<li><strong>Does this affect Google?<\/strong> \u2013 No. Google and most major search engines ignore the <code>Crawl-delay<\/code> directive.<\/li>\n<li><b>What if I want to manage my own robots.txt file?\u00a0<\/b>&#8211; You can easily do that by logging into the sFTP details, as suggested above, and then delete the comment <code># Default WPStaq robots.txt for WordPress hosting<\/code> at the top of the file. Once that line is removed, Staq will stop overriding the file during updates.<\/li>\n<li><strong>Can I change the Crawl-delay?<\/strong> Sure. Log into the sFTP and remove\u00a0the comment <code># Default WPStaq robots.txt for WordPress hosting<\/code> at the top of the file and then you can configure your Crawl-delay value.<\/li>\n<\/ul>\n","protected":false},"featured_media":0,"parent":0,"template":"","class_list":["post-38137","knowledgebase","type-knowledgebase","status-publish","hentry","categories-seo-tools","categories-wordpress-hosting"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Robots.txt file - managing the default file - Managed Hosting Platform for WordPress<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wpstaq.com\/knowledgebase\/robots-txt-file-managing-the-default-file\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Robots.txt file - managing the default file - Managed Hosting Platform for WordPress\" \/>\n<meta property=\"og:description\" content=\"What We Do by Default Every site hosted on Staq comes with a pre-configured robots.txt file designed to help protect sensitive areas of your WordPress site while still allowing essential assets (like CSS and JS) to be crawled. Here\u2019s the default content: # Default WPStaq robots.txt for WordPress hosting # WARNING: This file is automatically&hellip;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wpstaq.com\/knowledgebase\/robots-txt-file-managing-the-default-file\/\" \/>\n<meta property=\"og:site_name\" content=\"Managed Hosting Platform for WordPress\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/staqhosting\/\" \/>\n<meta property=\"article:modified_time\" content=\"2025-10-01T05:37:05+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.wpstaq.com\\\/knowledgebase\\\/robots-txt-file-managing-the-default-file\\\/\",\"url\":\"https:\\\/\\\/www.wpstaq.com\\\/knowledgebase\\\/robots-txt-file-managing-the-default-file\\\/\",\"name\":\"Robots.txt file - managing the default file - Managed Hosting Platform for WordPress\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.wpstaq.com\\\/#website\"},\"datePublished\":\"2025-09-12T20:18:29+00:00\",\"dateModified\":\"2025-10-01T05:37:05+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.wpstaq.com\\\/knowledgebase\\\/robots-txt-file-managing-the-default-file\\\/#breadcrumb\"},\"inLanguage\":\"en-hk\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.wpstaq.com\\\/knowledgebase\\\/robots-txt-file-managing-the-default-file\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.wpstaq.com\\\/knowledgebase\\\/robots-txt-file-managing-the-default-file\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.wpstaq.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Robots.txt file &#8211; managing the default file\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.wpstaq.com\\\/#website\",\"url\":\"https:\\\/\\\/www.wpstaq.com\\\/\",\"name\":\"Managed Hosting Platform for WordPress\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.wpstaq.com\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.wpstaq.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-hk\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.wpstaq.com\\\/#organization\",\"name\":\"Staq\",\"alternateName\":\"WPStaq\",\"url\":\"https:\\\/\\\/www.wpstaq.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-hk\",\"@id\":\"https:\\\/\\\/www.wpstaq.com\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.wpstaq.com\\\/wp-content\\\/uploads\\\/2023\\\/05\\\/cropped-LogoNoText.png\",\"contentUrl\":\"https:\\\/\\\/www.wpstaq.com\\\/wp-content\\\/uploads\\\/2023\\\/05\\\/cropped-LogoNoText.png\",\"width\":512,\"height\":512,\"caption\":\"Staq\"},\"image\":{\"@id\":\"https:\\\/\\\/www.wpstaq.com\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/staqhosting\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/wpstaq\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Robots.txt file - managing the default file - Managed Hosting Platform for WordPress","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wpstaq.com\/knowledgebase\/robots-txt-file-managing-the-default-file\/","og_locale":"en_US","og_type":"article","og_title":"Robots.txt file - managing the default file - Managed Hosting Platform for WordPress","og_description":"What We Do by Default Every site hosted on Staq comes with a pre-configured robots.txt file designed to help protect sensitive areas of your WordPress site while still allowing essential assets (like CSS and JS) to be crawled. Here\u2019s the default content: # Default WPStaq robots.txt for WordPress hosting # WARNING: This file is automatically&hellip;","og_url":"https:\/\/www.wpstaq.com\/knowledgebase\/robots-txt-file-managing-the-default-file\/","og_site_name":"Managed Hosting Platform for WordPress","article_publisher":"https:\/\/www.facebook.com\/staqhosting\/","article_modified_time":"2025-10-01T05:37:05+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wpstaq.com\/knowledgebase\/robots-txt-file-managing-the-default-file\/","url":"https:\/\/www.wpstaq.com\/knowledgebase\/robots-txt-file-managing-the-default-file\/","name":"Robots.txt file - managing the default file - Managed Hosting Platform for WordPress","isPartOf":{"@id":"https:\/\/www.wpstaq.com\/#website"},"datePublished":"2025-09-12T20:18:29+00:00","dateModified":"2025-10-01T05:37:05+00:00","breadcrumb":{"@id":"https:\/\/www.wpstaq.com\/knowledgebase\/robots-txt-file-managing-the-default-file\/#breadcrumb"},"inLanguage":"en-hk","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wpstaq.com\/knowledgebase\/robots-txt-file-managing-the-default-file\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wpstaq.com\/knowledgebase\/robots-txt-file-managing-the-default-file\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.wpstaq.com\/"},{"@type":"ListItem","position":2,"name":"Robots.txt file &#8211; managing the default file"}]},{"@type":"WebSite","@id":"https:\/\/www.wpstaq.com\/#website","url":"https:\/\/www.wpstaq.com\/","name":"Managed Hosting Platform for WordPress","description":"","publisher":{"@id":"https:\/\/www.wpstaq.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wpstaq.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-hk"},{"@type":"Organization","@id":"https:\/\/www.wpstaq.com\/#organization","name":"Staq","alternateName":"WPStaq","url":"https:\/\/www.wpstaq.com\/","logo":{"@type":"ImageObject","inLanguage":"en-hk","@id":"https:\/\/www.wpstaq.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.wpstaq.com\/wp-content\/uploads\/2023\/05\/cropped-LogoNoText.png","contentUrl":"https:\/\/www.wpstaq.com\/wp-content\/uploads\/2023\/05\/cropped-LogoNoText.png","width":512,"height":512,"caption":"Staq"},"image":{"@id":"https:\/\/www.wpstaq.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/staqhosting\/","https:\/\/www.linkedin.com\/company\/wpstaq"]}]}},"_links":{"self":[{"href":"https:\/\/www.wpstaq.com\/hk\/wp-json\/wp\/v2\/knowledgebase\/38137","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wpstaq.com\/hk\/wp-json\/wp\/v2\/knowledgebase"}],"about":[{"href":"https:\/\/www.wpstaq.com\/hk\/wp-json\/wp\/v2\/types\/knowledgebase"}],"wp:attachment":[{"href":"https:\/\/www.wpstaq.com\/hk\/wp-json\/wp\/v2\/media?parent=38137"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}