User-agent: * # commented this one out on 2018-05-04, because admin folder has now its own robots meta x-http header that prevents it from being crawled # more information: https://yoast.com/wordpress-robots-txt-example/ # Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # new additional disallows as recommend by wp dev codex # Disallow: /wp-includes/ # commented this one out on 2018-05-04, because this folder could contain relevant css or js for google # more information: https://yoast.com/wordpress-robots-txt-example/ # feeds, tracksbacks and comments are not that relavant for SEO/Crawlers # https://wpcorner.de/seo/robots-txt-fuer-wordpress-erstellen/ Disallow: /trackback/ Disallow: /feed/ Disallow: /comments/ Disallow: */trackback/ Disallow: */feed/ Disallow: */comments/ # last but not least the sitemap (updated path) Sitemap: https://www.openproject.org/sitemap_index.xml