.Google.com's John Mueller responded to a question concerning why Google.com indexes web pages that are refused coming from crawling through robots.txt and why the it is actually risk-free to ignore the relevant Browse Console files regarding those crawls.Robot Traffic To Concern Guideline URLs.The person asking the question recorded that crawlers were actually creating web links to non-existent inquiry guideline Links (? q= xyz) to web pages along with noindex meta tags that are likewise blocked in robots.txt. What cued the question is actually that Google is creeping the links to those pages, acquiring shut out through robots.txt (without watching a noindex robots meta tag) at that point receiving reported in Google Look Console as "Indexed, though obstructed through robots.txt.".The individual inquired the adhering to concern:." Yet here is actually the significant concern: why would certainly Google.com mark webpages when they can't even find the material? What's the advantage because?".Google.com's John Mueller affirmed that if they can not creep the webpage they can't observe the noindex meta tag. He also produces an interesting mention of the internet site: hunt operator, urging to ignore the results considering that the "ordinary" users will not see those results.He created:." Yes, you're proper: if we can't crawl the webpage, our experts can not see the noindex. That claimed, if our company can not crawl the web pages, after that there's not a lot for our company to index. So while you could find a few of those webpages along with a targeted website:- inquiry, the ordinary consumer will not view them, so I would not fuss over it. Noindex is actually additionally fine (without robots.txt disallow), it merely indicates the URLs will end up being actually crawled (as well as wind up in the Browse Console file for crawled/not indexed-- neither of these standings trigger concerns to the remainder of the internet site). The vital part is actually that you do not produce them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the limitations in using the Internet site: search progressed hunt operator for analysis main reasons. Some of those factors is due to the fact that it is actually certainly not attached to the frequent hunt index, it is actually a different trait completely.Google's John Mueller talked about the site hunt driver in 2021:." The brief answer is actually that a web site: query is certainly not indicated to become full, nor utilized for diagnostics purposes.An internet site query is actually a certain type of search that restricts the outcomes to a particular web site. It is actually primarily just the word web site, a colon, and then the internet site's domain name.This question limits the end results to a particular site. It's certainly not meant to become a thorough assortment of all the webpages coming from that site.".2. Noindex tag without utilizing a robots.txt is actually great for these sort of situations where a bot is actually connecting to non-existent webpages that are actually getting found out through Googlebot.3. Links with the noindex tag will definitely produce a "crawled/not recorded" entry in Explore Console which those will not have an adverse effect on the rest of the web site.Review the inquiry as well as address on LinkedIn:.Why would certainly Google.com index web pages when they can't also find the information?Included Graphic by Shutterstock/Krakenimages. com.