.Google's John Mueller responded to a question about why Google.com marks pages that are disallowed coming from crawling through robots.txt and why the it is actually secure to neglect the similar Explore Console reports regarding those crawls.Crawler Visitor Traffic To Concern Guideline URLs.The individual inquiring the concern chronicled that bots were creating hyperlinks to non-existent query guideline URLs (? q= xyz) to webpages along with noindex meta tags that are actually also obstructed in robots.txt. What cued the inquiry is actually that Google.com is creeping the links to those web pages, receiving blocked out through robots.txt (without seeing a noindex robots meta tag) then receiving turned up in Google.com Search Console as "Indexed, though obstructed by robots.txt.".The person asked the adhering to question:." But listed here is actually the major question: why would Google.com mark web pages when they can not even observe the web content? What is actually the advantage during that?".Google.com's John Mueller validated that if they can not creep the page they can not view the noindex meta tag. He likewise produces an intriguing acknowledgment of the site: search driver, suggesting to neglect the end results since the "normal" customers will not see those end results.He wrote:." Yes, you are actually proper: if our team can't creep the webpage, our company can't find the noindex. That stated, if we can't crawl the web pages, at that point there's not a lot for our company to mark. So while you may see several of those webpages with a targeted internet site:- concern, the normal user will not observe all of them, so I wouldn't bother it. Noindex is actually also alright (without robots.txt disallow), it simply indicates the URLs will certainly end up being actually crawled (and end up in the Search Console report for crawled/not recorded-- neither of these statuses trigger concerns to the rest of the web site). The important part is that you don't make them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the limitations being used the Site: search progressed hunt operator for analysis main reasons. One of those factors is actually given that it's not linked to the regular hunt index, it's a different thing completely.Google's John Mueller discussed the website search driver in 2021:." The short answer is actually that a site: inquiry is actually not implied to be comprehensive, nor used for diagnostics functions.A web site query is actually a certain type of hunt that limits the end results to a particular website. It's generally merely words internet site, a digestive tract, and after that the site's domain name.This query restricts the results to a specific website. It is actually not meant to be a detailed selection of all the webpages from that site.".2. Noindex tag without utilizing a robots.txt is great for these sort of scenarios where a robot is connecting to non-existent web pages that are receiving discovered by Googlebot.3. Links along with the noindex tag will produce a "crawled/not recorded" entry in Search Console and also those will not have a damaging impact on the remainder of the internet site.Review the inquiry and also respond to on LinkedIn:.Why will Google.com mark webpages when they can not also view the web content?Featured Picture by Shutterstock/Krakenimages. com.