Tag Archive for: blocked URL

Yes, Google will still show pages even if you have set them up to be blocked in your robots.txt file.  Why exactly, you’re wondering, I’m sure.

Google is actually not crawling any of the URLs you have listed as blocked in your robots.txt file.  What they are doing is displaying the page, just not showing your description meta tag or any information from inside that page.  Matt Cutts gets into more detail about that here:

So to prevent Google from displaying your pages at all, you need to remove the page inside your Webmaster Tools or else use a robots meta tag with a noindex value in it for each page.  Then your page will not show up in Google’s results at all.

You can get more details on this information from Quick Online Tips.