Thread: Correct Robots.Txt Code, but Google Will Not Crawl Site

Correct Robots.Txt Code, but Google Will Not Crawl Site

I first installed Google XML Sitemaps but removed the plug in and then installed Yoast SEO. My site is new so I just allowed Worpress to be searched by robots in the privacy settings. Everthing seems fine except Google Webmaster Tools states that a warning as listed below. Could someone please take a look at my robot.txt which is listed below and let me know if I should be blocking wp-admin and wp-includes as when I search for my URL in Google it states that it cannot crawl site as robot.txt is blocking it. Thanks Todd


Sitemap URL setup as: /sitemap_index.xml

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

It is normal to have some URLS blocked. Note that the file "robots.txt" it blocks some areas of the site that need not be tracked. It seems that the file "sitemap_index.xml" not being loaded, you can try to solve this by saving the option Sitemap XML again.

Thank you for your reply... Google search now shows my Classipress Ads which is good, but to find my home page in a google search I must use the entire URL and then Google displays "A description for this result is not available because of this site's robots.txt". My robots.txt text file has not changed, but maybe I have to wait more than 3 days for Google to index my home page, list it in the results with a discription? Seems like maybe something is blocked that should nto be, but I do not get the Google Webmaster warning anymore stating that "Warning Sitemap contains urls which are blocked by robots.txt"

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

The robots.txt file is correct. You need to have patience, Google does not always index your site quickly.

Barukar,

Thanks so much... One thing I noticed is now my domain shows up if I directly search for it but the discription on Google is one of my ads rather that a description of my site. Any good tips of where to put a description of my site show it will show up in Google? I do have an about page as well as a Facebook andn Twitter page which both have a description as well. Thanks! Todd