Results 1 to 5 of 5

Thread: Correct Robots.Txt Code, but Google Will Not Crawl Site

  1. #1
    Thread Starter
    tlewis's Avatar
    Join Date
    Jan 2012
    Location
    United States
    Posts
    17
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Question Correct Robots.Txt Code, but Google Will Not Crawl Site

    I first installed Google XML Sitemaps but removed the plug in and then installed Yoast SEO. My site is new so I just allowed Worpress to be searched by robots in the privacy settings. Everthing seems fine except Google Webmaster Tools states that a warning as listed below. Could someone please take a look at my robot.txt which is listed below and let me know if I should be blocking wp-admin and wp-includes as when I search for my URL in Google it states that it cannot crawl site as robot.txt is blocking it. Thanks Todd
    "Warnings Url blocked by robots.txt. Sitemap contains urls which are blocked by robots.txt. 16"


    Sitemap URL setup as: /sitemap_index.xml

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

  2. #2
    barukar's Avatar
    Join Date
    Sep 2010
    Location
    Brasil, São Paulo, SP
    Posts
    6,779
    Thanks
    185
    Thanked 740 Times in 622 Posts
    It is normal to have some URLS blocked. Note that the file "robots.txt" it blocks some areas of the site that need not be tracked. It seems that the file "sitemap_index.xml" not being loaded, you can try to solve this by saving the option Sitemap XML again.
    Please help our moderating team work more efficiently by not sending us support questions via PM. You can read more about how AppThemes support works here. Thank you.

    -------------------------------------------------------------------------------------------
    Projects: ClassiNoiva - Classimóveis - vocênoenem - i50 - Clube DETRAN

  3. #3
    Thread Starter
    tlewis's Avatar
    Join Date
    Jan 2012
    Location
    United States
    Posts
    17
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thank you for your reply... Google search now shows my Classipress Ads which is good, but to find my home page in a google search I must use the entire URL and then Google displays "A description for this result is not available because of this site's robots.txt". My robots.txt text file has not changed, but maybe I have to wait more than 3 days for Google to index my home page, list it in the results with a discription? Seems like maybe something is blocked that should nto be, but I do not get the Google Webmaster warning anymore stating that "Warning Sitemap contains urls which are blocked by robots.txt"

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

  4. #4
    barukar's Avatar
    Join Date
    Sep 2010
    Location
    Brasil, São Paulo, SP
    Posts
    6,779
    Thanks
    185
    Thanked 740 Times in 622 Posts
    The robots.txt file is correct. You need to have patience, Google does not always index your site quickly.
    Please help our moderating team work more efficiently by not sending us support questions via PM. You can read more about how AppThemes support works here. Thank you.

    -------------------------------------------------------------------------------------------
    Projects: ClassiNoiva - Classimóveis - vocênoenem - i50 - Clube DETRAN

  5. #5
    Thread Starter
    tlewis's Avatar
    Join Date
    Jan 2012
    Location
    United States
    Posts
    17
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Barukar,

    Thanks so much... One thing I noticed is now my domain shows up if I directly search for it but the discription on Google is one of my ads rather that a description of my site. Any good tips of where to put a description of my site show it will show up in Google? I do have an about page as well as a Facebook andn Twitter page which both have a description as well. Thanks! Todd

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Google won't crawl my homepage!!
    By nicknolting in forum Help Using JobRoller
    Replies: 35
    Last Post: January 28th, 2013, 10:17 PM
  2. Vantage + Seo By Yoast - How to let Google Crawl search results
    By decao in forum Vantage General Discussion (Legacy)
    Replies: 0
    Last Post: August 21st, 2012, 11:04 AM
  3. Google Crawl & Weird Links
    By lehuya in forum Report ClassiPress Bugs
    Replies: 1
    Last Post: June 13th, 2011, 04:17 AM