Sitemap contains urls which are blocked by robots.txt under Google Webmaster Tools

Tag: seo,google-webmaster-tools,google-sitemap

Really frustrated. It is because of technical errors or Google's cache effect? Do I have to wait for a few days more?

I have been updating my robots file and sitemap to Google Webmaster Tools several times since last night. But each time I resubmit the sitemap, it keeps saying:

Sitemap contains urls which are blocked by robots.txt.

It seems that all URLs in the sitemap are blocked. Examples:

http://janwawa.com/en
http://janwawa.com/en/
http://janwawa.com/en/album

This is my robots file

User-agent: *
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /qsystem/
Disallow: /piwik/
Disallow: /webpublisher/
Disallow: /en/quotation/
Disallow: /th/quotation/

This is my sitemap

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">

    <url>
        <loc>http://janwawa.com/en</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hc005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hk005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/jp005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/kr005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ib005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ib006.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ib010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ij005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/im005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/eu005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/eu010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m010f.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m015f.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m020.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m025.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m030.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m035.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m040.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m045.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m050.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m055.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m060.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m065.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m070.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m075.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms020.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms025.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms030.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms035.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms040.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms045.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms050.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/s005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/s010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hd005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hd010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hd015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/t005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/vn005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/calendar/month.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/heritage.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/visits.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/meals.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/shopping.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/album</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/contact.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/calendar/month.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/start_here.php</loc>
    </url>
        <url>
        <loc>http://janwawa.com/th</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hc005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hk005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/jp005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/kr005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ib005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ib006.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ib010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ij005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/im005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/eu005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/eu010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m010f.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m015f.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m020.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m025.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m030.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m035.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m040.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m045.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m050.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m055.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m060.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m065.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m070.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m075.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms020.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms025.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms030.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms035.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms040.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms045.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms050.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/s005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/s010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hd005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hd010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hd015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/t005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/vn005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/calendar/month.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/heritage.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/visits.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/meals.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/shopping.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/album</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/contact.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/calendar/month.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/start_here.php</loc>
    </url>

<url>
    <loc>http://janwawa.com/en/</loc>
    <xhtml:link 
                 rel="alternate"
                 hreflang="th"
                 href="http://janwawa.com/th/"
                 />
    <xhtml:link 
                 rel="alternate"
                 hreflang="en"
                 href="http://janwawa.com/en/"
                 />
</url>

<url>
    <loc>http://janwawa.com/th/</loc>
    <xhtml:link 
                 rel="alternate"
                 hreflang="en"
                 href="http://janwawa.com/en/"
                 />
     <xhtml:link 
                 rel="alternate"
                 hreflang="th"
                 href="http://janwawa.com/th/"
                 />
</url>

</urlset>

Best How To :

Okay , i have been to your site once , looked your sitemap as well robots.txt , you are seems doing right but the problem is google webmaster tool . Here is some tips you should try and let me know if you are able to figure out or not , go to your Webmaster dashborad -> select the site -> Fetch as Google -> then put any URL in belongs to your site directory which wanted to let Google crawl , then submit for index , if Google has not found any error then your problem would be solved by doing below method Again go to the fetch as Google and then enter h ttp://janwawa.com/en and submit for index with the option of CRAWL DIRECT URL AND LINK .

okay let me know if you found any problems .

fullPage.js: Make all slides and sections visible in search engine results

jquery,seo,web-crawler,single-page-application,fullpage.js

You probably won't be able to force Google to index your anchor links as different pages. You will be able to index them as a single page. Google will read your page as what it really is. A single page. There are some recommendations which suggest to use the id...

Robots.txt file in MVC.NET 4

asp.net,asp.net-mvc-4,seo,robots.txt

do robots crawl the controllers which has [Authorization] attribute like Administration If they find a link to it, they are likely to try and crawl it, but they will fail just like anyone with a web browser that does not log in. Robots have no special ability to access...

Have google finding .com instead of .ru

google-webmaster-tools

According to Google's content guidelines, you can use a canonical URL to setting a preferred domain, and also do this with Google webmaster tools. Put this link in your site header: <link rel="canonical" href="https://example.com"> ...

Convert MVC Website to be Responsive

mobile,model-view-controller,seo

There are typically three methods for handling Responsive Design : Using A Responsive Design Framework Using CSS Media Queries Using Percentages The ease of using each of these will ultimately be based on how complex your existing Project is and how much you are willing to do and change to...

Change the unique generated title names of friendly-id using attribute of another table

ruby-on-rails,ruby,seo,friendly-url,friendly-id

class Company < ActiveRecord::Base ............................. def city self.address.city end end ...