Sitemap contains urls which are blocked by robots.txt under Google Webmaster Tools

Tag: seo,google-webmaster-tools,google-sitemap

Really frustrated. It is because of technical errors or Google's cache effect? Do I have to wait for a few days more?

I have been updating my robots file and sitemap to Google Webmaster Tools several times since last night. But each time I resubmit the sitemap, it keeps saying:

Sitemap contains urls which are blocked by robots.txt.

It seems that all URLs in the sitemap are blocked. Examples:

http://janwawa.com/en
http://janwawa.com/en/
http://janwawa.com/en/album

This is my robots file

User-agent: *
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /qsystem/
Disallow: /piwik/
Disallow: /webpublisher/
Disallow: /en/quotation/
Disallow: /th/quotation/

This is my sitemap

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">

    <url>
        <loc>http://janwawa.com/en</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hc005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hk005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/jp005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/kr005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ib005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ib006.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ib010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ij005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/im005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/eu005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/eu010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m010f.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m015f.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m020.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m025.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m030.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m035.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m040.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m045.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m050.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m055.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m060.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m065.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m070.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/m075.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms020.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms025.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms030.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms035.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms040.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms045.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/ms050.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/s005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/s010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hd005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hd010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/hd015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/t005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/tour/vn005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/calendar/month.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/heritage.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/visits.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/meals.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/shopping.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/album</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/contact.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/calendar/month.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/en/start_here.php</loc>
    </url>
        <url>
        <loc>http://janwawa.com/th</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hc005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hk005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/jp005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/kr005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ib005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ib006.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ib010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ij005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/im005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/eu005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/eu010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m010f.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m015f.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m020.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m025.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m030.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m035.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m040.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m045.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m050.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m055.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m060.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m065.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m070.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/m075.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms020.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms025.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms030.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms035.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms040.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms045.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/ms050.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/s005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/s010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hd005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hd010.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/hd015.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/t005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/tour/vn005.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/calendar/month.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/heritage.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/visits.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/meals.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/shopping.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/album</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/contact.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/calendar/month.php</loc>
    </url>
    <url>
        <loc>http://janwawa.com/th/start_here.php</loc>
    </url>

<url>
    <loc>http://janwawa.com/en/</loc>
    <xhtml:link 
                 rel="alternate"
                 hreflang="th"
                 href="http://janwawa.com/th/"
                 />
    <xhtml:link 
                 rel="alternate"
                 hreflang="en"
                 href="http://janwawa.com/en/"
                 />
</url>

<url>
    <loc>http://janwawa.com/th/</loc>
    <xhtml:link 
                 rel="alternate"
                 hreflang="en"
                 href="http://janwawa.com/en/"
                 />
     <xhtml:link 
                 rel="alternate"
                 hreflang="th"
                 href="http://janwawa.com/th/"
                 />
</url>

</urlset>

Best How To :

Okay , i have been to your site once , looked your sitemap as well robots.txt , you are seems doing right but the problem is google webmaster tool . Here is some tips you should try and let me know if you are able to figure out or not , go to your Webmaster dashborad -> select the site -> Fetch as Google -> then put any URL in belongs to your site directory which wanted to let Google crawl , then submit for index , if Google has not found any error then your problem would be solved by doing below method Again go to the fetch as Google and then enter h ttp://janwawa.com/en and submit for index with the option of CRAWL DIRECT URL AND LINK .

okay let me know if you found any problems .

How to “remove” indexed pages in Google and AngularJS

angularjs,nginx,seo,google-index

The moment Google index my new Urls, I guess It will have many urls with same duplicated code, right? I´m a bit worried because I have read Google penalizes this scenarios. No, Google does not penalize duplicate content. It is a hard to die myth perpetrated over and over...

Google Webmaster API - Mark crawlerros as fixed

api,http-status-code-403,google-webmaster-tools,google-crawlers

I've found the solution. Just set the Scope to WebmastersService.Scope.Webmasters instead of WebmastersService.Scope.WebmastersReadonly string[] scopes = { WebmastersService.Scope.Webmasters }; UserCredential credential = GoogleWebAuthorizationBroker.AuthorizeAsync(new ClientSecrets { ClientId = "clientId", ClientSecret = "clientSecret" } , scopes , "My Name" , CancellationToken.None).Result; ...

How Google “distinguishes” website articles from news? [closed]

html,seo,google-search

Google will use a number of approaches when building their databases and one cannot say exactly how you would get your site to register within google as a news sight. However you will notice the following meta information within many articles that show up on google news. <meta property="og:type" content="article"...

proper use link hreflang

html,seo,multiple-languages

should I add < link href="http://domain.com/" rel="canonical" > in url http://domain.com/?hl=en or http://domain.com/?hl=es? Neither, you can use: <link href="http://domain.com/" rel="canonical"> in http://domain.com/ <link href="http://domain.com/?hl=en" rel="canonical"> in http://domain.com/?hl=en <link href="http://domain.com/?hl=es" rel="canonical"> in http://domain.com/?hl=es should I add < html lang='en' > in url http://domain.com/?hl=en or http://domain.com/?hl=es? You can put it in...

Slidershow jquery and convert to css

jquery,css,html5,seo,slider

Please use class name instead of id to define css. .slider1_container { ... } To pass html validation please replace u attribute with data-u. reference: http://www.jssor.com/development/pass-html-validation.html...

Wordpress - Robotx.txt allows admin login?

wordpress,seo,robots.txt

You should create your own robots.txt file and upload it to website root directory. Follow the following steps to create and upload into root folder: Open notepad Add the following text into this file, remember add also your website sitemap path sitemap: http://www.yoursite.com/sitemap.xml User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow:...

Canonical url for google to prevent duplicate meta?

seo,meta-tags,google-webmaster-tools

You should go to the URL parameters setting in WMT and set the variable rs to be ignored, by indicating that it does not change page content seen by users. Note that rel="canonical" is just a suggestion and not a directive. Google can choose to ignore it if it feels...

How to fix Google search SEO after transfer old website and hosting to new framework website and hosting? [closed]

search,seo,transfer

Maybe Google has not reprocessed your website completely and is confused. You can create a sitemap.xml with all the URLs of your website and a recent lastmod date. Then submit it into Google Webmaster Tools and let crawlers revisit your site. May sure you don't block those links with your...

Disallow specific folders in robots.txt with wildcards

seo,search-engine,robots.txt,google-crawlers

You don't need wildcards at all for this. Your example will work, but it would work just as well without the wildcard. Trailing wildcards do not do anything useful. For example, this: Disallow: /x means: "Block any path that starts with '/x', followed by zero or more characters." And this:...

Removing the number of first page in Yii2 Pagination from the URL

.htaccess,pagination,seo,yii2

According to docs you should set yii\data\Pagination::forcePageParam to false by passing it in Pagination constructor $pages = new Pagination([ 'totalCount' => $books['booksCount'], 'pageParam' => 'start', 'defaultPageSize' => 10, 'forcePageParam' => false, ]); ...

Best HTML5 Semantics for SEO article and section?

html5,seo

This has nothing to do with SEO, the two structures have a different meaning: A list of blog posts could be a section with an article for each blog post. A long blog post could be an article with a section for each section/chapter. See also my answer with markup...

Different addresses for different products

php,seo

Are you familiar with using the $_GET query string variables in PHP? You would simply do this: echo '<a href="shoepage.php?shoe_id='.$result['id'].'">'.$result['shoename'].'</a>'; Then on shoepage.php fetch the shoe from the database with the matching ID. For better SEO, you would no doubt want to use the shoe's name (probably normalized to be...

Grails produce seo friendly URLS

grails,seo

/used-products page Create a ProductController with an action like this: def usedProducts = { [productList:Product.findAllByUsed(true)] } Then in UrlMappings, you setup your URL to point to this action: "/used-products" { controller = "product" action = "usedProducts" } product detail page Follow this excellent tutorial: http://www.inoneo.com/en/blog/12/grails/tutorial-on-how-to-create-grails-seo-friendly-url...

What should be the name of the sitemap file for Google SEO?

seo,sitemap,google-search

basically the site map file name is if XML then /sitemap.xml but its not any issue if you use any other name but we have to add this to webmaster....

How can I get better google indexing results?

seo,google-search,pagerank

I didnt see any mention of "Managed SMF hosting" on your pages, so why would you hope to rank for it ? http://static.googleusercontent.com/media/www.google.co.uk/en/uk/webmasters/docs/search-engine-optimization-starter-guide.pdf...

sitemap-tax-post_tag.xml not found - webmaster tools

wordpress,seo

Solved ! Because I used Yoast plugin. I removed it and install Google XML Sitemaps and All In One SEO Pack then you must fetch same URL's from your webmaster tools(googlebot-fetch)....

Wordpress - customized pages with blocks - prohibit google seo index of blocks

wordpress,seo,woocommerce,robots.txt,google-sitemap

Noindex tags would be useful. https://support.google.com/webmasters/answer/93710?hl=en

SPDY on shared host & SEO Sematics

seo,semantics,shared-hosting,spdy,http2

how about SEO sematics, will it harm my ranking? It won't. are there any better solutions for this problems? It is not a problem. can i use ipv6 to mask the port somehow? No....

How do I strip out ?_escaped_fragment_= using .htaccess

ajax,.htaccess,mod-rewrite,seo

This is how I rewrote it so that all ?_escaped_fragment_=/XXXXX requests got redirected to /XXXXX without the query RewriteCond %{QUERY_STRING} ^_escaped_fragment_=(.*)$ RewriteRule ^(.*)$ http://www.%{HTTP_HOST}%1? [L,R=301] This makes www.domain.com/?_escaped_fragment_=/somepage redirect (permanently) to www.domain.com/somepage ...which is just what I wanted....

Does document format change behaviour of Google bot in terms of SEO? Like /path and /path.htm .html .php [closed]

seo,googlebot

No not at all. You don't get more or lose ranking because of that. It is not worth the effort.

How do I retain search engine accessibility when loading external file into html

javascript,jquery,html,seo,cross-platform

Well the problem is that at first, the section you're about to insert data in is blank, which means that sharing this page on Facebook for example will result in just a blank box. So, what you want to achieve is a way to run that javascript before crawling the...

SEO and … tags with JavaScript functionality

javascript,html,twitter-bootstrap,seo

Then some fancy JavaScript code wil attach an event handler on the element so the product will be added to the cart when someone clicks this link. Google is capable of executing some Javascript. Hence, if the click leads to another page, it likely bots will try to follow...

My website Images not indexed by Google, Yahoo and Bing [closed]

php,codeigniter,seo

Yes, your robots.txt file is wrong. You have Disallow: /assests/[sic] and your images are in that folder... Search engines that obey this file, will not index things in the assets folder because you asked them not to. The fix is to remove that line from your robots.txt file and wait....

fullPage.js: Make all slides and sections visible in search engine results

jquery,seo,web-crawler,single-page-application,fullpage.js

You probably won't be able to force Google to index your anchor links as different pages. You will be able to index them as a single page. Google will read your page as what it really is. A single page. There are some recommendations which suggest to use the id...

Block “cloner” servers rendering content from our server

apache,seo,clone,cracking

As I can see, you web server configured to serve the same virtual host for any requested server name. Example: # telnet freeofficefinder.com 80 Trying 78.109.169.208... Connected to freeofficefinder.com. Escape character is '^]'. GET / HTTP/1.1 Host: blablabla.com HTTP/1.1 200 OK Date: Wed, 10 Jun 2015 15:54:24 GMT Server: Apache/2.4.12...

How to avoid the multiple path to same file in php using htaccess?

php,html,apache,.htaccess,seo

Add such line in your htaccess after RewriteBase RewriteRule cbn/index.php /cbn/either/index.php [R,L] ...

MixItUp vs PageSpeed Insigths

jquery,seo,pagespeed,mixitup

I've added defer attribute to the script tag and seems its working. Thanks for your help....

Have google finding .com instead of .ru

google-webmaster-tools

According to Google's content guidelines, you can use a canonical URL to setting a preferred domain, and also do this with Google webmaster tools. Put this link in your site header: <link rel="canonical" href="https://example.com"> ...

How seo implemented for distancebetween.com website?

seo,google-search

If you would take a look at the Google results for a search query site:distancesbetween.com you would see 654000 results, which basically means that most of the links generated are indexed by Google. As Rob already mentioned, you can find links to the popular searches on the website and each...

WooRank Touchscreen Readiness Failing

responsive-design,seo,responsiveness

It actually says 48px. Check your search console (formerly Google Webmaster Tools) under 'Search Traffic' > 'Mobile Usability' to see if any issues are showing up in there. You can drill down to see specific pages that are flagged as having issues.

SEO with single page application

node.js,seo,single-page-application

I've toyed around with this before. A good place to start is… http://backbonetutorials.com/seo-for-single-page-apps/ There are also services and libraries for node that will render your app server side in phantom (or the like) and serve it to the bots....

Server side vs client side website

javascript,html,ajax,html5,seo

There is only one real SEO issue at stake here: content accessibility. Some search engines don't execute Javascript as part of crawling and indexing. So if your content cannot be accessed with a URL defined in a sitemap without using Javascript, then such content will not be indexed and ranked....

Multiple modals with galleries vs. a single dynamic one

javascript,dom,seo,image-gallery,bootstrap-modal

I suppose that it is better to output all images during page load instead of loading them up using AJAX if SEO is important, right? Both methods are equivalent from an SEO perspective for Google, but it may take it a bit more time to index your content for...

Site name in Google search results for multi-language websites

html,seo,schema.org,google-rich-snippets

Whether you use different paths (…/en/ + …/de/), different subdomains (en.example.com + de.example.com) or different domains (example.com + example.net), you have different (albeit translated) websites. So use a separate WebSite item for each of your sites. Example (using RDFa) On http://example.com/en/, you could specify: <head typeof="schema:WebSite"> <title property="schema:name">Example</title> <link property="schema:url"...

Schema.org mandatory fields and the time needed until Google shows changes

seo,schema.org,google-rich-snippets

There are no mandatory properties/types in Schema.org. However, consumers of the data, like Google Search, might have rules under which conditions they will make something with your data (e.g., they are looking for specific properties). So you’d have to check their documentation. For Google Search, their Rich Snippets are documented...

Heading order in HTML5

html5,seo,semantic-markup

Both variants are valid HTML5. The HTML5 spec says about headings and sections: Sections may contain headings of any rank, and authors are strongly encouraged to use headings of the appropriate rank for the section's nesting level. So your first example is recommended by HTML5. HTML5 does not define that...

Convert MVC Website to be Responsive

mobile,model-view-controller,seo

There are typically three methods for handling Responsive Design : Using A Responsive Design Framework Using CSS Media Queries Using Percentages The ease of using each of these will ultimately be based on how complex your existing Project is and how much you are willing to do and change to...

What is more important for images - alt tag or name

seo

So now I am wondering if I should stay with this Flickr gallery, because it eases my job thousand times, or I should look for a way to have both good link (www.......red-carpet-turkey.jpg) and an alt tag. Google searches for keywords in the filename and in the alt tag...

how to create direct link to cms articles

php,mysql,seo,content-management-system

You'll need to create a slug to use in your URL. Either you allow the user to enter his slug or you create the slug automatically from the article's title. One thing to keep in mind is that you'll need to make sure the slug is unique. Imagine the title...

Change the unique generated title names of friendly-id using attribute of another table

ruby-on-rails,ruby,seo,friendly-url,friendly-id

class Company < ActiveRecord::Base ............................. def city self.address.city end end ...