How to customize DNN robots.txt to allow a module specific sitemap to be crawled by search engines?

Tag: seo,dotnetnuke,robots.txt,googlebot

I am using the EasyDNN News module for the blog, news articles, etc. on our DNN website. The core DNN sitemap does not include the articles generated by this module, but the module creates its own sitemap.

For example: domain.com/blog/mid/1005/ctl/sitemap

When I try to submit this sitemap to Google, it says my Robots.txt file is blocking it.

Looking at the Robots.txt file that ships with DNN, I noticed the following lines under the Slurp and Googlebot user-agents:

Disallow: /*/ctl/       # Slurp permits *
Disallow: /*/ctl/       # Googlebot permits *

I'd like to submit the module's sitemap, but I'd like to know why the /ctl is disallowed for these user-agents, and what would the impact be if I just removed these lines from the file? Specifically, as it pertains to Google crawling the site.

As an added reference, I have read the article below about avoiding a duplicate content penalty by disallowing specific urls that contain /ctl such as login, register, terms, etc. I'm wondering if this is why DNN just disallowed any url with /ctl.

http://www.codeproject.com/Articles/18151/DotNetNuke-Search-Engine-Optimization-Part-Remov

Best How To :

The proper way to do this would be to use the DNN Sitemap provider, something that is pretty darn easy to do as a module developer.

I don't have a blog post/tutorial on it, but I do have sample code which can be found in

http://dnnsimplearticle.codeplex.com/SourceControl/latest#cs/Providers/Sitemap/Sitemap.cs

This will allow custom modules to add their own information to the DNN Sitemap.

The reason /CTL is disallowed is because the normal way to load the Login/Registration/Profile controls is to do site?ctl=login and that is typically not something that people want to have indexed.

The other option is just edit the robots.txt file.

How to “remove” indexed pages in Google and AngularJS

angularjs,nginx,seo,google-index

The moment Google index my new Urls, I guess It will have many urls with same duplicated code, right? I´m a bit worried because I have read Google penalizes this scenarios. No, Google does not penalize duplicate content. It is a hard to die myth perpetrated over and over...

SEO and user-friendly URLs for multi-language website

url,seo,multilingual,usability

It's better to use the second one for better SEO results. example.com/en/link example.com/ru/ссылка Google likes the tree-like structure. Of course you can use the main language without prefix. Also it's better to have all languages in one domain: domain.com/en domain.com/ru not on subdomains like this en.domain.com ru.domain.com Wish you luck....

How can I get better google indexing results?

seo,google-search,pagerank

I didnt see any mention of "Managed SMF hosting" on your pages, so why would you hope to rank for it ? http://static.googleusercontent.com/media/www.google.co.uk/en/uk/webmasters/docs/search-engine-optimization-starter-guide.pdf...

How Google “distinguishes” website articles from news? [closed]

html,seo,google-search

Google will use a number of approaches when building their databases and one cannot say exactly how you would get your site to register within google as a news sight. However you will notice the following meta information within many articles that show up on google news. <meta property="og:type" content="article"...

Block “cloner” servers rendering content from our server

apache,seo,clone,cracking

As I can see, you web server configured to serve the same virtual host for any requested server name. Example: # telnet freeofficefinder.com 80 Trying 78.109.169.208... Connected to freeofficefinder.com. Escape character is '^]'. GET / HTTP/1.1 Host: blablabla.com HTTP/1.1 200 OK Date: Wed, 10 Jun 2015 15:54:24 GMT Server: Apache/2.4.12...

Disallow specific folders in robots.txt with wildcards

seo,search-engine,robots.txt,google-crawlers

You don't need wildcards at all for this. Your example will work, but it would work just as well without the wildcard. Trailing wildcards do not do anything useful. For example, this: Disallow: /x means: "Block any path that starts with '/x', followed by zero or more characters." And this:...

Auto logout in DNN, once browser is closed

dotnetnuke

If you disable the "remember me" option in the Host Settings, DNN will log users out once the browser session is closed (close all windows). Is that what you need to ensure? ...

Site name in Google search results for multi-language websites

html,seo,schema.org,google-rich-snippets

Whether you use different paths (…/en/ + …/de/), different subdomains (en.example.com + de.example.com) or different domains (example.com + example.net), you have different (albeit translated) websites. So use a separate WebSite item for each of your sites. Example (using RDFa) On http://example.com/en/, you could specify: <head typeof="schema:WebSite"> <title property="schema:name">Example</title> <link property="schema:url"...

Wordpress - Robotx.txt allows admin login?

wordpress,seo,robots.txt

You should create your own robots.txt file and upload it to website root directory. Follow the following steps to create and upload into root folder: Open notepad Add the following text into this file, remember add also your website sitemap path sitemap: http://www.yoursite.com/sitemap.xml User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow:...

MixItUp vs PageSpeed Insigths

jquery,seo,pagespeed,mixitup

I've added defer attribute to the script tag and seems its working. Thanks for your help....

What should be the name of the sitemap file for Google SEO?

seo,sitemap,google-search

basically the site map file name is if XML then /sitemap.xml but its not any issue if you use any other name but we have to add this to webmaster....

Heading order in HTML5

html5,seo,semantic-markup

Both variants are valid HTML5. The HTML5 spec says about headings and sections: Sections may contain headings of any rank, and authors are strongly encouraged to use headings of the appropriate rank for the section's nesting level. So your first example is recommended by HTML5. HTML5 does not define that...

Does DNN Have anything similar to Views in Drupal or ACF in Wordpress?

dotnetnuke,dotnetnuke-7

DNN have recently announced that DNN 8 will include the Dynamic Content Creator API and modules, through which the platform will support these sorts of custom content types (i.e. collections of fields). Other 3rd party modules that are in that space include: Form & List (formerly User Defined Table) 2sxc...

Removing the number of first page in Yii2 Pagination from the URL

.htaccess,pagination,seo,yii2

According to docs you should set yii\data\Pagination::forcePageParam to false by passing it in Pagination constructor $pages = new Pagination([ 'totalCount' => $books['booksCount'], 'pageParam' => 'start', 'defaultPageSize' => 10, 'forcePageParam' => false, ]); ...

Schema.org mandatory fields and the time needed until Google shows changes

seo,schema.org,google-rich-snippets

There are no mandatory properties/types in Schema.org. However, consumers of the data, like Google Search, might have rules under which conditions they will make something with your data (e.g., they are looking for specific properties). So you’d have to check their documentation. For Google Search, their Rich Snippets are documented...

Grails produce seo friendly URLS

grails,seo

/used-products page Create a ProductController with an action like this: def usedProducts = { [productList:Product.findAllByUsed(true)] } Then in UrlMappings, you setup your URL to point to this action: "/used-products" { controller = "product" action = "usedProducts" } product detail page Follow this excellent tutorial: http://www.inoneo.com/en/blog/12/grails/tutorial-on-how-to-create-grails-seo-friendly-url...

AngularJS form in develop module of dotnetnuke 7

javascript,angularjs,dotnetnuke

The form tag is certainly an issue when running within DNN/ASP.NET Webforms. I did a tutorial on DNNHero.com on building Angular applications as DNN modules. Two pieces of advice: Use a div tag to add your angular controller directives: <div id="userForm" ng-controller="validateCtrl" ng-init="init(<%=this.ModuleId%>)"> Don't hard-code the ng-app directive in your...

SEO with single page application

node.js,seo,single-page-application

I've toyed around with this before. A good place to start is… http://backbonetutorials.com/seo-for-single-page-apps/ There are also services and libraries for node that will render your app server side in phantom (or the like) and serve it to the bots....

Different addresses for different products

php,seo

Are you familiar with using the $_GET query string variables in PHP? You would simply do this: echo '<a href="shoepage.php?shoe_id='.$result['id'].'">'.$result['shoename'].'</a>'; Then on shoepage.php fetch the shoe from the database with the matching ID. For better SEO, you would no doubt want to use the shoe's name (probably normalized to be...

Server side vs client side website

javascript,html,ajax,html5,seo

There is only one real SEO issue at stake here: content accessibility. Some search engines don't execute Javascript as part of crawling and indexing. So if your content cannot be accessed with a URL defined in a sitemap without using Javascript, then such content will not be indexed and ranked....

My website Images not indexed by Google, Yahoo and Bing [closed]

php,codeigniter,seo

Yes, your robots.txt file is wrong. You have Disallow: /assests/[sic] and your images are in that folder... Search engines that obey this file, will not index things in the assets folder because you asked them not to. The fix is to remove that line from your robots.txt file and wait....

404 - File or directory not found

c#,asp.net,.net,iis,dotnetnuke

Check portal alias table. Maybe it is still referring the old site

how to set content edit access and No module setting access in dotnetnuke 7.4

dotnetnuke,dotnetnuke-module,dotnetnuke-7,dotnetnuke-settings

You would do this using the ModuleActions call on your VIEW control public ModuleActionCollection ModuleActions { get { var actions = new ModuleActionCollection { { GetNextActionID(), Localization.GetString("EditModule", LocalResourceFile), "", "", "", EditUrl(), false, SecurityAccessLevel.Edit, true, false } }; return actions; } } The SecurityAccessLevel.Edit there, you have View, Edit, Admin,...

What is more important for images - alt tag or name

seo

So now I am wondering if I should stay with this Flickr gallery, because it eases my job thousand times, or I should look for a way to have both good link (www.......red-carpet-turkey.jpg) and an alt tag. Google searches for keywords in the filename and in the alt tag...