Menu
  • HOME
  • TAGS

How to customize DNN robots.txt to allow a module specific sitemap to be crawled by search engines?

Tag: seo,dotnetnuke,robots.txt,googlebot

I am using the EasyDNN News module for the blog, news articles, etc. on our DNN website. The core DNN sitemap does not include the articles generated by this module, but the module creates its own sitemap.

For example: domain.com/blog/mid/1005/ctl/sitemap

When I try to submit this sitemap to Google, it says my Robots.txt file is blocking it.

Looking at the Robots.txt file that ships with DNN, I noticed the following lines under the Slurp and Googlebot user-agents:

Disallow: /*/ctl/       # Slurp permits *
Disallow: /*/ctl/       # Googlebot permits *

I'd like to submit the module's sitemap, but I'd like to know why the /ctl is disallowed for these user-agents, and what would the impact be if I just removed these lines from the file? Specifically, as it pertains to Google crawling the site.

As an added reference, I have read the article below about avoiding a duplicate content penalty by disallowing specific urls that contain /ctl such as login, register, terms, etc. I'm wondering if this is why DNN just disallowed any url with /ctl.

http://www.codeproject.com/Articles/18151/DotNetNuke-Search-Engine-Optimization-Part-Remov

Best How To :

The proper way to do this would be to use the DNN Sitemap provider, something that is pretty darn easy to do as a module developer.

I don't have a blog post/tutorial on it, but I do have sample code which can be found in

http://dnnsimplearticle.codeplex.com/SourceControl/latest#cs/Providers/Sitemap/Sitemap.cs

This will allow custom modules to add their own information to the DNN Sitemap.

The reason /CTL is disallowed is because the normal way to load the Login/Registration/Profile controls is to do site?ctl=login and that is typically not something that people want to have indexed.

The other option is just edit the robots.txt file.

AngularJS form in develop module of dotnetnuke 7

javascript,angularjs,dotnetnuke

The form tag is certainly an issue when running within DNN/ASP.NET Webforms. I did a tutorial on DNNHero.com on building Angular applications as DNN modules. Two pieces of advice: Use a div tag to add your angular controller directives: <div id="userForm" ng-controller="validateCtrl" ng-init="init(<%=this.ModuleId%>)"> Don't hard-code the ng-app directive in your...

SEO and … tags with JavaScript functionality

javascript,html,twitter-bootstrap,seo

Then some fancy JavaScript code wil attach an event handler on the element so the product will be added to the cart when someone clicks this link. Google is capable of executing some Javascript. Hence, if the click leads to another page, it likely bots will try to follow...

Heading order in HTML5

html5,seo,semantic-markup

Both variants are valid HTML5. The HTML5 spec says about headings and sections: Sections may contain headings of any rank, and authors are strongly encouraged to use headings of the appropriate rank for the section's nesting level. So your first example is recommended by HTML5. HTML5 does not define that...

How can I get better google indexing results?

seo,google-search,pagerank

I didnt see any mention of "Managed SMF hosting" on your pages, so why would you hope to rank for it ? http://static.googleusercontent.com/media/www.google.co.uk/en/uk/webmasters/docs/search-engine-optimization-starter-guide.pdf...

Robots.txt file in MVC.NET 4

asp.net,asp.net-mvc-4,seo,robots.txt

do robots crawl the controllers which has [Authorization] attribute like Administration If they find a link to it, they are likely to try and crawl it, but they will fail just like anyone with a web browser that does not log in. Robots have no special ability to access...

Grails produce seo friendly URLS

grails,seo

/used-products page Create a ProductController with an action like this: def usedProducts = { [productList:Product.findAllByUsed(true)] } Then in UrlMappings, you setup your URL to point to this action: "/used-products" { controller = "product" action = "usedProducts" } product detail page Follow this excellent tutorial: http://www.inoneo.com/en/blog/12/grails/tutorial-on-how-to-create-grails-seo-friendly-url...

si​tem​ap-​tax​-po​st_​tag​.xm​l not found - webmaster tools

wordpress,seo

Solved ! Because I used Yoast plugin. I removed it and install Google XML Sitemaps and All In One SEO Pack then you must fetch same URL's from your webmaster tools(googlebot-fetch)....

DNN module uninstall how to remove settings from web.config

web-config,settings,dotnetnuke,manifest,uninstall

Fixed it: <uninstall> <configuration> <nodes> <node path="/configuration/appSettings/add[@key='InvoiceEmailSubject']" action="remove" /> Turns out that I needed "appSettings" because xml nodes are case sensitive. I also needed to use @key= as provided by Dexterity above....

Best JSON-LD practices: using multiple