How to customize DNN robots.txt to allow a module specific sitemap to be crawled by search engines?

Tag: seo,dotnetnuke,robots.txt,googlebot

I am using the EasyDNN News module for the blog, news articles, etc. on our DNN website. The core DNN sitemap does not include the articles generated by this module, but the module creates its own sitemap.

For example: domain.com/blog/mid/1005/ctl/sitemap

When I try to submit this sitemap to Google, it says my Robots.txt file is blocking it.

Looking at the Robots.txt file that ships with DNN, I noticed the following lines under the Slurp and Googlebot user-agents:

Disallow: /*/ctl/       # Slurp permits *
Disallow: /*/ctl/       # Googlebot permits *

I'd like to submit the module's sitemap, but I'd like to know why the /ctl is disallowed for these user-agents, and what would the impact be if I just removed these lines from the file? Specifically, as it pertains to Google crawling the site.

As an added reference, I have read the article below about avoiding a duplicate content penalty by disallowing specific urls that contain /ctl such as login, register, terms, etc. I'm wondering if this is why DNN just disallowed any url with /ctl.

http://www.codeproject.com/Articles/18151/DotNetNuke-Search-Engine-Optimization-Part-Remov

Best How To :

The proper way to do this would be to use the DNN Sitemap provider, something that is pretty darn easy to do as a module developer.

I don't have a blog post/tutorial on it, but I do have sample code which can be found in

http://dnnsimplearticle.codeplex.com/SourceControl/latest#cs/Providers/Sitemap/Sitemap.cs

This will allow custom modules to add their own information to the DNN Sitemap.

The reason /CTL is disallowed is because the normal way to load the Login/Registration/Profile controls is to do site?ctl=login and that is typically not something that people want to have indexed.

The other option is just edit the robots.txt file.

DNN Can`t add reference

asp.net,dotnetnuke,dotnetnuke-7

The problem was in target framework of external lib . It was .net 4.5 , but module is on 4.0 , so that was the problem. Recompiled lib in 4.0 , add it again and everything works fine now. Thank you all for the help

How do I retain search engine accessibility when loading external file into html

javascript,jquery,html,seo,cross-platform

Well the problem is that at first, the section you're about to insert data in is blank, which means that sharing this page on Facebook for example will result in just a blank box. So, what you want to achieve is a way to run that javascript before crawling the...

Slidershow jquery and convert to css

jquery,css,html5,seo,slider

Please use class name instead of id to define css. .slider1_container { ... } To pass html validation please replace u attribute with data-u. reference: http://www.jssor.com/development/pass-html-validation.html...

Site name in Google search results for multi-language websites

html,seo,schema.org,google-rich-snippets

Whether you use different paths (…/en/ + …/de/), different subdomains (en.example.com + de.example.com) or different domains (example.com + example.net), you have different (albeit translated) websites. So use a separate WebSite item for each of your sites. Example (using RDFa) On http://example.com/en/, you could specify: <head typeof="schema:WebSite"> <title property="schema:name">Example</title> <link property="schema:url"...

Heading order in HTML5

html5,seo,semantic-markup

Both variants are valid HTML5. The HTML5 spec says about headings and sections: Sections may contain headings of any rank, and authors are strongly encouraged to use headings of the appropriate rank for the section's nesting level. So your first example is recommended by HTML5. HTML5 does not define that...

Removing the number of first page in Yii2 Pagination from the URL

.htaccess,pagination,seo,yii2

According to docs you should set yii\data\Pagination::forcePageParam to false by passing it in Pagination constructor $pages = new Pagination([ 'totalCount' => $books['booksCount'], 'pageParam' => 'start', 'defaultPageSize' => 10, 'forcePageParam' => false, ]); ...

fullPage.js: Make all slides and sections visible in search engine results

jquery,seo,web-crawler,single-page-application,fullpage.js

You probably won't be able to force Google to index your anchor links as different pages. You will be able to index them as a single page. Google will read your page as what it really is. A single page. There are some recommendations which suggest to use the id...

Multiple modals with galleries vs. a single dynamic one

javascript,dom,seo,image-gallery,bootstrap-modal

I suppose that it is better to output all images during page load instead of loading them up using AJAX if SEO is important, right? Both methods are equivalent from an SEO perspective for Google, but it may take it a bit more time to index your content for...

How to “remove” indexed pages in Google and AngularJS

angularjs,nginx,seo,google-index

The moment Google index my new Urls, I guess It will have many urls with same duplicated code, right? I´m a bit worried because I have read Google penalizes this scenarios. No, Google does not penalize duplicate content. It is a hard to die myth perpetrated over and over...

How seo implemented for distancebetween.com website?

seo,google-search

If you would take a look at the Google results for a search query site:distancesbetween.com you would see 654000 results, which basically means that most of the links generated are indexed by Google. As Rob already mentioned, you can find links to the popular searches on the website and each...

Block “cloner” servers rendering content from our server

apache,seo,clone,cracking

As I can see, you web server configured to serve the same virtual host for any requested server name. Example: # telnet freeofficefinder.com 80 Trying 78.109.169.208... Connected to freeofficefinder.com. Escape character is '^]'. GET / HTTP/1.1 Host: blablabla.com HTTP/1.1 200 OK Date: Wed, 10 Jun 2015 15:54:24 GMT Server: Apache/2.4.12...

What should be the name of the sitemap file for Google SEO?

seo,sitemap,google-search

basically the site map file name is if XML then /sitemap.xml but its not any issue if you use any other name but we have to add this to webmaster....

Canonical url for google to prevent duplicate meta?

seo,meta-tags,google-webmaster-tools

You should go to the URL parameters setting in WMT and set the variable rs to be ignored, by indicating that it does not change page content seen by users. Note that rel="canonical" is just a suggestion and not a directive. Google can choose to ignore it if it feels...

Disallow specific folders in robots.txt with wildcards

seo,search-engine,robots.txt,google-crawlers

You don't need wildcards at all for this. Your example will work, but it would work just as well without the wildcard. Trailing wildcards do not do anything useful. For example, this: Disallow: /x means: "Block any path that starts with '/x', followed by zero or more characters." And this:...

WooRank Touchscreen Readiness Failing

responsive-design,seo,responsiveness

It actually says 48px. Check your search console (formerly Google Webmaster Tools) under 'Search Traffic' > 'Mobile Usability' to see if any issues are showing up in there. You can drill down to see specific pages that are flagged as having issues.

Wordpress - customized pages with blocks - prohibit google seo index of blocks

wordpress,seo,woocommerce,robots.txt,google-sitemap

Noindex tags would be useful. https://support.google.com/webmasters/answer/93710?hl=en

How do I strip out ?_escaped_fragment_= using .htaccess

ajax,.htaccess,mod-rewrite,seo

This is how I rewrote it so that all ?_escaped_fragment_=/XXXXX requests got redirected to /XXXXX without the query RewriteCond %{QUERY_STRING} ^_escaped_fragment_=(.*)$ RewriteRule ^(.*)$ http://www.%{HTTP_HOST}%1? [L,R=301] This makes www.domain.com/?_escaped_fragment_=/somepage redirect (permanently) to www.domain.com/somepage ...which is just what I wanted....

google analytics code on landing page and cookie law

jquery,google-analytics,seo

GA opt-out You can allow people to opt-out from web tracking. It is made via cookie https://developers.google.com/analytics/devguides/collection/analyticsjs/advanced#optout <a href="javascript:gaOptout()">Click here to opt-out of Google Analytics</a> <script> // Set to the same value as the web property used on the site var gaProperty = 'UA-XXXX-Y'; // Disable tracking if the opt-out...

404 - File or directory not found

c#,asp.net,.net,iis,dotnetnuke

Check portal alias table. Maybe it is still referring the old site

Wordpress - Robotx.txt allows admin login?

wordpress,seo,robots.txt

You should create your own robots.txt file and upload it to website root directory. Follow the following steps to create and upload into root folder: Open notepad Add the following text into this file, remember add also your website sitemap path sitemap: http://www.yoursite.com/sitemap.xml User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow:...

SEO and user-friendly URLs for multi-language website

url,seo,multilingual,usability

It's better to use the second one for better SEO results. example.com/en/link example.com/ru/ссылка Google likes the tree-like structure. Of course you can use the main language without prefix. Also it's better to have all languages in one domain: domain.com/en domain.com/ru not on subdomains like this en.domain.com ru.domain.com Wish you luck....

Auto logout in DNN, once browser is closed

dotnetnuke

If you disable the "remember me" option in the Host Settings, DNN will log users out once the browser session is closed (close all windows). Is that what you need to ensure? ...

SPDY on shared host & SEO Sematics

seo,semantics,shared-hosting,spdy,http2

how about SEO sematics, will it harm my ranking? It won't. are there any better solutions for this problems? It is not a problem. can i use ipv6 to mask the port somehow? No....

how to set content edit access and No module setting access in dotnetnuke 7.4

dotnetnuke,dotnetnuke-module,dotnetnuke-7,dotnetnuke-settings

You would do this using the ModuleActions call on your VIEW control public ModuleActionCollection ModuleActions { get { var actions = new ModuleActionCollection { { GetNextActionID(), Localization.GetString("EditModule", LocalResourceFile), "", "", "", EditUrl(), false, SecurityAccessLevel.Edit, true, false } }; return actions; } } The SecurityAccessLevel.Edit there, you have View, Edit, Admin,...

Different addresses for different products

php,seo

Are you familiar with using the $_GET query string variables in PHP? You would simply do this: echo '<a href="shoepage.php?shoe_id='.$result['id'].'">'.$result['shoename'].'</a>'; Then on shoepage.php fetch the shoe from the database with the matching ID. For better SEO, you would no doubt want to use the shoe's name (probably normalized to be...

My website Images not indexed by Google, Yahoo and Bing [closed]

php,codeigniter,seo

Yes, your robots.txt file is wrong. You have Disallow: /assests/[sic] and your images are in that folder... Search engines that obey this file, will not index things in the assets folder because you asked them not to. The fix is to remove that line from your robots.txt file and wait....

DNN module uninstall how to remove settings from web.config

web-config,settings,dotnetnuke,manifest,uninstall

Fixed it: <uninstall> <configuration> <nodes> <node path="/configuration/appSettings/add[@key='InvoiceEmailSubject']" action="remove" /> Turns out that I needed "appSettings" because xml nodes are case sensitive. I also needed to use @key= as provided by Dexterity above....

how to create direct link to cms articles

php,mysql,seo,content-management-system

You'll need to create a slug to use in your URL. Either you allow the user to enter his slug or you create the slug automatically from the article's title. One thing to keep in mind is that you'll need to make sure the slug is unique. Imagine the title...

How to avoid the multiple path to same file in php using htaccess?

php,html,apache,.htaccess,seo

Add such line in your htaccess after RewriteBase RewriteRule cbn/index.php /cbn/either/index.php [R,L] ...

Schema.org mandatory fields and the time needed until Google shows changes

seo,schema.org,google-rich-snippets

There are no mandatory properties/types in Schema.org. However, consumers of the data, like Google Search, might have rules under which conditions they will make something with your data (e.g., they are looking for specific properties). So you’d have to check their documentation. For Google Search, their Rich Snippets are documented...

Dotnetnuke URL tracking

sql,dotnetnuke

You can enable the SITE LOG in host settings, that will start to track a lot of information in DNN, by default it is set to 0 days, which won't store any history. For existing portals, you will need to go into the individual portals settings and update the SITE...

AngularJS form in develop module of dotnetnuke 7

javascript,angularjs,dotnetnuke

The form tag is certainly an issue when running within DNN/ASP.NET Webforms. I did a tutorial on DNNHero.com on building Angular applications as DNN modules. Two pieces of advice: Use a div tag to add your angular controller directives: <div id="userForm" ng-controller="validateCtrl" ng-init="init(<%=this.ModuleId%>)"> Don't hard-code the ng-app directive in your...

Change the unique generated title names of friendly-id using attribute of another table

ruby-on-rails,ruby,seo,friendly-url,friendly-id

class Company < ActiveRecord::Base ............................. def city self.address.city end end ...

Convert MVC Website to be Responsive

mobile,model-view-controller,seo

There are typically three methods for handling Responsive Design : Using A Responsive Design Framework Using CSS Media Queries Using Percentages The ease of using each of these will ultimately be based on how complex your existing Project is and how much you are willing to do and change to...