Dealing with Duplicate Content

Posted December 21st, 2006 by John

Duplicate content has plagued webmasters over the recent years because of its adverse effect in Google search rankings. For those of you who have not come across the duplicate content issue, here is a quick definition. Duplicate content refers to large blocks of web content that is exact or extremely similar to each other. This is common across forums, store items, printer-friendly formatting, mobile web use, multi language stories, and news stories. The majority of the time duplicate content is unintentional, however some people duplicate content in an attempt to manipulate search engine rankings.

You may ask, how do they manipulate Google rankings with duplicate content. Well, there are 2 techniques. First by duplicating another website’s content, that site may be penalized if the search engine believes you are the authority of the content and the other site duplicated your content. Second, if your website is more trusted or has a higher Pagerank than the other site, then you will out rank them on the search results page.

For those reasons webmasters have been in uproars with Google, and it continued in the SES Chicago conference. So, Adam Lasnik of Google answered some questions webmasters had at the Google Webmaster Central Blog.

Lasnik noted that multi language articles will not be viewed as duplicate content, as well as occasional snippets. Also, webmasters have plenty of options to address the duplicate content issue. Lasnik mentions using the robots.txt file to block the search crawler from indexing certain versions of the content, or use a 301 redirect to redirect users and spiders to one version of the content. Also, use TLDs whenever possible to handle country specific content and avoid publishing ‘empty’ pages, otherwise known as stubs. Stubs are very common on real estate and travel websites where there are directories, therefore a page may come up saying “Check out the listings for New York below”..with no actual listings.

You may ask why does Google care about duplicate content? Well the answer is easy. Would you be happy with the search results page blanketed with the same story, just on different domains? I think not. They need to diversify their results page with unique content to please their web searchers.

Therefore, when duplicate content can be avoided I would have to suggest avoiding it, but where it’s unavoidable use one of Adam’s techniques to prevent penalizing your website in any way.