Is it a case of duplicate content if content is made available both as an HTML page and in PDF format? We show you what is important.
HTML and PDF = Duplicate Content?
From a technical standpoint, that would be a case of internal duplicate content. External duplicate content, on the other hand, occurs, for example, if the manufacturer’s user manual for every product in your online shop is offered there as a downloadable PDF file, while also being available on the manufacturer’s website – and presumably also in other online shops.
Google says that, in the case of internal duplicate content, they usually prefer the HTML version and display it in the search results. If this scenario does not occur too often on your website, you usually do not need to worry too much about it.
If Google were to show a duplicate content warning in the Google Search Console (GSC) under the “HTML improvements” menu, for example, you could block the PDF document through an entry in the robots.txt and thereby keep the Google-Bot from crawling the file.
- Please keep in mind: If you block a URL in the robots.txt, it may still appear in the search results.
Alternatively, you can exclude the PDF file from being indexed by using the x-robots-tag in the HTTP header, or refer to the HTML version via canonical.
- More on the noindex in the x-robots-tag in the HTTP header: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag?hl=en
- More on the rel=”canonical” in the HTTP header: https://developers.google.com/search/blog/2011/06/supporting-relcanonical-http-headers
In the case of the external duplicate content in the example above, it is advisable to use a rel=”canonical” in the HTTP header of the PDF file with the manufacturer or the original website as the source.
Should PDF files really be crawled and indexed?
When using PDF files on your website, you should always ask yourself whether you primarily want to rank with them. If not, you should exclude these files from being indexed by the Google-Bot, taking into account the crawling & index budget of your website.
What Google says
You generally do not need to worry about duplicate content in a situation like this, even if you decide to mirror the content of your PDFs on HTML pages. If we recognize the URLs as containing duplicate content, we'll just show one of them to users when they search; your site generally wouldn't have any disadvantage by doing this.
Source: John Mueller
