We use the uploaded link files from different sources to expand the data basis for our link crawler.
In doing so, we only use the source URLs that were not featured in other data sources. In addition, we use a maximum of 100 URLs per host so as not to not put too great of a demand on the respective server and to not let sitewide links have too strong an influence on the evaluation.
