Crawling a large number of websites can sometimes take hours or even days. To provide you with faster results, the Optimizer allows more than one crawler to work on the project at a time. In the crawler settings (Settings > Crawler) select the number of parallel crawlers to use for each project from the the drop-down menu.
It is possible to select from one to ten parallel crawlers for a project, where the default value is two crawlers.
As our crawler will always wait until the current page is delivered before the next page is requested, an increase from two to four crawlers allows us to query twice as many pages in the same amount of time. In this case, however, the server would also experience double the load.
