Editing Web crawler (section)

==Architectures==
[[File:WebCrawlerArchitecture.svg|thumb|300px|High-level architecture of a standard Web crawler]]
A crawler must not only have a good crawling strategy, as noted in the previous sections, but it should also have a highly optimized architecture.

Shkapenyuk and Suel noted that:<ref name=shkapenyuk2002>Shkapenyuk, V. and [[Torsten Suel|Suel, T.]] (2002). [http://cis.poly.edu/tr/tr-cis-2001-03.pdf Design and implementation of a high performance distributed web crawler]. In Proceedings of the 18th International Conference on Data Engineering (ICDE), pages 357-368, San Jose, California. IEEE CS Press.</ref>

{{quote|While it is fairly easy to build a slow crawler that downloads a few pages per second for a short period of time, building a high-performance system that can download hundreds of millions of pages over several weeks presents a number of challenges in system design, I/O and network efficiency, and robustness and manageability.}}

Web crawlers are a central part of search engines, and details on their algorithms and architecture are kept as business secrets. When crawler designs are published, there is often an important lack of detail that prevents others from reproducing the work. There are also emerging concerns about "[[Spamdexing|search engine spamming]]", which prevent major search engines from publishing their ranking algorithms.