Editing Web crawler (section)

== Visual vs programmatic crawlers ==
There are a number of "visual web scraper/crawler" products available on the web which will crawl pages and structure data into columns and rows based on the users requirements. One of the main difference between a classic and a visual crawler is the level of programming ability required to set up a crawler. The latest generation of "visual scrapers" remove the majority of the programming skill needed to be able to program and start a crawl to scrape web data.

The visual scraping/crawling method relies on the user "teaching" a piece of crawler technology, which then follows patterns in semi-structured data sources. The dominant method for teaching a visual crawler is by highlighting data in a browser and training columns and rows. While the technology is not new, for example it was the basis of Needlebase which has been bought by Google (as part of a larger acquisition of ITA Labs<ref>ITA Labs [http://semanticweb.com/more-semantics-for-google-ita-software-acquisition-comes-with-needlebase-too_b19329 "ITA Labs Acquisition"] {{Webarchive|url=https://web.archive.org/web/20140318134631/http://semanticweb.com/more-semantics-for-google-ita-software-acquisition-comes-with-needlebase-too_b19329 |date=18 March 2014 }} 20 April 2011 1:28 AM</ref>), there is continued growth and investment in this area by investors and end-users.{{citation needed|date=June 2021}}