web crawler

or web·crawl·er

[ web-kraw-ler ]

noun

, Digital Technology.

1. a computer program that digitally surveys websites, as in order to index web pages for a search engine.

Other Words From

• web crawling noun

Word History and Origins

Origin of web crawler¹

First recorded in 1990–95

Example Sentences

Publishers could also try to protect their content from Google by forbidding its web crawler from sharing any content snippets from their sites.

From New York Times

The company’s patent application details its use of a “web crawler” to acquire images, even noting that “online photos associated with a person’s account may help to create additional records of facial recognition data points,” which its machine learning algorithm can then use to find and identify matches.

From The Verge

The company originally operated a web crawler that searched for GIFs on the internet.

From Slate

He rewrote the code for a slow web crawler that Mr. Page had created to understand the relationship between links on different websites.

From New York Times

He went on to explain how one could use a Web crawler to capture masses of data and mark voters as having been contacted by the campaign in order to alter its metrics.

From The New Yorker