Overview

WebCrawler was developed by Brian Pinkerton at the University of Washington during 1994. The project began as a desktop application for locating information on the World Wide Web before being adapted into a publicly accessible web search service.

WebCrawler became publicly available on 20 April 1994 with an index containing material drawn from several thousand websites.

Its principal historical significance lies in full-text web search. WebCrawler enabled users to search for words occurring throughout the content of indexed webpages rather than relying solely on manually organized categories or limited document metadata.

This approach anticipated a fundamental characteristic of subsequent web search engines: automated collection and indexing of webpage content followed by user retrieval through textual queries.

WebCrawler subsequently passed through several owners, including America Online and Excite. By 2001, WebCrawler no longer operated its original independent search index and the name continued through later search implementations.

Preservation Note

A website using the WebCrawler name remains publicly accessible. The Bureau does not consider continuity of the WebCrawler brand and domain sufficient evidence that the original 1994 search system remains operational.

For catalogue purposes, this record represents the original WebCrawler full-text search and indexing service and its subsequent development as an independent search system.

Later services operating under the WebCrawler name are treated as successor implementations rather than uninterrupted instances of the Digital Object represented by this record.

Surviving historical interfaces, documentation, archived webpages, and technical records provide evidence of earlier states of the service. The Bureau therefore records the original WebCrawler system as an archival preservation case.