Early Web Infrastructure

Overview

The early World Wide Web required more than the ability to publish interconnected documents. As the number of available websites expanded, users also required mechanisms for locating, organizing, and retrieving information distributed across the network.

This collection brings together three Digital Objects representing different stages in the development of that environment: the first website at CERN, Yahoo! Directory, and WebCrawler.

Together they document the emergence of web publishing and two influential approaches to the subsequent problem of information discovery.

From Publication to Discovery

The first website at CERN represents the environment in which the World Wide Web itself was developed. It demonstrated a system of interconnected hypertext resources accessible through common network protocols and provided information concerning the developing Web project.

As adoption of the Web increased, the expanding number of available resources created a new problem: discovery.

Yahoo! Directory addressed this problem through human organization. Websites were selected and arranged within a hierarchy of subjects and subcategories through which users could browse.

WebCrawler represented a different approach. Automated collection and full-text indexing allowed users to locate webpages by searching for words contained within their content.

These approaches — hierarchical browsing and automated textual retrieval — represent important stages in the development of mechanisms through which users navigated the increasingly complex information environment of the World Wide Web.

Curatorial Note

The Digital Objects within this collection are not presented as a complete history of early web infrastructure or information retrieval.

Instead, they have been selected to illustrate a historical progression from the establishment of the Web as a publishing environment to the development of systems intended to make its rapidly expanding information space navigable.

The relationship between Yahoo! Directory and WebCrawler is particularly significant. Developed during the same period, they demonstrate contrasting responses to the same underlying problem: how to find useful information on a Web growing beyond the scale at which individual resources could be discovered through direct knowledge and manually maintained links alone.

The collection therefore interprets these Digital Objects not merely as surviving early Internet services, but as evidence of the evolving information architecture of the public Web.