AI Data Center Boom Drives Up Storage Costs, Threatening Internet Archiving Efforts

The rapid expansion of AI data centers has triggered a severe shortage of hard drives and storage solutions, driving up costs and complicating efforts to archive the internet. Digital archivists, academics, Wikipedia, and even hobbyist data collectors are struggling to afford and secure the storage they need to preserve digital content.

According to archivists, specific high-capacity drives once favored by prominent organizations like the Internet Archive have become either exorbitantly expensive or nearly impossible to obtain. Over the past several months, prices for both consumer and enterprise-grade solid-state drives (SSDs), hard drives, and other storage solutions have surged. For example, a 2TB external Samsung SSD purchased for $159 in fall 2023 now costs $575.

PC Part Picker, a platform that tracks storage device prices, reports a universal increase in storage costs beginning in October 2023. Many tracked drives have seen their prices double or rise by more than 150%, with some models completely sold out at major retailers. A secondary market has even emerged, with some SSDs being resold at inflated prices on platforms like eBay.

Internet Archive Faces Critical Storage Shortages

Brewster Kahle, founder of the Internet Archive and the Wayback Machine, described the storage crisis as a “very real issue costing us time and money.”

“We have found that the preferred 28-30TB drives are just not available or at very high price,” Kahle said. “We gather over 100 terabytes of new materials each day, and we have over 210 Petabytes of materials already archived on machines that need continuous upgrades and maintenance, so we need to constantly get new hard drives.”

“We are fortunate to have an active community that donates to the Archive, and we are also looking for help from hard drive manufacturers in these difficult times. We are always looking for more help,” he added. “So far we have ways to work around these shortages, but it is a very real issue causing us time and money.”

Wikipedia and Wikimedia Projects Struggle with Rising Costs

The Wikimedia Foundation, which operates Wikipedia and other projects like Wikimedia Commons, has also been impacted by the storage crisis. With over 65 million articles on Wikipedia alone, access to server and storage capacity is critical to its operations.

A Wikimedia Foundation spokesperson told 404 Media: “With over 65 million articles on Wikipedia alone, access to server and storage capacity is vital to us. We’ve certainly seen price increases since the end of 2023. These price increases are of concern to us, as with every other player in the industry. We see the primary impact in the purchase of memory and hard drives but also in terms of lead times on server deliveries and our capacity to place future orders.”

The spokesperson added that the Wikimedia Foundation, as a non-profit, carefully allocates its budget. “We maintain our own data centers to serve our users from all over the world. We’re putting workarounds in place where we can, mainly involving being smart with how we prioritize investment,” they said.