Will We Ever Solve the Problem of Losing Data?

More and more, our data—private photos, work documents, confidential information, and so on—are migrating to digital clouds and platforms. In a more recent and perhaps controversial turn of events, some gaming firms are pushing for digital downloads in lieu of physical CDs.

Read more Help! I Generated a Song With Google’s New Music Model and I Kind of Like It

One recurring argument against digital options has been that it’s not as “permanent” as their physical counterparts. After all, aren’t physical storage systems safer from the whims of a tech firm wiping your cloud storage—or, even worse, forcing you to choose a subscription plan?

But CDs break. Books burn. USBs go defunct. Personally, I—a stickler for physical media—have a very bad track record of breaking my game chips and scratching my movie discs. There are real challenges in dealing with physical storage systems, too, and big players like Microsoft are investing significant resources to overcome the inevitable decay caused by, well, the passage of time.

So, for this Giz Asks, we asked experts in computer science, cybersecurity, and archival research to consider the idea of “permanence,” specifically in the context of how we record, track, and store information. That is, is there such a thing as permanent data storage? What are the main issues with data storage mediums that exist today? Most importantly, can we do something about it—if there’s something to be done at all?

The following responses may have been lightly edited for length and clarity.

Mohiuddin Ahmed

Computer scientist, Adelaide University, Australia.

Data storage is a contextual matter. Given the transient nature of everything around us, permanent data storage is some sort of a myth at this stage. That said, multiple copies of data and moving them to a new storage every couple of years can be thought of as a practical solution toward permanent data storage.

Francesca Musiani

Research director, Center for Internet and Society at the National Center for Scientific Research, France.

I find the idea of “permanent” data storage quite elusive, not only in our digital “era.” Every medium we have ever invented eventually fails, from stone inscriptions that erode to paper burning, CDs’ “disc rot” and solid-state drives that gradually lose stored charge. Further, even if a storage medium could survive for centuries, the devices and software needed to read it may not; a perfectly preserved floppy disk is of little use if no one can build or maintain a compatible drive. In this sense, data loss is a techno-historical problem, I would say: preserving information requires preserving the means of interpreting it.

Cloud computing has redistributed this challenge but not eliminated it. Storing data with a technology company protects users from some risks, such as local hardware failure; but it introduces or complicates others, including changing business models, account closures, service shutdowns, and corporate decisions about what deserves to be retained. As often said but always good to remember, the ‘cloud’ relies on physical infrastructures, continuous electricity, economic incentives, and organizations that may not exist indefinitely. What looks permanent is often a promise of, and reliance on, ongoing maintenance.

This is why many of the most promising approaches to long-term preservation focus on building resilient systems. Emerging technologies may dramatically extend the lifespan of the physical substrate, perhaps to hundreds or even thousands of years. Yet even these innovations cannot solve problems of accessibility, interpretation, or institutional continuity. Long-term preservation ultimately depends on redundancy, open standards, regular migration to newer media, distributed archives, and communities committed to maintaining them. In other words, information survives because it is repeatedly cared for.

Perhaps, then, we can partially reframe the original question by asking what kinds of societies are capable of preserving knowledge across generations. Libraries, archives, scientific repositories, and decentralized preservation projects have long understood that durability comes from constant stewardship. Data is never simply stored once and for all; it is continuously recreated through copying, verification, and migration.

We are unlikely ever to invent a storage medium that makes data loss impossible: entropy, technological change, and institutional fragility ensure that some information will always disappear. But this does not mean preservation is futile. It means that permanence is not an attribute of particular artifacts but a set of ongoing practices. Finding a forever indestructible device appears unlikely, yet the future of our digital memory will depend on sustaining the infrastructures, institutions, and collective commitments that keep information alive. “Permanence” of data storage, if looked through the lens of science and technology studies, infrastructure studies and the “material turn” in the study of digital technologies, is a property of ongoing social, technical, and institutional maintenance.

Jack Cushman

Director, Library Innovation Lab at Harvard Law School; creator of the Public Data Project.

I know of one data storage form that can be permanent: a library. Or rather, a world full of libraries. Libraries aren’t perfect storage devices (nothing is), but they are fiercely committed to self-repair. A good library is a collection of people, technologies, and practices that pass an obligation—to remember the things we must remember—from the last generation to the next.

Libraries, rather than better hard drives, are what solve correlated failure, cataloged in the LOCKSS (Lots Of Copies Keep Stuff Safe) threat model. A correlated failure is when you make really great, durable copies of the data you care about, and then they all get hit by an asteroid. Shouldn’t have had them all in one physical location. Or a hacker marks them all for deletion (they were all behind one sysadmin), or a government orders them all destroyed (all in one regulatory regime), or they all rot at the same time (all from one flawed batch), or the money to store them dries up (all on one funding source).

On a long enough timeline any copy will break. You need multiple copies with different vulnerabilities so they don’t all break at the same time. Then you need to repair the broken ones. A worldwide network of libraries does the repair work, whether the objects are 30 medieval Magna Carta manuscripts or 300,000 government datasets.

Read more EU Will Reportedly Label ChatGPT and Roblox ‘Very Large Online Platforms’ (Derogatory)

Long-lived storage media are wonderful (“a CD that lasts 1,000 years!”) because they invite us to imagine who will be here in 1,000 years and what they will care about. As we build a digital civilization that can preserve knowledge for centuries, the problems we solve will transcend storage media: how to build digital archives that are endowed and defended, and how to knit them into networks of mutual aid that reach across institutions and borders. Technology is a necessary building block, but permanence isn’t something we buy. It’s a promise we keep renewing by agreeing to hold each other’s treasures.

Kevin Curran

Computer scientist and cybersecurity researcher, Ulster University, U.K.

No form of digital storage media is truly reliable for passive, long-term archival use. Entropy eventually wins, failures occur faster than expected, and the only practical solution is active management—periodic verification, rewriting data, and maintaining multiple redundant copies on fresh media. The most common form of storage which is hard drives—should not be viewed as archival media. Hard drives should be treated as consumables.

A recent Iron Mountain report highlighted that one-fifth of 1990s era hard drives sent to them were entirely unreadable, even under ideal storage conditions. Quite simply, standard hard drives were not designed for long-term archival use. You can never decouple the magnetic disks from the reading hardware inside so if either fails, then the whole drive dies. Hard drives magnetic charge fades over time, bearings seize, and there is often no partial recovery option. There are methods to try to combat this such as periodic full rewrites to refresh the magnetic domains but how many would ever do this?

More up-to-date solid state drives (SSDs) aka flash memory are also poor for unpowered long-term storage. SSD cells are essentially leaky capacitors so when left unpowered, charge slowly leaks away, leading to bit errors, uncorrectable sectors, and eventual data loss. Temperature has a dramatic effect as higher temperatures accelerate leakage. SSDs may retain data for only 12 months at 30 degrees C [86 degrees F] when unpowered and enterprise drives can be even shorter under higher temperatures. Real-world testing of SATA SSDs left unpowered for up to two years showed ECC errors, corrupted files, and dramatically slower verification times, especially on worn drives.

Other common media also suffer flaws such as CD-R/DVD-R discs which suffer from rot, physical warping  and binder degradation. Tape suffers from substrate separation, binder issues, and generational incompatibility and newer drives often cannot read older tapes. Therefore, even these common “archival” formats eventually require migration.

We should never assume anything left on a shelf for years will still be readable. We should aim to maintain multiple independent copies (the classic 3-2-1 rule) and periodically power on, verify, and rewrite the data to strengthen the bits. Ultimately, we cannot trust any medium, so we should practice copying important things over and over, into fresh storage.

Adnene Guabtni

Principal research scientist, Commonwealth Scientific and Industrial Research Organisation, Australia.

Data loss, whether from hardware failure, natural disaster, cyberattack, human error, or simply a discontinued cloud service, has plagued computing since its inception. Solving it requires a multi-faceted approach combining data replication and distribution, decentralization, and data sovereignty. First, data replication and distribution across multiple cloud storage services reduces the risk of data loss from a single point of failure. This approach costs more, since the user must subscribe to multiple providers. Yet, it doesn’t protect against a global outage or a coordinated cyberattack affecting several providers at once.

Second, data decentralization, such as the use of a blockchain network, distributes data across thousands of independent nodes rather than relying on any single custodian. Once written to a sufficiently decentralized chain, data is replicated globally and cryptographically verified, so no single disaster, company collapse, or bad actor can erase it. For data to be lost, the entire blockchain network would have to disappear. Contrary to common belief, a blockchain network can still shut down due to financial pressure, loss of developer traction, or a collapsing crypto economy. Furthermore, storing large volumes of data directly on-chain is expensive and slow, which is why most systems store cryptographic proofs and pointers on-chain while the actual data is stored off-chain in distributed storage networks that are economically incentivized to keep it available. Despite its drawbacks, blockchain-based data storage comes close enough to permanent, tamper-proof, loss-resistant storage.

Third, data sovereignty addresses a different failure mode: loss of access and control, not just loss of the data itself. Data can be effectively “lost” to its owner if a platform revokes access, changes its terms, gets acquired, or ceases operating in a jurisdiction. Data sovereignty means the individual who owns the data retains cryptographic control over it. The individual would hold and manage their own cryptographic keys, instead of relying on a third-party cloud-based provider to manage access control on their behalf. This brings back control into the hands of the users, letting them decide who gets access without depending on a platform’s goodwill or solvency. Ownership and access rights are enforced by cryptography and consent rather than by a company’s terms of service. Self-sovereign identity tools and encrypted personal data vaults are at an early stage of adoption and represent a solid attempt at solving the problem of data loss, particularly when combined with data replication across decentralized infrastructure. Without a data sovereignty layer, decentralization alone doesn’t fully solve data loss, as it just moves the point of failure from infrastructure to permissions.

Melanie Hubbard

Head of Digital Scholarship & Data Services, Boston College Libraries

When we talk about data these days, we typically mean data that is born-digital or digitized. It is the digital aspect that adds to its instability, though it is also what makes it more discoverable, accessible, usable, and manageable. From the moment something is made digital, it becomes harder to grasp, like sand sifting through fingers, as the risks of degradation, context loss, format obsolescence, container vulnerability, and shifting technology loom. For instance, I can’t plug in a FireWire drive or load a CD directly on my laptop anymore. I can plug in my USB drive, but the data is corrupted.

The closest we come to data permanence is when it is in a pure analog format, such as books, punchcards, and stone tablets, but these are not practical for data storage and access at scale, and analog offers no guarantees. After all, ancient libraries stand in ruins today, if they stand at all. And even when technological limitations don’t cause data loss, there are always the risks posed by politics, corporate greed, the laws of thermodynamics, or natural disaster.

My colleagues and I are readily aware of the challenges of data impermanence, and we do everything we can to stave it off. As librarians and archivists, we do what we do not because we believe we can completely stop the losses but because we are duty-bound and possess the skills and knowledge to mitigate them. To that end, if there is any hope, it won’t be the machines that save the data we want to preserve; it will be the people.

Giz Asks is a recurring Gizmodo series in which experts answer big questions in their own words, offering a range of perspectives on the ideas, discoveries, and debates that affect our lives and shape our understanding of the world.

Read more Meta Struggles With Limited Returns on Its AI Spending, Social Media Legal Woes

By admin

Leave a Reply

Your email address will not be published. Required fields are marked *