Anyone that's never had the (dis)pleasure of working with tapes, there's some gotchas to know about before archiving and forgetting about it.
When you create an archive, do not skip the step of reading it back. I'm not talking about just having the system confirm it is readable. I mean actually restoring the data from the tape. I've had tapes fail to retrieve data off of the tape even though the writing app confirmed it was readable.
Drives can have issues reading data written by another drive. Drives can have issues reading a tape it wrote itself.
Why write one when you can write two at twice the time/cost? In all of my experience with tape, I've never written just one tape. In the early days of DVDs, the images were written to LTO to send to have the discs pressed. Two were always sent on the chance one wasn't readable and schedules were so tight that the loss of a couple of days to send a new tape would wreck delivery dates. Another facility wrote three tapes: one kept on site, one sent to IronMountain, one sent to the client directly. Plenty of occasions saw the local copy unreadable and the IronMountain copy was retrieved. These were managed by large robotic tape libraries so each tape was written in a different drive at the same time so it was 3x cost while only 1x time.
Use the drive cleaning cassette as recommended. Your dirty heads can cause the data to not be written/read correctly depending on which head is dirty. If you're having issues reading, clean the heads before giving up on the tape. If you're having issues reading a tape just written, do not think someone else will be able to read it.
Ensure whoever might be using a tape you've written understands what was used to write the tape. Not all archivers are compatible. If you use the tar command to write to tape, people using a GUI app won't be able to read it.
I have worked with 5 different LTO tapes versions, 5 different other cassette formats (DAT, DDS, etc), and open spools… never, ever, ever in the 15 years had ANY problem (I’ve probably used 1000 of them). We did cycle them very often, we did have proper maintenance of the drives, through. Now that, compared with RAID Disk arrays, where we had monthly failures, was much better. Also I would not trust the „semi-permanent“ NVMEs, where after 1 year without power they get basically reset
> Also I would not trust the „semi-permanent“ NVMEs, where after 1 year without power they get basically reset
Retention decreases exponentially with temperature, but the flip side is that it also increases exponentially when refrigeration is added. A conservative estimate is 30 years at -20 Celsius. Another added benefit of flash is that it's now much denser than tape (~10x) and doesn't need to be moved - it can be powered down on the storage rack (and powered up for refresh every 6 months or so). Unfortunately, it's still ludicrously expensive (and even more at the higher densities).
Yes, the only real disadvantage is the too high price per TB in comparison with both tapes and HDDs, because otherwise, as you say, keeping it in a freezer would solve the retention problem.
Yeah, we'd do 1000 in a year easy, but that number is easy to grow when every write takes 3 tapes. We had 3 robot towers each with 10 drives in them in our facility, and there were others at other facilities. There was an entire department for interacting with them. If the tape with your data was in a robot, you could have the restore started within an hour. If the tape was in the onsite library, it would be within a few hours as they decided what tape to unload from a robot.
Then you should have worked at $dayjob... our DDS drives would fail pretty much once a year after being run for 12 hours a day/365. We'd clean them once a week, regardless of whether it said it wanted it, and they still failed all over the place. $vendor finally moved the backups to nfs storage.
When management doesn't buy replacement tapes and you're forced to recycle other tapes that are probably 10 years old, you kindof do what you can... I think it was probably better to run the cleaning tape, than let ancient tapes spread particles all over the head. /shrug
When I hear wildly differing stories like this I often wonder how much environmental conditions play a factor in the different experiences that people have.
Perhaps one of you is living in a much more humid and/or hot environment that causes issues with the writing process.
Tapes were constantly kept in the data center, so constant temperature and humidity 25C/50%. As I said, Tapes were changed after N uses, drives properly serviced. So if you leave them at home, in the sun in summer and in winter expose them to 10C with humidity from 10% to 90%, old drives, never clean heads… IdK
That all being said, when I started working there, in my ignorance, I said “TAPES?!?! They are not reliable” due to my experience with audio tapes. But I was laughed at, and they told me about people putting 10 9 track spools in a car [0] (by the way, the same we used for billing backups!) and they went literally thousands of miles driving on partially dirt roads, in summer w/o AC, doing updates to systems, and also never ever had an issue. They were so sure they would work, that they did not take a second set, even when driving 1000 miles, and a read error would mean going back!!!
Usually it wasn't the tapes that were unreliable, but the tape devices (firmware) and/or the drivers. Years back I used to work on a backup product - one of the tape devices would write a random amount of zeroes after each round of N megabytes written to the tape. So if you were "lucky" and all the backups were below N megabytes in size, everything worked perfectly. We weren't even aware there was an issue until some of our customers started complaining.
Debugging this was really fun, especially due to the randomness factor. :)
This. Environmental is everything with tape. If you don't know your temps (operational, transport, storage), RH and particulate counts then you probably shouldn't be using tape at all.
Understanding why this is as important as it is requires some visualization of the bit densities of modern tape drives, the capacity is enormous, the tape is ridiculously thin and the bits are small enough that it doesn't take much to make a part of a tape unreadable beyond what the error recovery logic can deal with. And that's before we get into longer term effects such as layer-to-layer copy and demagnetization.
Don't they ever turn into "tape salad", which was a very common issue with audio tapes (at least in car radios - in my top of the line Walkman I never had any tape salad)?
1000 tapes over 15 years sounds doesn't sound like a lot tbh.
I suspect that the number of problems you encounter from tapes scales superlinearly because they physically move. This leads to a lot of unpredictable failure modes. HDD racks also fail a lot, but in more predictable ways because all the data is managed in software.
And supply factors. Given that one of these two people spoke of multiple types of type and multiple types of drives, it’s possible that the other one just only ever worked with very high quality media.
Seriously, TF you were using to hit so many problems.
We use LTO, and I think I saw ONE failed tape in 15 years and it failed during write. We do restores regularly on a pool of tapes that is rotated so each tape is touched at least every few months and it. Far lower failure rate than hard drives
Drives can be a bit temperamental when they got older but it's all generally reporter before actual failure
> Use the drive cleaning cassette as recommended. Your dirty heads can cause the data to not be written/read correctly depending on which head is dirty. If you're having issues reading, clean the heads before giving up on the tape. If you're having issues reading a tape just written, do not think someone else will be able to read it.
Every tape library we used that was automatic, the drives asked for tape and the tape library just looked up in which slot it is.
> Every tape library we used that was automatic, the drives asked for tape and the tape library just looked up in which slot it is.
This is an interesting point - these days, tape is in the very top end of the market, both when you have too much data to fit on spinning metal (it's twice as dense in volume), and have regulators making sure you have off-site offline archives going back a number of years.
An interesting shift is happening, however - flash is now much denser per volume than tape, and can be re-written more times. It needs periodic refreshing, but, if you keep it at, say, -10C it should keep data for decades. As data volumes increase, flash might become an option for near and off-line storage. The on and near-line archives also would look a lot more like racks of storage servers, because the medium has a built-in drive and doesn't need a robot moving it to a set of mechanical devices. Downside is that it's thousands of times more expensive than tape.
Are there different qualities of Flash memory (as in different kinds, different categories? Seriously, by now every MicroSD I've ever used in phones has failed rather quickly, and I wasn't buying noname stuff, just middle of the road medium capacity microSDs, so I would be veeery careful to trust any flash-based medium. OTOH, phones have never failed me, so maybe they use different memory than is built in micro SDs? Or did the contacts go bad (unlikely, because I put stuff in phone, set and forget)?
Yup, take the following as an inaccurate joke as a whole but with realistic elements:
The highest grade, reliability wise, goes in industrial grade products; the next following grade goes in name-brand SSDs; the lower ones in mass-market memory cards. One day an employee at a famous NAND manufacturer said "That's not even worth of a C grade", on that day the USB flash drive was invented...
Reality, especially modern one (as in early 2010s+n), is obviously more complicated, all sorts of consumer products are made with all sorts of chips, with name brands and even specific models/price class being only loosely indicative of quality, specifications (such as density: SLC/MLC/TLC/QLC/PLC - how many bits crammed into each capacitor) not always available and in any case not contractual with respect to the model, counterfeiting and/or Flash Fraud (inflated capacity with address-overflow overwriting) still ubiquitous in certain segments, ...
eMMCs are indeed generally a tier above the common microSD in reliability (compensating for soldered vs replaceable?) and in performance (how many UHS-II cards and readers have you seen in person? Most eMMCs are designed for, and used in, 8-bit mode and are usually fast enough to benefit from it), never mind that your phone might be using a newer and faster technology like UFS for its internal storage!
That said, your specific example is unlikely to be coincidence; you're suspecting the contacts (which indeed I find to often go bad on the modern microSD+SIM tray designs), so do the cards work in an USB microSD reader? (Avoid micro/miniSD to SD adapters, their thickness-constrained design also loosens and goes bad at an appalling rate!)
This sounds like OP has had the fun of using DAT tapes as backup.
I worked in VFX and had the joy of looking after a few different tape systems.
Ranging from a single drive HP MSL4048 tape "robot" right up to various multi-rack monsters with 25 LTO6 drives.
First, you need good software, that is a critical thing. If you are making your own around tar, uts possible, but you need to test it. There are(or were) some opensource robot drivers, but they are somewhat painful to use (although LLMs are your friend here)
With big robots, the pain points are: mechanical.
THe systems I used had CCTV cameras to see what was going on. When it worked, the 25 drive monster could saturate 30gigs a second. THe issues we had were making sure the nearline had enough bandwidth to feed the library cache, and making sure the library cache was fast enough.
However the the robot used to get sad, it would drop tapes, randomly post tapes to the wrong slot, then get upset, or a drive would be sad and need vacuuming. The data was fine. But we regually pushed and pulled petabytes from it.
I lost nearly all of the software I wrote while at Caltech due to the university's magtape drive being out of spec so only that drive could read the tape.
There wasn't any value to it, but I wish I still had it.
Yes, indeed. This was so common that in studios that for playback of imported stuff on stereo tape you'd switch polarity on one output and then tweak the head for the null. This is more important for wider tapes. For multi-track (not stereo) tape this would be more difficult, picking the track with the highest frequencies on it was usually a good bet. Then you'd aim for maximum output.
We also had special head alignment tapes that would be used just prior to an important recording. Usually this was a single screw, one side of the head would be held fixed in place and the other could be adjusted up or down against a spring, the head would tilt relative to the tape running direction. On very fancy multi track recorders you could also lift the whole head up and down so the (very small) heads would match up with the tracks.
Is there some reason that the data is not written to a section of tape using one head, and then read off of that same tape 1 inch (hundreds of milliseconds? tens?) later using a separate head for verification, before it's tucked back into the spool?
> The read-write head has 32 read-write elements. Each element contains a write component sandwiched between two read components – one for reading, the other for read-after-write checking. The overall read-write head reads or writes 32 adjacent tracks at once. This group of 32 adjacent tracks running the length of the tape is called a wrap. The write tracks are partially overlapped, leaving a clearly readable read track, narrower and in the middle of its write track. This track overlapping is called shingling after shingled roof tiles, and is a way of cramming more write tracks into a data band to increase the tape’s capacity.
> The tape drives use a strong error correction algorithm that makes data recovery possible when lost data is within one track. Also, the head assembly includes separate read heads positioned just "behind" each write head, to verify the data just written without making a separate verification pass. This allows the drive to write a second copy of any data that fails the verify without the help of the host system.
I think the issue arises if the tape is out of alignment - ie. too high or too low. Then it writes in the wrong spot, reads in the wrong spot, so you think all is good...
But when you try to read it back with a different, correctly aligned machine, it is unreadable.
The professional video tape decks had separate read/write heads. The digital formats could do a pre-read by having the playback head in front of the record head. The signal could then be routed through whatever equipment in the studio to do whatever one needed and then recorded back to itself with the updated video. This was only allowed on a source with a CLONE sticker to guarantee the ORIGINAL was not being updated this way.
Been a long time since I've dealt with audio gear, but I'm pretty sure they also had separate playback/record heads in the tape path as well.
I used to do a ton of data-retrievals during litigation - pulling Iron Mountain LTO tapes back by the container (24 per container) - and restoring email and other data for the lawyers to comb through. We had a smallish 48-tape library, and this would have been 2003 - so old school LTO-1 I'm guessing (?).
Data was only 3-4 years old at most, but there was never a recovery failure - at least one that was visible during the restore process.
Oh - important data-point - we never re-used tapes. It was one-and-done-and-off-to-iron-mountain. (That eventually would have changed - but we were only 4 years into our tape-rotation when I left)
Anecdote of 1, but it was over probably 10+ restores of data. Not certain how many tapes were actually read - we would just jam them in 24 at a time and let the index figure out which ones to restore from.
The caution to validate backups by doing constant restore + validation is a good one though. Nobody cares about backups, it's the restores they want.
* Don't rely on vendor-specific compression and/or encryption which may vary between different tape drives.
* If you do use encryption, try to ensure you have a way to recover from corruption in the middle of the bitstream (see parent's point on two copies, or CTR/GCM mode, or FEC/erasure coding).
* Index the contents (file names, sizes, at least), or use an archive format that puts the index at the start of the tape (note: tar does not).
I've never used tape drive hardware compression. It would not have done anything as the vast majority of data was large video assets. It was my understanding that using the built-in hardware compression was universal between devices.
The big cloud providers use something like erasure coding across a collection of tapes (or hard drives) to preserve information. If you've ever downloaded something from usenet and restored it with quickpar, this is basically the same idea. Every physical cartridge gets a chunk of the file. Depending on parity settings you would have to lose many tapes distributed across an arbitrary # of locations to actually lose any data.
There's a theory that Glacier is just a different QoS class on hard drives that also handle regular S3, because the capacity to throughput ratio got too bad to use the whole capacity for S3 and still meet throughput needs.
FAcebook had a tape team, however it never really went into prod, as far as I'm aware it was shut down in 2024.
I think the issue with tape at scale is that there is a lot of mechanical stuff that needs to be overcome. You will need a custom robot to allow for the management of hundreds of thousands of slots. I think also you'll need a custom tool for identifying the individual tapes (LTO does have a few ways of ID-ing the individual tapes)
Then you'll need make sure that your data stream don't mix. that seems easy, but you are optimising for filling tapes, and minimum retrievals per object. One advantage is that ibjecst should be encrypted before they even enter the tape storage system so the risk oulling the wrong file becomes one of correctly decrpyting, rather than fine management of file paths.
Note that the hardware the big cloud providers use for tape archives is very different from the devices the GP poster appears to have used, and usually have robust verification systems.
In addition the need for cloud providers to comply with userdata erasure means they have to frequently test their ability to read and write their backups - which should increase your confidence in their correctness. Yes problems can and do still happen, but IME far less often than the more traditional tape setups GP describes.
Erasure can be done much more efficiently with crypto erasure, where you encrypt the user data and just delete the key when users request deletion. This way you don't need to spend expensive tape vault / library time rewriting a tape.
AFAIK, symmetric encryption (which is what you'd use for crypto erasure) is not thought to be vulnerable to quantum computers, and it would be a huge mathematical breakthrough if it turned out that currently-known symmetric crypto algorithms were actually weak.
Anecdotally, I worked in a large (American) state university computing center with many VAXen and an Amdahl mainframe for a few years. Graveyard shift, when all the batch jobs ran. Good times. I'm not sure where our experiences with this technology diverge, but I almost never ran into problems with tapes, and we had a library with rows and rows and rows. Tens of thousands of tapes. Some tapes hadn't left their drives in years. Others sat on their shelves for decades. It was extremely rare for us to encounter an error with the tape, the drive, or something in between. To be fair, I've always heard about problems with tapes, just never ran into them in an environment saturated with them.
Companies should only ever use tape with a good robotic tape library. LTO tape drives automatically read every single write to verify the data immediately.
Symantec Backup Exec was genuinely one of the worst pieces of software I have ever had to use. Later in life I got to use the IBM Storage Archive setup with much larger tape libraries which still sucked but my god the level of competency difference between the two applications was insane.
If you want to take advantage of tape's low cost per TB for your own data backups, without the big capex of tape hardware, I heartily recommend rustic's cold storage support for Amazon S3 Glacier Deep Archive. It ends up being $1/TB/month, as long as you plan to not restore from it. (Use it as a secondary backup, and plan to restore from your primary.) I have been using rustic+GDA for about a year now, and I contributed some enhancements to this software ecosystem to make this path smooth for others. With this solution, I am no longer tempted to buy my own tape hardware!
You test the tape before sending it to Iron Mountain, and you just have to trust that Iron Mountain isn't degaussing them. You retrieve one tape from them, as a test.
Yeah, it's nice for small amounts, but $60 per TB per 5 years is more than hard drives plus parity, even with recent price increases. If you're anywhere near the sweet spot for tapes they'll be significantly cheaper.
Are there any online cloud storage services that cost less than $1 per TB per month?
I have several thousand video files totaling several TB that I store as a hobby.
This is purely for backup purposes, so I almost never need to retrieve the files. That said, I'd hesitate to use a service that charges something like $100 to retrieve a 1 GB file.
My understanding is that with AWS S3, you can retrieve up to 1 TB of data almost for free by using CloudFront.
I'm currently storing my files in the us-east-1 region.
Big tech is trying their best to label generated stuff as such to exclude it from the dataset, but unless the cat is put back in the bag there is no more free reliably human generated training data in what is increasingly a sea of slop. The tech could have been rolled out responsibly, but they shot themselves in the foot with their own greed and impatience.
One thing worth noting is that the tape cartridges themselves aren't necessarily the biggest expense. Standalone LTO-10 drives are currently listed at around $25,000.
> One thing worth noting is that the tape cartridges themselves aren't necessarily the biggest expense. Standalone LTO-10 drives are currently listed at around $25,000.
The drives are a fairly fixed CapEx: once they're purchased they don't cost must/anything to run. You may replace them every few years with newer tech for higher capacities of course.
Libraries/robotics last even longer, and can stick around for a decade or more once purchased.
when I was dealing with tape, LTO-6 drives were really really expensive, but LTO-5 wasn't sure you got an extra 1/3rd of space per tape, and some speed.
Right now, the list price for LTO8 is about 2/3rds the list price of LTO9. In single units, LTO9 drives are around £8k. As soon as you buy 25-40 of them and a robot, the per unit cost drops dramatically
When I was dealing with tape the main problem was that the drives were very expensive, and they never lasted as long as their expense would have you hope. They have a complex exposed mechanism full of fairly fragile tiny plastic parts.
And for long term archive storage, the main problem is not the media, it is how are you going to read the media. With the drives being considerably more fragile that the tapes. A good plan is to store an entire spare restoration kit with the media. With the understanding it probably will not work when it needs to. So store the really critical stuff printed out so the mark one eyeball can parse it.
> LTO-10 30TB: $259–$300 per cartridge (~$8.63–$10/TB) LTO-10 40TB: $491–$555 per cartridge (~$12.28–$13.88/TB) LTO-9 18TB: $92–$129 per cartridge (~$5.14–$7.17/TB)
Just a note: do NOT buy LTO-10 right now. The capacity increase is not worth the price bump, and the drives are still very new. There are still no half-height drives for tape libraries available, for example.
You don't need a $25,000 LTO-10 drive, $100,000 library, and $300 tapes. The drives are frequently backwards compatible multiple generations, so your archives are safe. You can get LTO-7, LTO-8, and LTO-9 libraries for cheap (right now there are lots of LTO-7 standalone drives for under $1000 on ebay), as they've aged out of enterprise leases.
You got me briefly excited, but the only LTO-7 drives on eBay under $1000 are listed as parts-only. The price jump from LTO-6 to LTO-7 has remained steep for years; either industry is holding on to them, or they're being resold or released successfully.
You're right, I didn't catch the (For Parts Only) labels when you click into the auctions. It looks like there are some working LTO-7 drive sleds that can be adapted into storage vaults for under $1000, though you're in for another $1000-$2500 for the vault.
On top of the price, I think most of the LTO drives have minimum speeds that need to be sustained in order to avoid drive starts and stops or shoeshining where the heads repeatedly go over the same section of tape waiting for data. That'll slow down the backup or prematurely wear out your data.
And the minimum speeds for something like LTO-7 is 100MB/s so you do need to work a bit to make sure your LTO drive will get fed properly (e.g. cache and consolidate small files so that you're doing sequential reads during the backup).
That's got worse recently. LTO used to be guaranteed compatible two generations back; an LTO-7 drive can read LTO-6 and LTO-5 tape. With newer standards, they've cut that to one generation. Your point stands, but folks should be aware the treadmill has been sped up in the past few years.
Ouch, only one generation now? Back in the day, we started with a second-hand LTO-3 tape library, then upgraded to an LTO-5 when that died. I remember learning about this stuff (we were only an SME, not an enterprise, so didn't have any knowledge about these things), and realising we'd have to be careful with our upgrade plan.
Yes, drives are very expensive and that's why there is a business for renting them. Even Amazon Snowball offered that [1]. There are other rental services such as [2] but many of them are not LTO-10 yet (don't know if it really matters right now).
This is not very realistic. Recordable and rewritable Blu-ray Discs are a much more consumer/prosumer-friendly backup solution, and yet, due to low demand, the market is slowly dying. The market size for prosumer tape drives would probably be even several times smaller.
I used to do Blu-ray backups, if you keep the discs dry and cool, they are durable. And until the last few years, pretty ubiquitous. In addition to spreading the data across many discs, blu-ray is slow to write and recover the data. Probably faster if it’s a couple precision files but much slower if your moving a lot of data. Ltfs on lto isn’t bad at all for backups
It is not that small considering how well NAS is doing in the pro / consumer market.
The problem is BluRay capacity is too low. Had it been 1TB per Disc it might stand a chance. But using 40 disc just isn't something prosumer is willing to stomach.
Not to mention both Tape and Optical Disc aren't that durable. I just read Walterbright, the author of D programming language had to throw away his code because of inadequate backup solution at the time. We dont need to store and save every video, but there are things that is worth saving. And I wish we have better solutions.
I've never thought of (re)writable optical media for backups as having a particularly good user experience. I'm not saying tape is good or better but blu-ray is a lot of faffing about. In theory, the much larger capacity that tape can provide should reduce the faff factor, but the devil is in the details.
Yes, DVD and Blu Ray suffered a lot from varying degrees of quality in the media. Low quality chemicals led to huge error rates and unusable media, even after it was written.
About 20 years ago, we used to keep track of this stuff. Even batches from normally excellent brands would be bad. We considered this short term, volatile storage.
At the end, even the best brands outsourced and we quit using them. Now, M-Disk from Verbatim is decent, although rarely used.
All the remaining blu-rays are actually pretty good. The cheap ones have died off, and everything that's left is made by one of a few manufacturers with non-organic dye and at least passable quality that I would expect to last 20+ years in proper conditions. The good ones I would expect to last longer.
I was at Microcenter looking for rewritable Blu-ray drives and discs. Almost bought some, but did the math and realized overpriced external SSDs still cost less per GB.
While this isn't what you asked, there are plenty of used older generation LTO drives to be had for cheap on ebay. A few years back I got an HP LTO-5 drive for $85.
Plus, having your backups on used hardware _instills_ the importance of proper backup hygiene like nothing else!
I think HDDs (the spinning magnetic ones) are cheap enough that nobody would bother to buy them (for most people's storage amounts/requirements). They last long (as an archival medium), you get random access to data etc.
It was hard to do this before the future of personal computing became uncertain. Now that normal people are being priced out of computing altogether, it's essentially become impossible.
because share of consumers needing to archive 50TB+ and access it very infrequently (coz it is PITA), is vanishingly small. Even if the price of drive was 1/4 it would still be vanishingly small.
Even if you're say Youtuber and archive your raw footage, you still want to have access to it be reasonably fast, so a NAS with a bunch of hard drives is better deal
If you don't have a lot of specifically video you want to move to cold storage, your prosumer needs have traditionally been met quite well by a couple external hard drives from best buy. A wedding photographer can probably archive 500 shoots onto a single hard drive. Or 5x that with very light compression. Easy to have 2-3 copies.
And if that drive dies, 500 shoots die with it. For proper archival, each project gets its own backup device not shared with other projects. At least for professional jobs where I got paid this is how I handle it. For my personal projects, my storage is a cluttered as my mind strewn across multiple storage devices/formats.
Sure, if you ignore my last line saying it's easy to have 2-3 copies.
> For proper archival, each project gets its own backup device not shared with other projects
"its own backup device" implies only 1 copy? That's so much worse than 3 copies on shared drives.
And who isn't using shared drives? Companies use shared drives. Amazon uses shared drives. And you could double it to 6 copies for ridiculous levels of reliability and it would still be under $10 per shoot.
Now you're twisting my intent. Commingling client content is something I avoid. I prefer the ability to just give a client a copy of a device rather than digging out content. Multiple copies of the same data is just expected to be understood at this stage of the discussion, and the implication that I'd only have one is just bad intent on your part.
I'm not twisting anything on purpose. That was only a question for one line and I wasn't sure.
And if that's twisting intent, then you twisted mine much harder with "500 shoots die with it", because I explicitly suggested multiple copies instead of just treating it as assumed. What's with this one-sided accusation?
And if you're buying three drives per client that's an extra $200 per photoshoot in this scenario if we're talking hard drives. It's a massive cost. And it would need to be hard drives, since the client certainly can't read tapes, probably can't read a bluray, and a USB stick wouldn't be reliable unplugged. Also the idea of handing over a whole drive depends on them having a full license to every file you're archiving. For a photoshoot it's just not worth it in comparison to copying their deliverables folder out of the shared vault.
Edit: I guess you could go with stacks and stacks of blurays and be ready to hand out a drive every time you hand a disc out.
Circa 1994 my parents purchased a PC that came with a QIC (https://en.wikipedia.org/wiki/Quarter-inch_cartridge) drive, which was less than a $250 addition (closer to ~$500 inflation adjusted). This was just before ZIP drives were announced, back when the standard backup was limited to 1.44MB drives. Wish there was something comparable in the market today.
It's cheaper for consumers to have a drive bundled with the data storage, such as a hard drive.
Tape drives will always be needed for those who need fewer IOPS but more capacity. After all, tape drives allow you to build tape libraries with custom capacity vs drives (aka IOPS and bandwidth).
Consumers are more predictable and want IOPS more than capacity. So SSDs already are better, let alone HDDs.
Done right, cloud backup actually is "the right thing".
There are problems with cloud backups, but they are solvable. One is confidentiality/privacy, that's what encryption is for. The other is that you can lose your data because the could provider doesn't like you, it is a failure mode, in the same way that your tapes can burn. Use redundancy, like with any unreliable storage solution.
With an onsite NAS, local copies and cloud backup, you meet the "3 2 1" rule (3 copies, 2 storage types, 1 offsite) without the hassle of dealing with tapes, and it will probably be cheaper too, unless you are particularly data-hungry.
Tape drives are really picky compared to HDDs: they don't like it when the environment is too hot, or too cold, or too humid, or too dry, or too dusty, or with too many vibrations, etc. They're really only meant for datacenters with controlled environments. A prosumer version would need additional engineering for environmental tolerance that the enterprise ones don't have, and end up costing more, not less, especially with the tiny markets involved.
It kinda blows my mind that every tape drive I've seen disassembled seems to make no attempt to filter the air.
As a nerd that formerly had a DDS4 drive it's tempting to get an older LTO drive, but sadly I can't expect it to survive for long in a house full of cat hair.
And it's kinda maddening because it can't be all that hard to have some filters.
I've supported QIC, DDS, VXA, DLT, and LTO drives since the late 90s. The common factor for a good experience has been stable environmentals for the drive. QIC and VXA were probably the most forgiving, with DDS being the worst. I had some bad experiences with DLT, too, but nowhere as bad as DDS. Drives that were over-temped or in dusty environments were the worst, regardless of technology.
Depends on level of "prosumer". If "work pays for it", then a $5000 Thunderbolt connected LTO-7 is on the table. These have been lurking around the Mac space for a few years now for video folks to dump footage to LTFS.
The cost of tape is a big deal for for some bigger science experiments.
They often cost billions to build and they take loads of data, but most of it ends up heavily distilled by the time it goes to the final analysis pipeline, so keeping the raw data on tape is pretty common.
There was a time when tape was more specialized and we were worried about loosing our limited suppliers. The explosion in the ML that powers search and more recently the AI boom makes the technology look a lot more stable these days, which is a relief for us.
Tape is a fascinating technology, I recall reading the account of the SONY cofounder https://en.wikipedia.org/wiki/Made_in_Japan:_Akio_Morita_and...
and the crazy things they were trying to make their own tape media because they had no supply because of the war. They were ultimately successful in creating a compact tape machine but they had to build up their expertise in tape media from literally nothing.
"Fujifilm highlights the ability to air-gap tape cartridges as a last defense for user data: “Given the sophistication of cyberattacks, magnetic tape is attracting increasing attention as a means of protecting data from threats such as ransomware.”"
I mean, it's plausible, yet I find it hard to believe this solely drives demand 5 times. Because if it does then what, companies were not using cold storage backups before?
> I mean, it's plausible, yet I find it hard to believe this solely drives demand 5 times. Because if it does then what, companies were not using cold storage backups before?
As someone in HPC: people never delete stuff. It can stick around forever, and so if more bits are generated, there are more bits that people don't want to delete "just in case". Depending on the organization, quota enforcement can be tighter or looser.
HSM is fairly popular in the HPC space for this reason:
Your newest/hottest bits are on NVMe, anything older than (e.g.) 3 months gets move to SSD, anything older than that to spinning rust HDDs, and then to tape.
There are several file systems where this happens in the background: the file still shows up at /path/for/stuff/foo.txt but the contents are shuffled about behind the scenes.
To back this up, even if the dataset is "live" unless someone has specifically made it read only, it will change. Those changes need to be stored and versioned over time.
Filesystems vary, a lot have COW, but that doesn't mean that you can export those changes as neat changesets for fast recovery. Sometimes rebuilding the filesystem from changes is more time-consuming than pointing to an older copy thats on the nearline.
Where I worked, we had a jumping off point that made the master folder arrangement, which then pointed to different folders on different servers. It meant that broken servers only affected those folders. It also meant the largest file you could have was ~120tb.
Each server was dumped to the nearline, which was a lustre cluster. The lustre cluster then drained into a tape library. This meant that we could keep about 1.5 PB "fast" with daily backups rotating into weekly drive dumps. Then when a project was finished we pruned from the nearline and the extra copies in the library.
It's a total loss only for people that were too cheap to pay for multi-regional data storage. Most people balk at monthly S3 fees, so willingly paying double the cost is a non-starter for those led by the bean counters. Disaster recovery is a known thing that should not be blamed on the loss of a single data center in a stupid war. AWS even has automation of duplicating selected data to another region.
At least they have a process. That gives you a chance. Often the process itself is good - you didn't test it but the vendors all did and so the data is there - it might be hard to restore, but it can be done by experts after the fact. The drill means you can recover in a "short time" instead of a "long time" (short time can range from minutes to days, while long time days to years)
I've ran drills with companies which concluded that the incremental backups couldn't be restored due to some data corruption issue (no readback test was ever done).
Luckily the company also did a full backup every 6 months, so a disaster would have only thrown them back 6 months at max! /s ;)
But yes, some issues were even more simple: I had a case where performing a restore required twice the storage size of the uncompressed data, and the company actually didn't have the infrastructure ready to do that in case of a disaster...
I'll bet that if it really was important you could pay someone (LLMs make this easier) to dig through the data byte by byte and get everything back despite the corruption. It would take years, and be very expensive, but typically there is enough redundancy to figure it out. Often you only really need a small subset of the data and can live without the rest.
Don't get me wrong, drills are important! You should do them. However backups without a restore drill better than no backups at all. Even if you only get the last full backup, the data from the last 6 months will be the easiest to recreate from scratch if you must.
Alas, M-Disks are hugely more expensive than magnetic tapes, even when not taking into account their much greater volume.
The biggest M-Disks are of 100 GB and the cheapest that I see on Amazon (in 25-piece sets) are at about $9 without taxes and shipping. At that price, I can buy not 0.1 TB, but 2 TB of magnetic tapes, which also have a much lower volume, at 18 TB per tape for the older and cheaper LTO-9 cartridges. I can easily carry with me a suitcase with 360 TB of data (20 LTO-9 cartridges), or store it in a safe place. Carrying 3600 M-Disks or storing them in my home would be impossible.
M-Disks may be used only to archive some small amounts of more valuable data, e.g. documents or programs that you have written yourself, a few scanned legal documents, a few more important books, a small amount of music that you prefer, a few family photographs.
For bulk data, if you cannot afford the cost of tape drives, the only alternative is HDDs, but with HDDs you must take care to always use both duplicate HDDs and archive files with redundancy, which allow the regeneration of their content when any not too large part of the file becomes corrupted. You must also take care to migrate the data periodically to new HDDs, after a few years.
Due to their small size I have not tried to use M-Disks until now, so I do not know if my older Blu-ray recorder can write them. There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
I just put my image archive for the last 20 years on a single 100 GB BDXL disc, around 11 000 files. Granted these are mostly cell phone photos, not RAW or anything like that. I did the same with documents on an M-Disc DVD and email mbox-archive on another M-Disc DVD. All disks with copies of dvdisaster and a separate slimline usb-c drive. Chucked offsite in a fire proof safe. I think M-Discs or any HTL bluray a fine supplement to an otherwise comprehensive backup strategy.
> but with HDDs you must take care to always use both duplicate HDDs and archive files with redundancy
I'm a noob when it comes to tape backups. Is copying the same stuff to multiple tapes standard practice?
> There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
LTO drives used to read tapes from 2 generations back, then just 1. LTO 10 isn't backwards compatible. Future availability of compatible tape drives is also uncertain.
I do not know how standard such a practice is, but I am old enough to have witnessed enough disasters caused by data lost due to carelessness, so I have no intention for something like that to happen to me.
Therefore with HDDs I have always written the same data on 2 HDDs, and this saved me a few times, when after several years of storage both HDDs contained a few corrupted sectors, but not in the same places.
With tapes, I also always write at least 2, but frequently even 3, because their low cost makes this affordable. With tapes, the greatest danger is that if a drive malfunctions mechanically while a tape cartridge is inserted, it might damage that tape.
Having at least 2 copies of each tape or HDD also allows their storage in different locations.
Like I have said, I always combine this redundancy of having duplicate storage media with intra-archive-file redundancy, so I typically add 10% redundancy to each compressed and encrypted archive, so that for any corruption that affects less than 10% of the file the content is still recoverable.
I am still using the older LTO-7. While I might upgrade to LTO-8 or even LTO-9, an upgrade to LTO-10 is unlikely, unless great price reductions will happen in the future, which I doubt.
So I am very concerned by the lack of a good archival solution that is affordable by ordinary people and by small businesses, not only by very rich companies.
Microsoft has demonstrated an ideal archival medium (in glass), but for now the cost of their laser system is too great for commercialization. That appears to be mostly a chicken and egg problem, the lasers and the other optical components are too expensive because they currently are produced only in very small quantities, but they cannot be incorporated into a high-volume application, because they are too expensive.
I use on Linux the utilities "par2create" and "par2verify".
On Gentoo Linux, the name of the software package that provides them is "app-arch/par2cmdline".
"par2create" needs 2 parameters, the amount of redundancy to be added and the name of the input file (which should be a compressed then encrypted archive file).
It produces several additional files, with recovery information. My script creates then a directory, moves there all the files, including the original archive file, and then aggregates the directory with all its files into a single ".pax" archive file. The ".pax" file can then be stored on a couple of HDDs or of tape cartridges.
Before creating the initial archive file, I scan the directories to be archived and I gather all the metadata of the files (names, sizes, timestamps, content hashes), which I store in a database. The metadata can later be used to search for files, to find out from which archival device they should be restored, and also to detect duplicates (by using the content hashes) that may be omitted from later backups.
When retrieving such a file with redundancy, after extracting its component files the "par2verify" utility verifies the integrity and reconstructs corrupted data, if necessary.
For working with tape drives, I prefer to use a FreeBSD server, because the FreeBSD utilities for tape drives and in general for SAS/SCSI peripherals are more convenient and more complete than those of Linux (it is also possible to use a FreeBSD VM on a Linux computer, with PCIe pass-through for the SAS HBA card used to attach the tape drive).
It is inefficient to store small files on tapes, so I usually collect data to be archived on tapes until I reach an amount of about 50 GB, then I compress, encrypt, add redundancy, aggregate again into a single file and then save it into the tape cartridges.
Maybe. For the average person here though they will already have and want a blu-ray drive for "reasons", and so using it for backup is very cheap. LTO-5 might be cheap, but they still need to buy a drive that they won't use for anything else.
If you are a business of course tape is cheap. Even then, a few usb harddrives are cheap as well and much faster.
You have to take into account that there aren't (or very few usb) LTO-5 enclosures. While you can potentially find an LTO-5 drive >$200 these days, you also need an equipment with SAS or 8GB FC interface to be able to use it.
M-Disks are not cheaper than tape for bulk data storage. Once you're doing enough of this that you're not limited by the cost of the drives, it's about a factor of 25: $150/TB for M-disk vs $6/TB for tape.
Which LTO are you comparing with? Locally (Spain) it seems I get similar EUR/GB for BlueRay M-Disks as LTO-8, but LTO-7 and LTO-6 is way cheaper than BD. What exactly are you comparing here?
I can't really tell what happened to this. It seems like Microsoft wound down the "Project Silica" thing they had been talking about, their final post/research paper on it said "The research phase is now complete, and we are continuing to consider learnings from Project Silica" which kinda sounds like corp-speak for "it's over", and I haven't seen any commercial activity in this area since.
I remember when they announced you could soon put terrabytes of data on a Tesa strip with a laser. AFAIK, the technology went to some lab, was bought by a company, and disappeared.
There might be a huge potential for an AI which scrapes your (and my!) data and concludes what I can safely delete, saving the world tons of resources.
The millions of pictures I never got to sort, the multiple versions of a file that I wrote 2 years ago, etc.
But in the view of an AI-company that's probably a paradox. All this data is charged and could be used to learn more about me, it should never be deleted! Instead, some hole should be carved out of a mountain to store it, billing me every month!
Actually, just the other day I thought of restoring my blog and found that I hadn't backed it up completely but because I had stored the backups as just `dd` images of drives, Claude looked in the free space on the drive image and got me back my backup! https://wiki.roshangeorge.dev/w/Blog/2026-11-01/Embarrassmen...
You never know when you might want it. I store this stuff religiously now haha! Backup to R2!
> There might be a huge potential for an AI which scrapes your (and my!) data and concludes what I can safely delete, saving the world tons of resources.
Thinking...
User has 15 pictures of his dead mother, deleting 14.
Why use AI when duplicate finders have been a thing for decades?
For the pictures, I set up a batch script the one time that used exiftool to read all of my photo's exif data (for creation date), rename the file to that date / time, and put it in a year/month folder structure. Once that's normalized it was pretty straightforward to identify and delete duplicates (assuming the exif data on duplicates has the same date/time in it).
It's quite a neat thing that each AI lab is effectively archiving all human content ever as a curated dataset. Would absolutely love to work on that team.
it's fascinating how the general direction became to never delete data. All the attempts of taking a photo, all the downloads of bank-statements, all the useless emails.
At some point in the future the amount of data being stored indiscriminately must consume such a huge amount of resources and cost, that it becomes a hidden fixed cost within a country's GDP, without any tangible benefit...
Historians actually would love this. Currently many historians try to reconstruct how life was like centuries ago from accidentally preserved artifacts. Future historians would have no trouble understanding how our daily life is like.
don't we inflate this problem away? storage should get more abundant and cheaper and the old media looks comparitevly small compared to higher fidelity and higher definition formats that come later. so it doesn't matter if your gmail stores 20 years of useless emails.
That continues to be the assumption, yes. Disregarding all the waste being created in the process.
What's fascinating is the fact that the production of TAPE sees a 5x boost. That's the format with the slowest retrieval rate, only making sense for long-term storage, usually 1+ years.
So is there really 5x more valuable data now, or is it just not economic to actually judge the need for preserving data...?
Someone on social media must have made a post or video gushing about how awesome it is. I have noticed 5-10x+ increases on what is great but not worth that much equipment many times in the past.
BTW, it's often on good (caveat: for the cheap price) equipment while the prices on good (caveat: but expensive) equipment remains stable.
My favorite examples are Takumar lenses. Their fast 55mm lenses, in a vacuum, are just OK, but for the $10-20 they were being offered for because the company made so many of them? 10/10.
Fujifilm is pretty clearly trying to sell shovels in this market bubble. I haven't read the latest, but I would suspect their clientele has moved from cloud backups to AI training data, and they need to make a big show and dance of ramping production to join the bubble.
Ridiculous that there is no proper data archiving solution for the masses. Tapes... in 2026 at exorbitant prices. Blu ray taps out at 128GB, meh. The DVDs I wrote 20 years ago still perform well and each of them readable, so don't tell me there wouldn't be a market for this at proper prices.
Cloud backup is good until a new world war erupts since data centers are gonna be bombarded first.
Sigh, I recently got into stereo photography and saw there's a shortage of transparency film. I suppose the stock's not coming back up any time soon. We can't even take photos thanks to the clankers can we.
When you create an archive, do not skip the step of reading it back. I'm not talking about just having the system confirm it is readable. I mean actually restoring the data from the tape. I've had tapes fail to retrieve data off of the tape even though the writing app confirmed it was readable.
Drives can have issues reading data written by another drive. Drives can have issues reading a tape it wrote itself.
Why write one when you can write two at twice the time/cost? In all of my experience with tape, I've never written just one tape. In the early days of DVDs, the images were written to LTO to send to have the discs pressed. Two were always sent on the chance one wasn't readable and schedules were so tight that the loss of a couple of days to send a new tape would wreck delivery dates. Another facility wrote three tapes: one kept on site, one sent to IronMountain, one sent to the client directly. Plenty of occasions saw the local copy unreadable and the IronMountain copy was retrieved. These were managed by large robotic tape libraries so each tape was written in a different drive at the same time so it was 3x cost while only 1x time.
Use the drive cleaning cassette as recommended. Your dirty heads can cause the data to not be written/read correctly depending on which head is dirty. If you're having issues reading, clean the heads before giving up on the tape. If you're having issues reading a tape just written, do not think someone else will be able to read it.
Ensure whoever might be using a tape you've written understands what was used to write the tape. Not all archivers are compatible. If you use the tar command to write to tape, people using a GUI app won't be able to read it.
Retention decreases exponentially with temperature, but the flip side is that it also increases exponentially when refrigeration is added. A conservative estimate is 30 years at -20 Celsius. Another added benefit of flash is that it's now much denser than tape (~10x) and doesn't need to be moved - it can be powered down on the storage rack (and powered up for refresh every 6 months or so). Unfortunately, it's still ludicrously expensive (and even more at the higher densities).
Perhaps one of you is living in a much more humid and/or hot environment that causes issues with the writing process.
[0] https://en.wikipedia.org/wiki/9-track_tape
Debugging this was really fun, especially due to the randomness factor. :)
Understanding why this is as important as it is requires some visualization of the bit densities of modern tape drives, the capacity is enormous, the tape is ridiculously thin and the bits are small enough that it doesn't take much to make a part of a tape unreadable beyond what the error recovery logic can deal with. And that's before we get into longer term effects such as layer-to-layer copy and demagnetization.
I suspect that the number of problems you encounter from tapes scales superlinearly because they physically move. This leads to a lot of unpredictable failure modes. HDD racks also fail a lot, but in more predictable ways because all the data is managed in software.
We use LTO, and I think I saw ONE failed tape in 15 years and it failed during write. We do restores regularly on a pool of tapes that is rotated so each tape is touched at least every few months and it. Far lower failure rate than hard drives
Drives can be a bit temperamental when they got older but it's all generally reporter before actual failure
> Use the drive cleaning cassette as recommended. Your dirty heads can cause the data to not be written/read correctly depending on which head is dirty. If you're having issues reading, clean the heads before giving up on the tape. If you're having issues reading a tape just written, do not think someone else will be able to read it.
Every tape library we used that was automatic, the drives asked for tape and the tape library just looked up in which slot it is.
This is an interesting point - these days, tape is in the very top end of the market, both when you have too much data to fit on spinning metal (it's twice as dense in volume), and have regulators making sure you have off-site offline archives going back a number of years.
An interesting shift is happening, however - flash is now much denser per volume than tape, and can be re-written more times. It needs periodic refreshing, but, if you keep it at, say, -10C it should keep data for decades. As data volumes increase, flash might become an option for near and off-line storage. The on and near-line archives also would look a lot more like racks of storage servers, because the medium has a built-in drive and doesn't need a robot moving it to a set of mechanical devices. Downside is that it's thousands of times more expensive than tape.
The highest grade, reliability wise, goes in industrial grade products; the next following grade goes in name-brand SSDs; the lower ones in mass-market memory cards. One day an employee at a famous NAND manufacturer said "That's not even worth of a C grade", on that day the USB flash drive was invented...
Reality, especially modern one (as in early 2010s+n), is obviously more complicated, all sorts of consumer products are made with all sorts of chips, with name brands and even specific models/price class being only loosely indicative of quality, specifications (such as density: SLC/MLC/TLC/QLC/PLC - how many bits crammed into each capacitor) not always available and in any case not contractual with respect to the model, counterfeiting and/or Flash Fraud (inflated capacity with address-overflow overwriting) still ubiquitous in certain segments, ...
eMMCs are indeed generally a tier above the common microSD in reliability (compensating for soldered vs replaceable?) and in performance (how many UHS-II cards and readers have you seen in person? Most eMMCs are designed for, and used in, 8-bit mode and are usually fast enough to benefit from it), never mind that your phone might be using a newer and faster technology like UFS for its internal storage!
That said, your specific example is unlikely to be coincidence; you're suspecting the contacts (which indeed I find to often go bad on the modern microSD+SIM tray designs), so do the cards work in an USB microSD reader? (Avoid micro/miniSD to SD adapters, their thickness-constrained design also loosens and goes bad at an appalling rate!)
I worked in VFX and had the joy of looking after a few different tape systems.
Ranging from a single drive HP MSL4048 tape "robot" right up to various multi-rack monsters with 25 LTO6 drives.
First, you need good software, that is a critical thing. If you are making your own around tar, uts possible, but you need to test it. There are(or were) some opensource robot drivers, but they are somewhat painful to use (although LLMs are your friend here)
With big robots, the pain points are: mechanical.
THe systems I used had CCTV cameras to see what was going on. When it worked, the 25 drive monster could saturate 30gigs a second. THe issues we had were making sure the nearline had enough bandwidth to feed the library cache, and making sure the library cache was fast enough.
However the the robot used to get sad, it would drop tapes, randomly post tapes to the wrong slot, then get upset, or a drive would be sad and need vacuuming. The data was fine. But we regually pushed and pulled petabytes from it.
For anyone curious, tape libraries come in a variety of sizes, including down to 1U (5 tapes) and 2U (15 tapes), e.g.:
* https://buy.hpe.com/ca/en/storage/tape-storage/business-clas...
* https://buy.hpe.com/ca/en/storage/tape-storage/business-clas...
Here's what the inside of an old 2U library looks like:
* https://www.youtube.com/watch?v=UczFC_Ri6uI&t=53m28s
There wasn't any value to it, but I wish I still had it.
I'm also sorry I threw away my punch card decks.
And I'm sorry I sold my H11 for $25.
We also had special head alignment tapes that would be used just prior to an important recording. Usually this was a single screw, one side of the head would be held fixed in place and the other could be adjusted up or down against a spring, the head would tilt relative to the tape running direction. On very fancy multi track recorders you could also lift the whole head up and down so the (very small) heads would match up with the tracks.
* https://www.blocksandfiles.com/tape/2025/07/29/why-lto-10-fe...
> The tape drives use a strong error correction algorithm that makes data recovery possible when lost data is within one track. Also, the head assembly includes separate read heads positioned just "behind" each write head, to verify the data just written without making a separate verification pass. This allows the drive to write a second copy of any data that fails the verify without the help of the host system.
* https://en.wikipedia.org/wiki/Linear_Tape-Open#Logical_struc...
But when you try to read it back with a different, correctly aligned machine, it is unreadable.
Been a long time since I've dealt with audio gear, but I'm pretty sure they also had separate playback/record heads in the tape path as well.
Data was only 3-4 years old at most, but there was never a recovery failure - at least one that was visible during the restore process.
Oh - important data-point - we never re-used tapes. It was one-and-done-and-off-to-iron-mountain. (That eventually would have changed - but we were only 4 years into our tape-rotation when I left)
Anecdote of 1, but it was over probably 10+ restores of data. Not certain how many tapes were actually read - we would just jam them in 24 at a time and let the index figure out which ones to restore from.
The caution to validate backups by doing constant restore + validation is a good one though. Nobody cares about backups, it's the restores they want.
* Don't rely on vendor-specific compression and/or encryption which may vary between different tape drives.
* If you do use encryption, try to ensure you have a way to recover from corruption in the middle of the bitstream (see parent's point on two copies, or CTR/GCM mode, or FEC/erasure coding).
* Index the contents (file names, sizes, at least), or use an archive format that puts the index at the start of the tape (note: tar does not).
I think the issue with tape at scale is that there is a lot of mechanical stuff that needs to be overcome. You will need a custom robot to allow for the management of hundreds of thousands of slots. I think also you'll need a custom tool for identifying the individual tapes (LTO does have a few ways of ID-ing the individual tapes)
Then you'll need make sure that your data stream don't mix. that seems easy, but you are optimising for filling tapes, and minimum retrievals per object. One advantage is that ibjecst should be encrypted before they even enter the tape storage system so the risk oulling the wrong file becomes one of correctly decrpyting, rather than fine management of file paths.
In addition the need for cloud providers to comply with userdata erasure means they have to frequently test their ability to read and write their backups - which should increase your confidence in their correctness. Yes problems can and do still happen, but IME far less often than the more traditional tape setups GP describes.
https://news.ycombinator.com/newsguidelines.html
[1]: https://rustic.cli.rs/docs/commands/init/cold_storage.html
if you actually do the math, that's expensive
I have several thousand video files totaling several TB that I store as a hobby.
This is purely for backup purposes, so I almost never need to retrieve the files. That said, I'd hesitate to use a service that charges something like $100 to retrieve a 1 GB file.
My understanding is that with AWS S3, you can retrieve up to 1 TB of data almost for free by using CloudFront.
I'm currently storing my files in the us-east-1 region.
LTO-10 30TB: $259–$300 per cartridge (~$8.63–$10/TB) LTO-10 40TB: $491–$555 per cartridge (~$12.28–$13.88/TB) LTO-9 18TB: $92–$129 per cartridge (~$5.14–$7.17/TB)
https://tapebackup.org/lto-tape-price-trend/lto10-price
One thing worth noting is that the tape cartridges themselves aren't necessarily the biggest expense. Standalone LTO-10 drives are currently listed at around $25,000.
The drives are a fairly fixed CapEx: once they're purchased they don't cost must/anything to run. You may replace them every few years with newer tech for higher capacities of course.
Libraries/robotics last even longer, and can stick around for a decade or more once purchased.
But the fact that -ALL- LTO drive-mechanisms are build by IBM maybe also doesn't help the pricing...
when I was dealing with tape, LTO-6 drives were really really expensive, but LTO-5 wasn't sure you got an extra 1/3rd of space per tape, and some speed.
Right now, the list price for LTO8 is about 2/3rds the list price of LTO9. In single units, LTO9 drives are around £8k. As soon as you buy 25-40 of them and a robot, the per unit cost drops dramatically
And for long term archive storage, the main problem is not the media, it is how are you going to read the media. With the drives being considerably more fragile that the tapes. A good plan is to store an entire spare restoration kit with the media. With the understanding it probably will not work when it needs to. So store the really critical stuff printed out so the mark one eyeball can parse it.
Just a note: do NOT buy LTO-10 right now. The capacity increase is not worth the price bump, and the drives are still very new. There are still no half-height drives for tape libraries available, for example.
And the minimum speeds for something like LTO-7 is 100MB/s so you do need to work a bit to make sure your LTO drive will get fed properly (e.g. cache and consolidate small files so that you're doing sequential reads during the backup).
[1] https://docs.aws.amazon.com/snowball/latest/developer-guide/...
[2] https://mediastoragegroup.com/
The problem is BluRay capacity is too low. Had it been 1TB per Disc it might stand a chance. But using 40 disc just isn't something prosumer is willing to stomach.
Not to mention both Tape and Optical Disc aren't that durable. I just read Walterbright, the author of D programming language had to throw away his code because of inadequate backup solution at the time. We dont need to store and save every video, but there are things that is worth saving. And I wish we have better solutions.
About 20 years ago, we used to keep track of this stuff. Even batches from normally excellent brands would be bad. We considered this short term, volatile storage.
At the end, even the best brands outsourced and we quit using them. Now, M-Disk from Verbatim is decent, although rarely used.
Plus, having your backups on used hardware _instills_ the importance of proper backup hygiene like nothing else!
Even if you're say Youtuber and archive your raw footage, you still want to have access to it be reasonably fast, so a NAS with a bunch of hard drives is better deal
Sure, if you ignore my last line saying it's easy to have 2-3 copies.
> For proper archival, each project gets its own backup device not shared with other projects
"its own backup device" implies only 1 copy? That's so much worse than 3 copies on shared drives.
And who isn't using shared drives? Companies use shared drives. Amazon uses shared drives. And you could double it to 6 copies for ridiculous levels of reliability and it would still be under $10 per shoot.
And if that's twisting intent, then you twisted mine much harder with "500 shoots die with it", because I explicitly suggested multiple copies instead of just treating it as assumed. What's with this one-sided accusation?
And if you're buying three drives per client that's an extra $200 per photoshoot in this scenario if we're talking hard drives. It's a massive cost. And it would need to be hard drives, since the client certainly can't read tapes, probably can't read a bluray, and a USB stick wouldn't be reliable unplugged. Also the idea of handing over a whole drive depends on them having a full license to every file you're archiving. For a photoshoot it's just not worth it in comparison to copying their deliverables folder out of the shared vault.
Edit: I guess you could go with stacks and stacks of blurays and be ready to hand out a drive every time you hand a disc out.
It's cheaper for consumers to have a drive bundled with the data storage, such as a hard drive.
Tape drives will always be needed for those who need fewer IOPS but more capacity. After all, tape drives allow you to build tape libraries with custom capacity vs drives (aka IOPS and bandwidth).
Consumers are more predictable and want IOPS more than capacity. So SSDs already are better, let alone HDDs.
There are problems with cloud backups, but they are solvable. One is confidentiality/privacy, that's what encryption is for. The other is that you can lose your data because the could provider doesn't like you, it is a failure mode, in the same way that your tapes can burn. Use redundancy, like with any unreliable storage solution.
With an onsite NAS, local copies and cloud backup, you meet the "3 2 1" rule (3 copies, 2 storage types, 1 offsite) without the hassle of dealing with tapes, and it will probably be cheaper too, unless you are particularly data-hungry.
50GB? Is that a joke?
As a nerd that formerly had a DDS4 drive it's tempting to get an older LTO drive, but sadly I can't expect it to survive for long in a house full of cat hair.
And it's kinda maddening because it can't be all that hard to have some filters.
They're not only used in such controlled environments, but yes, they are pickier.
https://mlogic.com/products/mlogic-mtape-desktop-thunderbolt...
They often cost billions to build and they take loads of data, but most of it ends up heavily distilled by the time it goes to the final analysis pipeline, so keeping the raw data on tape is pretty common.
There was a time when tape was more specialized and we were worried about loosing our limited suppliers. The explosion in the ML that powers search and more recently the AI boom makes the technology look a lot more stable these days, which is a relief for us.
I mean, it's plausible, yet I find it hard to believe this solely drives demand 5 times. Because if it does then what, companies were not using cold storage backups before?
As someone in HPC: people never delete stuff. It can stick around forever, and so if more bits are generated, there are more bits that people don't want to delete "just in case". Depending on the organization, quota enforcement can be tighter or looser.
HSM is fairly popular in the HPC space for this reason:
* https://en.wikipedia.org/wiki/Hierarchical_storage_managemen...
Your newest/hottest bits are on NVMe, anything older than (e.g.) 3 months gets move to SSD, anything older than that to spinning rust HDDs, and then to tape.
There are several file systems where this happens in the background: the file still shows up at /path/for/stuff/foo.txt but the contents are shuffled about behind the scenes.
Filesystems vary, a lot have COW, but that doesn't mean that you can export those changes as neat changesets for fast recovery. Sometimes rebuilding the filesystem from changes is more time-consuming than pointing to an older copy thats on the nearline.
Where I worked, we had a jumping off point that made the master folder arrangement, which then pointed to different folders on different servers. It meant that broken servers only affected those folders. It also meant the largest file you could have was ~120tb.
Each server was dumped to the nearline, which was a lustre cluster. The lustre cluster then drained into a tape library. This meant that we could keep about 1.5 PB "fast" with daily backups rotating into weekly drive dumps. Then when a project was finished we pruned from the nearline and the extra copies in the library.
This is a catastrophic event that you can't mitigate except with cold storage offline.
That's a bad way to do risk mitigation.
What I observed in the rising cybersecurity attacks happening around me:
(i) There are organizations thinking that they have [complete] backups and being aware they don't once the cyberattack if effective.
(ii) Even in organizations with complete backups they don't have a [good] process to recover them, and the operations suffered a lot.
Corollary: backups are, in general, not battle tested.
Even within the same company. Every division has backup processes, only few actually try to recover the data once in a while...
I've ran drills with companies which concluded that the incremental backups couldn't be restored due to some data corruption issue (no readback test was ever done).
Luckily the company also did a full backup every 6 months, so a disaster would have only thrown them back 6 months at max! /s ;)
But yes, some issues were even more simple: I had a case where performing a restore required twice the storage size of the uncompressed data, and the company actually didn't have the infrastructure ready to do that in case of a disaster...
Don't get me wrong, drills are important! You should do them. However backups without a restore drill better than no backups at all. Even if you only get the last full backup, the data from the last 6 months will be the easiest to recreate from scratch if you must.
I'm suspecting the AI companies simply save everything (even if you said "no"), just in case they need it later for training.
Why? Because there is no moral anymore. They blatantly ignored copyright after all.
Consumer tape. Like 5tb or what...
Alas, M-Disks are hugely more expensive than magnetic tapes, even when not taking into account their much greater volume.
The biggest M-Disks are of 100 GB and the cheapest that I see on Amazon (in 25-piece sets) are at about $9 without taxes and shipping. At that price, I can buy not 0.1 TB, but 2 TB of magnetic tapes, which also have a much lower volume, at 18 TB per tape for the older and cheaper LTO-9 cartridges. I can easily carry with me a suitcase with 360 TB of data (20 LTO-9 cartridges), or store it in a safe place. Carrying 3600 M-Disks or storing them in my home would be impossible.
M-Disks may be used only to archive some small amounts of more valuable data, e.g. documents or programs that you have written yourself, a few scanned legal documents, a few more important books, a small amount of music that you prefer, a few family photographs.
For bulk data, if you cannot afford the cost of tape drives, the only alternative is HDDs, but with HDDs you must take care to always use both duplicate HDDs and archive files with redundancy, which allow the regeneration of their content when any not too large part of the file becomes corrupted. You must also take care to migrate the data periodically to new HDDs, after a few years.
Due to their small size I have not tried to use M-Disks until now, so I do not know if my older Blu-ray recorder can write them. There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
I'm a noob when it comes to tape backups. Is copying the same stuff to multiple tapes standard practice?
> There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
LTO drives used to read tapes from 2 generations back, then just 1. LTO 10 isn't backwards compatible. Future availability of compatible tape drives is also uncertain.
Therefore with HDDs I have always written the same data on 2 HDDs, and this saved me a few times, when after several years of storage both HDDs contained a few corrupted sectors, but not in the same places.
With tapes, I also always write at least 2, but frequently even 3, because their low cost makes this affordable. With tapes, the greatest danger is that if a drive malfunctions mechanically while a tape cartridge is inserted, it might damage that tape.
Having at least 2 copies of each tape or HDD also allows their storage in different locations.
Like I have said, I always combine this redundancy of having duplicate storage media with intra-archive-file redundancy, so I typically add 10% redundancy to each compressed and encrypted archive, so that for any corruption that affects less than 10% of the file the content is still recoverable.
I am still using the older LTO-7. While I might upgrade to LTO-8 or even LTO-9, an upgrade to LTO-10 is unlikely, unless great price reductions will happen in the future, which I doubt.
So I am very concerned by the lack of a good archival solution that is affordable by ordinary people and by small businesses, not only by very rich companies.
Microsoft has demonstrated an ideal archival medium (in glass), but for now the cost of their laser system is too great for commercialization. That appears to be mostly a chicken and egg problem, the lasers and the other optical components are too expensive because they currently are produced only in very small quantities, but they cannot be incorporated into a high-volume application, because they are too expensive.
What software do you use for this?
On Gentoo Linux, the name of the software package that provides them is "app-arch/par2cmdline".
"par2create" needs 2 parameters, the amount of redundancy to be added and the name of the input file (which should be a compressed then encrypted archive file).
It produces several additional files, with recovery information. My script creates then a directory, moves there all the files, including the original archive file, and then aggregates the directory with all its files into a single ".pax" archive file. The ".pax" file can then be stored on a couple of HDDs or of tape cartridges.
Before creating the initial archive file, I scan the directories to be archived and I gather all the metadata of the files (names, sizes, timestamps, content hashes), which I store in a database. The metadata can later be used to search for files, to find out from which archival device they should be restored, and also to detect duplicates (by using the content hashes) that may be omitted from later backups.
When retrieving such a file with redundancy, after extracting its component files the "par2verify" utility verifies the integrity and reconstructs corrupted data, if necessary.
For working with tape drives, I prefer to use a FreeBSD server, because the FreeBSD utilities for tape drives and in general for SAS/SCSI peripherals are more convenient and more complete than those of Linux (it is also possible to use a FreeBSD VM on a Linux computer, with PCIe pass-through for the SAS HBA card used to attach the tape drive).
It is inefficient to store small files on tapes, so I usually collect data to be archived on tapes until I reach an amount of about 50 GB, then I compress, encrypt, add redundancy, aggregate again into a single file and then save it into the tape cartridges.
If you are a business of course tape is cheap. Even then, a few usb harddrives are cheap as well and much faster.
The millions of pictures I never got to sort, the multiple versions of a file that I wrote 2 years ago, etc.
But in the view of an AI-company that's probably a paradox. All this data is charged and could be used to learn more about me, it should never be deleted! Instead, some hole should be carved out of a mountain to store it, billing me every month!
You never know when you might want it. I store this stuff religiously now haha! Backup to R2!
Thinking...
User has 15 pictures of his dead mother, deleting 14.
Saved 20 MB! :)
For the pictures, I set up a batch script the one time that used exiftool to read all of my photo's exif data (for creation date), rename the file to that date / time, and put it in a year/month folder structure. Once that's normalized it was pretty straightforward to identify and delete duplicates (assuming the exif data on duplicates has the same date/time in it).
Duplicates are easy, that gets sorted quite fast. Picking 1-2 out of those 5 slightly different pictures of the same occasion is another task.
Shipping your drives to Sam Altman. If everybody did that, he could then deduplicate the pictures of the same thing that multiple people took.
As time goes by, people are sending more and more. It started with just prompts, then the local source code, then calendar, emails, local files...
https://blogs.intellique.com/tech/2022/01/27#TapeCLI
At some point in the future the amount of data being stored indiscriminately must consume such a huge amount of resources and cost, that it becomes a hidden fixed cost within a country's GDP, without any tangible benefit...
That continues to be the assumption, yes. Disregarding all the waste being created in the process.
What's fascinating is the fact that the production of TAPE sees a 5x boost. That's the format with the slowest retrieval rate, only making sense for long-term storage, usually 1+ years.
So is there really 5x more valuable data now, or is it just not economic to actually judge the need for preserving data...?
Maybe for you. Regards, the < insert your favourite 3 letter agency here > /s
BTW, it's often on good (caveat: for the cheap price) equipment while the prices on good (caveat: but expensive) equipment remains stable.
My favorite examples are Takumar lenses. Their fast 55mm lenses, in a vacuum, are just OK, but for the $10-20 they were being offered for because the company made so many of them? 10/10.
Cloud backup is good until a new world war erupts since data centers are gonna be bombarded first.
What is going on with cerabyte?
That's what they keep telling everyone else for years now...
That's where all my random pictures, duplicate bank statements and savegames will go. For the benefit of mankind! (and a small monthly fee)
Point.de
Point archival gateway.
Beautiful. I love it.