Not much, actually. Dual actuators only improve parallelism, not media transfer rate or latency. In practice, dual actuators don't even double external transfer rates due to occasional latency (which is unavoidable even for the best-case scenarios) and internal contention elsewhere in the drive. Even if you had four actuators working perfectly in concert, and weren't bottlenecked on the external interface speed (which you would be), a 4x improvement in transfer rate vs. a 8x increase in capacity would still mean a 2x increase in fill/empty time.
As with the shift from 5.25" to 3.5" to 2.5" and even 1.8" drives, the only way out of this bind is really to take advantage of the improved areal density to make drives that are the same capacity but smaller and pack more of those into the same volume or power/heat envelope. Drive manufacturers could help this along a little bit e.g. by sharing a motor and some environmental bits between what are otherwise completely separate drives (including separate external interfaces) within a single package, but mostly we'd all better get used to higher drive counts. Dual actuators - and this is far from the first time they've been tried BTW - are mostly a red herring.
Background: I worked on exactly these problems for the latter half of a thirty-year career, most relevantly at my last job working on an exabyte-scale storage system at a FAANG.
Also what market is there for big hard drives outside of FAANG and a few other places?
Retailers in my area (e.g. Best Buy) don't stock hard drives larger than 4 TB; I was going to tell somebody who lived in the Valley that he's lucky to be able to go to Fry's and then Fry's closed down.
Those same retailers stock both budget and quality SSD's up to 2TB in size. Most upgraders and the system builders are happy.
For the rest of us there is Amazon where the Seagate Exos "enterprise" drive costs half of what similar "consumer" drives cost, has a great reputation and does not seem hard to live with at home.
I would not take it for granted at all that a backup, RAID rebuild, restore, metadata scan or any full scan would work on such a disk if I hadn't tested it -- it is just that kind of technology. Synology is not crazy at all when they make you buy branded large drives to go in the enclosure.
It is common for retailers to stock low-spec parts, but that doesn't mean that there is no market for the higher-end stuff. Another example is routers: hypermarkets often only stock 10/100MB routers, and a gigabit router has to be ordered online from somewhere. But of course there are plenty of people with gigabit fiber connections who do a lot of torrenting or have a whole family using the connection who would benefit from ordering a gigabit router.
FWIW, I filled an 8T hard drive and then a 4T follow-up hard drive in just one summer of torrenting. Full-size Blu-ray rips are large, and the canon of great cinema is vast. Sure, it'll take me many months afterwards to watch everything that I have downloaded, but these large hard-drive sizes are well within the bounds of what a cinephile with a home theater building a collection would need, and they definitely just aren’t for large businesses. Of course, the hard drive discussed in the linked article is something else entirely.
> what market is there for big hard drives outside of FAANG and a few other places?
I wouldn't consider myself an expert on market numbers, but I'd say you generally shouldn't be using the larger drives. As others have said, disk is the new tape. Unless you have the kind of system where you might once have used tape - i.e. one with a substantial ice-cold-data component - larger disks are likely to be an ill fit. Even where they are a good choice, that's mostly going to be in a tiered architecture with flash etc. to suck all the "heat" out of the data going into them.
Hard drives are way cheaper if you need a lot of data. And not just cold backup.
It’s nice that 2.5” portable USB hard drives are cheap enough now (and USB ports providing enough power) that you don’t need the bulky power supply any longer. That means if you want to do any kind of bulk video storage, you can afford to do it. You can’t with SSD, which costs 10x as much per TB. A factor of 10 still matters to most people, ESPECIALLY if they’re not FAANG.
And I think relying on FAANG infrastructure is over-rated. Still a very good case for local storage. USB now provides enough throughput and power that it’s easier than ever to have a significant amount of local storage for cheap. And carry it with you if you travel.
I can get a used server on eBay with 96 TB of storage, usually in 12 x 8TB configs for around $2000 - $2500.
Flash drives? I'll spend 10 times that... hell, probably more. I love having my movies, music, television, etc. on my home server, but I don't love it at the level of 25 large. Now $2500? That's a lot more reasonable.
Sounds like you have a solution that works with today's medium-sized drives, so how is this even relevant to a discussion of 60-120TB drives? Do you think you'd be well served by putting that 96TB on one drive? I doubt it. If your active set (for any useful or relevant definition of "active") is X and the number of drives you need for IOPS or MB/s is Y, then the ceiling for how much you can effectively use per drive is X/Y. That's quite likely less than current drive capacities, and almost certainly within what can be achieved by current PMR technology. How would HAMR or BPM or any other more fundamentally new disk technology actually help you other than to store cold data on that "spare" capacity?
Yes? I mean, the sequential write speeds are fine for many video-heavy applications. You don’t want it in cold storage. Tape drives are not cheap. And neither is retrieving from tape terribly practical, either. There’s a large overhead for tape storage if you want it automatically accessible (ie a big tape jukebox). A big hard drive can store large volumes of data just a few milliseconds away, not minutes as you have to wind the tape to the right spot in a big tape jukebox... or hours or days if you have to ask someone to retrieve some tapes by hand. And can do hard drive storage for an order of magnitude lower cost than SSD.
Spinning disk is still really useful for video. Which is not shrinking any time soon.
That's a lot of time spent trashing tape, which I never suggested as a solution. The central question remains whether that single huge disk will provide the I/O rates that you need, and it won't. If video sizes increase and frames/second doesn't, then I/O need goes up so you'll need multiple drives even more. That means the sizes we already have, without fundamental shifts in technology, are even more likely to be sufficient. You'll just need more of them. What part of this math is escaping you?
No math is escaping me. Video is sequential I/O, which hard drives do fine at. Multiple drives is needed under many workloads, but not video.
If you want to do video stuff on a laptop you have room for maybe one extra hard drive. And if you are willing to have an external drive (which isn’t too bad), it’s not going to work to plug several hard drives in and expect to RAID them together.
Secondarily, there are only so many drives you can stuff in a workstation. A lot of compact ones only have a couple slots in them, and not everyone wants a RAID or JBOD controller with a bunch of ports on it just to do a video workflow. And if you have a video surveillance small server or appliance, you might only have a few slots in it (4 is fairly common).
So again, even one or two hard drives is fine for most uses. Not everyone is gonna put a 24 slot JBOD/RAID chassis in to just provide enough storage space for their video surveillance system or whatever.
"Video" is ambiguous use case. It would be fine for watching video on home server by single person or hoarding video files, but obviously not fine for video editing, video CDN edge, etc.
You asked, "What market is there for big hard drives outside of FAANG [etc.]?"
I answered.
The home prosumer market. And if anyone ever bothered to develop a multi-terabyte solution for "normal people", whereby they could easily have their DVD / Blu-ray / Compact Disc collection converted from disc to a media server, probably a lot more people.
100 TB hard drives will end up in consumer PCs by 2035, I'm certain of it, even if we won't really need it. 8K is just plain stupid for home use, I honestly don't know why its being pushed, since you'd need a 120" screen or more, but I'm sure there'll be new media types beyond 4K... 4K60 FPS for instance, or high-resolution VR movies. I could easily see a world where 4K60 FPS movies take up 200 GB space each, and high-res VR movies are that large or even larger.
> You asked, "What market is there for big hard drives outside of FAANG [etc.]?"
No, I didn't. PaulHoule did.
> 100 TB hard drives will end up in consumer PCs by 2035, I'm certain of it, even if we won't really need it.
I think you're confusing the need for more capacity with the need for more capacity within a single drive. This is the same distinction you'll find in the original RAID papers. With a big single drive ("SLED" in the papers) your performance per megabyte stored and your reliability per megabyte stored keep going down. At some point this will make your system unusable. I contend that we're already on the edge and these newer technologies that only increase capacity will push a lot of people over. Increasing video resolutions only make the problem worse, not better. How many 4K60 streams are you going to get over a single interface?
Now consider an alternative: same amount of storage, across multiple drives. In the same physical space, because that higher areal density can be used to make drives physically smaller just as well as it helps make them logically bigger. In a similar power/heat envelope, because smaller also means lighter. But with better performance across more heads and more external interfaces. And with better reliability because with multiple drives you can add some redundancy (and even if you don't at least a single failure will only cost you some of your data).
Why would you want a single big disk instead? Complexity? It's not actually that big a deal. Non-specialists build such systems every day. The only thing the drive vendors need to do is use those improvements in areal density to make drives physically smaller instead of logically larger. Stop making the capacity/bandwidth gap bigger. Let people build balanced systems that actually work, instead of on systems with a striking resemblance to those used for "virtual tape" cold storage in the bad old days.
> Increasing video resolutions only make the problem worse, not better. How many 4K60 streams are you going to get over a single interface?
How many does a prosumer need? One, maybe two. Maybe 5 in a NAS. A single drive today can already handle that in 8K.
When it comes to media files, we hit storage limits all the time but we're nowhere near the bandwidth limits of a hard drive. We're not on the edge when it comes to performance per megabyte. Program files and game files are miles away on one side, and videos and photos are miles away on the other side.
There's some point where increasing the density of hard drive platters is too slow for a prosumer media library, but I'm confident it's far out there, past a petabyte.
> we're nowhere near the bandwidth limits of a hard drive
According to my calculations, SATA-3 could support almost four 4K60 streams, but only if the data for those videos was very carefully interleaved (never happens). Also literally nothing else happening on the drive, no bad-block relocation, no bottlenecks elsewhere in the system, etc. Closer to reality, those files would be laid out on different parts of the disk and you'd lucky to get even two concurrent streams without seeks between them ruining your throughput. So yes, you are at the real-world bandwidth limits of a hard drive.
By contrast, two physically smaller drives adding up to the same capacity in the same space could reliably deliver one stream each, plus probably a third with data stored on both if you had decent buffering (because now you have enough MB/s headroom to buffer) to cover the seeks that remain. Just as when I was building storage systems for video professionals in 1994-95, if I had to deliver such a system and it had to work before I got paid I know which way I'd go.
You're talking about raw very-lightly-compressed footage I think? But cbozeman and I are talking about final products. The high end is around 10 megabytes per second per stream.
If you have raw 4K footage, and you're editing with it, and you need multiple streams and the ability to scrub around, I would simply say not to use any hard drive.
I would say that the vast majority of the consumer hard drive market is in the bracket where size matters much more than performance.
I have no idea what fraction of the server market.
But there's also an important thing to note about product families. You talk about using density improvements to make drives smaller, and then install more of them to keep performance up. I think that's reasonable, but I also think that one of the best ways to do that is to reduce the platter count and drive height. In that world, where the main product is thin drives, it takes only a small amount of engineering effort to keep making an XL model that has lower performance but is significantly cheaper per TB.
I said I'm not an expert on the HDD market, but I have seen some stuff so here's a bit of perspective. The hyperscalers are more than 50% of the market. They're each big enough that price distortion from their own buys is a real concern that they plan for. (Note BTW that these are often left out of analysts' charts, because leaving them in makes it harder to see patterns among the rest.) More than half of what's left is sold to businesses, with most of that going to companies big enough to build and run their own data centers (including supercomputer facilities). And then, finally, all consumer drive sales account for something less than 20% of the total.
Yes, size does matter a lot for those markets. If you want performance use flash. However, again, size per drive is a red herring. What matters is the capacity you can fit into a system, whether it's a laptop or a server. Having that much capacity present as a single volume through a single interface is simply not ideal either for performance (which might not be the same goal but still has a lower limit) or for reliability. That's all I've been saying. You're better off combining multiple lower-capacity drives, even if you can get by (at least for a while) with a single larger drive. Serious video folks and even gamers have known the advantages of dual drive RAID-0 or RAID-1 for years.
> where the main product is thin drives, it takes only a small amount of engineering effort
What you're now suggesting is no more than what I suggested nearly a day and several posts ago (look for "drive manufacturers could help"), which you and "others" took issue with. Yes, drive manufacturers can and should make those thin drives, and then sell multiples packed into a single enclosure like we already have today. The fact that it's multiple physical drives could be more or less transparent. The transparent version would be cheaper and offer the system designer more flexibility. The non-transparent version, akin to existing HW RAID or even multiple platters today, would be a bit easier to conceptualize for people not used to thinking of enclosures and spindles and platters and heads as separate things, but it would be a bit more expensive (controller plus memory as part of the package) and not necessarily better.
In short, using "drive" to mean both the package with connectors on the side and the piece(s) of oxide-coated metal inside it is sloppy, and leads to wrong conclusions. Once you realize that higher density creates more options than "every limit the same except for higher capacity" then it quickly becomes clear that 120TB on a single spindle isn't the best use of that technology.
> However, again, size per drive is a red herring. What matters is the capacity you can fit into a system, whether it's a laptop or a server.
I agree, but the argument I'm making is about cost per terabyte. I'm not inappropriately clinging to terabytes per drive.
> What you're now suggesting is no more than what I suggested nearly a day and several posts ago (look for "drive manufacturers could help"), which you and "others" took issue with.
I didn't take issue with smaller or multi-component drives existing, I just don't think they are necessary for all use cases. I was referring to what you said before on purpose, but disagreeing with the conclusion that "mostly we'd all better get used to higher drive counts". I got the impression you were treating it as a temporary transition measure.
Interesting. What do you think about Seagate's claim that dual actuator decreases their costs because it takes less time to test the drive? Is that the real reason for dual actuator?
As with the shift from 5.25" to 3.5" to 2.5" and even 1.8" drives, the only way out of this bind is really to take advantage of the improved areal density to make drives that are the same capacity but smaller and pack more of those into the same volume or power/heat envelope. Drive manufacturers could help this along a little bit e.g. by sharing a motor and some environmental bits between what are otherwise completely separate drives (including separate external interfaces) within a single package, but mostly we'd all better get used to higher drive counts. Dual actuators - and this is far from the first time they've been tried BTW - are mostly a red herring.
Background: I worked on exactly these problems for the latter half of a thirty-year career, most relevantly at my last job working on an exabyte-scale storage system at a FAANG.