Multiple Hard Drive Best Practices: Partitions, Libraries, Data and Mitigating Streaming Bottlenecks in an NVMe and SSD setup

[TimRideout]

Hello Wisest Ones;

I have just purchased a new computer and am in the setup phase. You know - the one where you re-think your entire setup and how to do it better from scratch

I am a producer for TV and film, so I often run fairly large setups that run the gamut from full orchestral templates, to hybrid production of (a few dozen) audio tracks plus the other "usual suspect" VSTs (Kontakt, Omnisphere, Spitfire etc).

I am currently musing on best practices for drive partitioning and data location for maximum speed and efficiency and would love your thoughts.

I will have three drives: 2X NVMe 4TB and 1X SSD 2TB. My first draft of partitioning is based on my current setup and looks like this:



Drive 1: NVMe 1 (3.7 TB)
OS : 600 GB
Users: 800 GB
Projects 1: 1000 GB
Projects 2: 1000 GB

Drive 2: NVMe 2 (3.7 TB)
Instrs 1: 1200 GB
Instrs 2: 1200 GB
Instrs 3: 800 GB
Loops: 500 GB

Drive 3: SSD 1 (1.8 TB)
Aux: 500 GB
Restore: 500 GB



The storage amounts are fine by me, and will give me room for growth over a projected 8 year horizon - but I would love to have your thoughts on the placement of specifically the INSTRUMENTS partitions (ie, the sample data for my VSTis) as the most bandwidth usage will come from there.

Let me explain further:

With all my previous systems (containing HDDs and SSDs), I have put each Instrument Drive (1, 2 and 3) on different PHYSICAL drives. This allows for more parallel bandwidth when streaming samples, and is an excellent optimization hack (FYI!).

However, this new system is a different story with the NVMe M.2 drive technology being (arguably) over 10 times faster than my older SATA 3 setup. So I am wondering:

1. Is the bandwidth of each NVMe drive so massive, that there is no way I could bottleneck the streaming, so I can just put all my Instruments / samples on the same drive
2. Or should I play it safe and still use the (arguably still un-antiquated) best practice of parallel streaming from different physical drives?

I will be running an AMD 7950x with 64Gb of DDR5 RAM. I have a robust backup system and am not concerned with RAID setups or drive life span.

Any thoughts appreciated - thanks!

Click to read more...

 

[gamma-ut]

Your bottleneck is probably going to be Kontakt.

 

[TimRideout]

Your bottleneck is probably going to be Kontakt.

Thanks Gamma - that kind of issue is very much in flux and depends on many factors (including the CPU and DAW multithreading).

So, I would contend that "software-based" factors are very much a different issue, and essentially beyond my control - unlike "hardware-based" physical drive and partitioning setups, and best practices relating specifically to those issues.

 

[T-LeffoH]

Hello Wisest Ones;

I have just purchased a new computer and am in the setup phase. You know - the one where you re-think your entire setup and how to do it better from scratch

I am a producer for TV and film, so I often run fairly large setups that run the gamut from full orchestral templates, to hybrid production of (a few dozen) audio tracks plus the other "usual suspect" VSTs (Kontakt, Omnisphere, Spitfire etc).

I am currently musing on best practices for drive partitioning and data location for maximum speed and efficiency and would love your thoughts.

I will have three drives: 2X NVMe 4TB and 1X SSD 2TB. My first draft of partitioning is based on my current setup and looks like this:



Drive 1: NVMe 1 (3.7 TB)
OS : 600 GB
Users: 800 GB
Projects 1: 1000 GB
Projects 2: 1000 GB

Drive 2: NVMe 2 (3.7 TB)
Instrs 1: 1200 GB
Instrs 2: 1200 GB
Instrs 3: 800 GB
Loops: 500 GB

Drive 3: SSD 1 (1.8 TB)
Aux: 500 GB
Restore: 500 GB



The storage amounts are fine by me, and will give me room for growth over a projected 8 year horizon - but I would love to have your thoughts on the placement of specifically the INSTRUMENTS partitions (ie, the sample data for my VSTis) as the most bandwidth usage will come from there.

Let me explain further:

With all my previous systems (containing HDDs and SSDs), I have put each Instrument Drive (1, 2 and 3) on different PHYSICAL drives. This allows for more parallel bandwidth when streaming samples, and is an excellent optimization hack (FYI!).

However, this new system is a different story with the NVMe M.2 drive technology being (arguably) over 10 times faster than my older SATA 3 setup. So I am wondering:

1. Is the bandwidth of each NVMe drive so massive, that there is no way I could bottleneck the streaming, so I can just put all my Instruments / samples on the same drive
2. Or should I play it safe and still use the (arguably still un-antiquated) best practice of parallel streaming from different physical drives?

I will be running an AMD 7950x with 64Gb of DDR5 RAM. I have a robust backup system and am not concerned with RAID setups or drive life span.

Any thoughts appreciated - thanks!

There's the possibility to bottleneck the components of any system, but best practices are usually there for a reason, not necessarily because technology outdates their application.

 

[easyrider]

A bizzare an unnecessary complex setup.

 

[EanS]

Have you created partitions within the Disk? Or are you willing to?

Rather than the reserved OS space that automatically provides on a partition, I wouldn't partition a single disk that holds samples or anything. You'll be indeed telling Windows to read, write cache on 8 Drive devices (for instance) simultaneusly if you are loading from different instrument groups, and a long etc...

I might be outdated on that, but 1 Windows/OS dedicated Drive, unbloated as much as you can with your Programs, and the rest, just single clean units as a whistle with no virtual managed crap that will make you rely on Windows stability and then start to question if it isn't your daw or kontakt etc... Try to picture that asshole windows indexing each one of these running in the back trying to find paths to not a disk, 4 partioned disks (so disable indexing )

For me partitioning is doom for failure. But I think I might be outdated on that, RAID I understand, it's all physical, no "stupid windows virtual machine let's see if we can be estable under any hardware config".

 

[José Herring]

Hello Wisest Ones;

I have just purchased a new computer and am in the setup phase. You know - the one where you re-think your entire setup and how to do it better from scratch

I am a producer for TV and film, so I often run fairly large setups that run the gamut from full orchestral templates, to hybrid production of (a few dozen) audio tracks plus the other "usual suspect" VSTs (Kontakt, Omnisphere, Spitfire etc).

I am currently musing on best practices for drive partitioning and data location for maximum speed and efficiency and would love your thoughts.

I will have three drives: 2X NVMe 4TB and 1X SSD 2TB. My first draft of partitioning is based on my current setup and looks like this:



Drive 1: NVMe 1 (3.7 TB)
OS : 600 GB
Users: 800 GB
Projects 1: 1000 GB
Projects 2: 1000 GB

Drive 2: NVMe 2 (3.7 TB)
Instrs 1: 1200 GB
Instrs 2: 1200 GB
Instrs 3: 800 GB
Loops: 500 GB

Drive 3: SSD 1 (1.8 TB)
Aux: 500 GB
Restore: 500 GB



The storage amounts are fine by me, and will give me room for growth over a projected 8 year horizon - but I would love to have your thoughts on the placement of specifically the INSTRUMENTS partitions (ie, the sample data for my VSTis) as the most bandwidth usage will come from there.

Let me explain further:

With all my previous systems (containing HDDs and SSDs), I have put each Instrument Drive (1, 2 and 3) on different PHYSICAL drives. This allows for more parallel bandwidth when streaming samples, and is an excellent optimization hack (FYI!).

However, this new system is a different story with the NVMe M.2 drive technology being (arguably) over 10 times faster than my older SATA 3 setup. So I am wondering:

1. Is the bandwidth of each NVMe drive so massive, that there is no way I could bottleneck the streaming, so I can just put all my Instruments / samples on the same drive
2. Or should I play it safe and still use the (arguably still un-antiquated) best practice of parallel streaming from different physical drives?

I will be running an AMD 7950x with 64Gb of DDR5 RAM. I have a robust backup system and am not concerned with RAID setups or drive life span.

Any thoughts appreciated - thanks!

I don't see any benefit in partitioning your harddrives this way.
I'd keep it real simple and just use the 1 2TB has your OS and project drive (perhaps you can do a partition for this) and the 2 4tb drives for sample storage.

There's no real need to do a separate partition for "loops" you can just make a folder for that.

NVMe drives are fast. I've not experience any bottle neck with the drives.

That being said it isn't going to hurt you to partition out the drives in the manner you described, I just can't see the benefit.

 

[Justin L. Franks]

Forget the partitions. Use the 2 TB drive for OS + programs + projects. Use the two 4 TB drives for everything else. Distribute your samples across both of those drives.

 

[rgames]

Yeah I haven't seen any performance benefit in partitioning/separating drives since SSDs came on the scene. That approach is a holdover from the HDD days. As stated above, it doesn't hurt. But I doubt it helps.

FYI the only difference I've seen between SATA SSD and NVMe SSD is load times - maybe 10% - 15% reduction in template load times for NVMe. But once everything is loaded I haven't seen any difference in performance for DAW work. I do see a difference in performance for video editing but it's not huge.

PCs are so powerful these days that it's pretty hard to find tweaks that affect real-life DAW performance. As long as you don't have crappy drivers you can set it up pretty much bone stock, or not, and it'll perform just fine.

rgames

 

[TimRideout]

Have you created partitions within the Disk? Or are you willing to?

Rather than the reserved OS space that automatically provides on a partition, I wouldn't partition a single disk that holds samples or anything. You'll be indeed telling Windows to read, write cache on 8 Drive devices (for instance) simultaneusly if you are loading from different instrument groups, and a long etc...

I might be outdated on that, but 1 Windows/OS dedicated Drive, unbloated as much as you can with your Programs, and the rest, just single clean units as a whistle with no virtual managed crap that will make you rely on Windows stability and then start to question if it isn't your daw or kontakt etc... Try to picture that asshole windows indexing each one of these running in the back trying to find paths to not a disk, 4 partioned disks (so disable indexing )

For me partitioning is doom for failure. But I think I might be outdated on that, RAID I understand, it's all physical, no "stupid windows virtual machine let's see if we can be estable under any hardware config".

Thanks EanS - I will definitely be partitioning, as two of the drives are 4TB.

The OS will be installed on its own partition indeed, with "unbloated" programs.

I'm not aware of Windows 11 caring about "virtual managed crap" or "single clean units." And I'mm pretty sure Windows will be just fine to "find paths to... 4 partitioned disks.,"

For me, partitioning is a better way to manage my different types of data (Libraries, Projects and Other) that are all different in terms of their needs (speed-wise, backup, workflow, network shares etc).

 

[easyrider]

Thanks EanS - I will definitely be partitioning, as two of the drives are 4TB.

The OS will be installed on its own partition indeed, with "unbloated" programs.

I'm not aware of Windows 11 caring about "virtual managed crap" or "single clean units." And I'mm pretty sure Windows will be just fine to "find paths to... 4 partitioned disks.,"

For me, partitioning is a better way to manage my different types of data (Libraries, Projects and Other) that are all different in terms of their needs (speed-wise, backup, workflow, network shares etc).

Why do you want to manage loads of drive letters?

 

[TimRideout]

I don't see any benefit in partitioning your harddrives this way.
I'd keep it real simple and just use the 1 2TB has your OS and project drive (perhaps you can do a partition for this) and the 2 4tb drives for sample storage.

There's no real need to do a separate partition for "loops" you can just make a folder for that.

NVMe drives are fast. I've not experience any bottle neck with the drives.

That being said it isn't going to hurt you to partition out the drives in the manner you described, I just can't see the benefit.

Thanks José - so you would recommend installing the Instruments / Samples on the fastest (NVMe) drives, as they would be doing the most streaming, correct?

Theoretically, the OS should reside in RAM as should the VSTs - hence no need for streaming there.

I think this is solid advice.

 

[easyrider]

Filling your system drive up with sample data is a bad idea not just from a performance perspective but from a data management one.

The default install for developers is C but this is not because its the best directory to use.

Storing large amounts of data on C drive makes backups more complex and slow, just like storing data on your desktop makes loading times for windows slow.

There literally is no reason to store samples or document data on the system drive in 2022.

I have my C drive for windows and programs and thats it.

Samples are stored on multiple SSD's pooled together into one drive volume labelled X:

I use Stablebit drive pool

This means...Kontakt, Native access, Spitfire player, Orchestral Tools sine juts focus on one drive letter X:

Windows drive becomes corrupt? I don't care... I can format and reinstall windows from an image file and point eveyting to X: in seconds.

I can add SSD's to the pool in seconds and all the sample players are unaffected and X: just gets expanded.

Here is an example Drivepool X:

As said samples on Mine are stored by Vendor

Drivepool X:

Spitfire
Native Instruments
Audio Imperia

etc...

My pool currently is a mixture of 2 x 2TB SSD and 1 X 4 TB for a drive pool of 8TB

Now all you have to do in the future in mange one drive letter X for everything. You just ignore the other drives completely. You can even hide them in windows so you dont see them.

So my a new spitfire library you down load it to Drivepool X: Spitfire

Point Native Access to Drivepool X: Native Access

Point OT Sine to Drivepool X: Sine

etc....

Makes things so easy to manage as all your sample are now on one single volume called Drivepool X:

Backup are super easy as you dont need to remember directories any more and makes backing up super simple.

I backup my Drivepool to a server also running Drivepool

Start running out of space on the pool? No problem just install new SSD in PC add the drive to the existing drivepool and voila the drivepool expands in seconds.

Want to remove a drive from the pool? No problem just remove it as long as there is enough free space in the pool to migrate the data.

PC dies one day? No problem you can access the data from any computer wether it has drivepool installed or not.

I'm down to 705GB free on my Pool and looking to add another 4TB SSD come amazon prime sales time. I dont have to do anything with my samples or data. I just add the 4TB SSD to the pool and the pool expands without me having to faff with moving files etc.

Hope this helps.

 

[easyrider]

To clarify:

Spitfire App Just looks for X: Spitfire

Sine just looks for X: Sine

Native Instruments just looks for X: Native Instruments

It complete negates the anything to do with missing samples...

Its a clean powerful way to manage your storage on windows.

And I really do wonder why people fanny about multiple drives on windows....

Say your windows machine dies and you have to reinstall windows...Native access can be back up running in seconds as no potential drive letters have changed.

D: has changed to E: for example and native instruments is throwing a tantrum cause it can't find samples.

Same with spitfire. You install the spitfire app point it to X: Spitfire and BOOM all your libraries are back online without fuss.

 

[TimRideout]

Yeah I haven't seen any performance benefit in partitioning/separating drives since SSDs came on the scene. That approach is a holdover from the HDD days. As stated above, it doesn't hurt. But I doubt it helps.

FYI the only difference I've seen between SATA SSD and NVMe SSD is load times - maybe 10% - 15% reduction in template load times for NVMe. But once everything is loaded I haven't seen any difference in performance for DAW work. I do see a difference in performance for video editing but it's not huge.

PCs are so powerful these days that it's pretty hard to find tweaks that affect real-life DAW performance. As long as you don't have crappy drivers you can set it up pretty much bone stock, or not, and it'll perform just fine.

rgames

Thanks RGames - partitioning my drives just helps me organize the data, really. Practically speaking, it also can help with network shares, backups, permissions, bandwidth etc.

At least, that's been my experience - but... it is* possible that I am over-thinking and micro-managing my partitions and data. I will meditate on this - thanks for the perspective

 

[TimRideout]

Filling your system drive up with sample data is a bad idea not just from a performance perspective but from a data management one.

The default install for developers is C but this is not because its the best directory to use.

Storing large amounts of data on C drive makes backups more complex and slow, just like storing data on your desktop makes loading times for windows slow.

There literally is no reason to store samples or document data on the system drive in 2022.

I have my C drive for windows and programs and thats it.

Samples are stored on multiple SSD's pooled together into one drive volume labelled X:

I use Stablebit drive pool

This means...Kontakt, Native access, Spitfire player, Orchestral Tools sine juts focus on one drive letter X:

Windows drive becomes corrupt? I don't care... I can format and reinstall windows from an image file and point eveyting to X: in seconds.

I can add SSD's to the pool in seconds and all the sample players are unaffected and X: just gets expanded.

Here is an example Drivepool X:

As said samples on Mine are stored by Vendor

Drivepool X:

Spitfire
Native Instruments
Audio Imperia

etc...

My pool currently is a mixture of 2 x 2TB SSD and 1 X 4 TB for a drive pool of 8TB

Now all you have to do in the future in mange one drive letter X for everything. You just ignore the other drives completely. You can even hide them in windows so you dont see them.

So my a new spitfire library you down load it to Drivepool X: Spitfire

Point Native Access to Drivepool X: Native Access

Point OT Sine to Drivepool X: Sine

etc....

Makes things so easy to manage as all your sample are now on one single volume called Drivepool X:

Backup are super easy as you dont need to remember directories any more and makes backing up super simple.

I backup my Drivepool to a server also running Drivepool

Start running out of space on the pool? No problem just install new SSD in PC add the drive to the existing drivepool and voila the drivepool expands in seconds.

Want to remove a drive from the pool? No problem just remove it as long as there is enough free space in the pool to migrate the data.

PC dies one day? No problem you can access the data from any computer wether it has drivepool installed or not.

I'm down to 705GB free on my Pool and looking to add another 4TB SSD come amazon prime sales time. I dont have to do anything with my samples or data. I just add the 4TB SSD to the pool and the pool expands without me having to faff with moving files etc.

Hope this helps.

Thanks EasyRider - great thoughts, and interesting to see your data management!

Regarding the System drive - I absolutely agree that the best practise is to have Windows on it's own drive or partition! Makes it easy to restore or revert. I store nothing else on the Windows partition (C for me) except Windows.

My "User files" I store on my USERS drive (D for me). This drive is synced to a NAS and across multiple workstations.

I love the idea of a Storage Pool (like in my NAS) - although, I don't relish the idea of using a third party solution for that. I believe Windows 11 has a built-in solution. My only issue would be the structure of teh data (ie, if one drive of the pool fails, do I lose all data in the pool).

My best practise to work around storage pools is try to anticipate my data needs well in advance. I haven't had any problems with my last system (7 years old) so I hope to continue with my luck there!

But I do* like the idea of simplifying. My only issue would be backups - but I suppose instead of Backup Jobs by DRIVE (partition), they could be by FOLDER...

 

[TimRideout]

To clarify:

Spitfire App Just looks for X: Spitfire

Sine just looks for X: Sine

Native Instruments just looks for X: Native Instruments

It complete negates the anything to do with missing samples...

Its a clean powerful way to manage your storage on windows.

And I really do wonder why people fanny about multiple drives on windows....

Say your windows machine dies and you have to reinstall windows...Native access can be back up running in seconds as no potential drive letters have changed.

D: has changed to E: for example and native instruments is throwing a tantrum cause it can't find samples.

Same with spitfire. You install the spitfire app point it to X: Spitfire and BOOM all your libraries are back online without fuss.

Again, very interesting. Perhaps I could look at my data in terms of its ROLE or USAGE, instead of looking at where it should LIVE, and splitting that up. Instead, I could use Storage Pools to consolidate drive letters, and have my best practises revolve around that idea - similar to my NAS.

How about:

C: Operating System
D: User files (by User)
L: Libraries (by Vendor)
X: Aux (Backup, Temp, Restore etc)

 

[easyrider]

Is don't relish the idea of using a third party solution for that. I believe Windows 11 has a built-in solution. My only issue would be the structure of teh data (ie, if one drive of the pool fails, do I lose all data in the pool).

Christopher (Drashna) They guy behind Stablebit Drivepool used to work for Microsoft as a windows programmer. Support and programming is exceptional.

Drive Pool is not Raid, the data is not spanned…you can remove a drive from the pool plug it into another computer not running drive pool and read the data just like any other disk.

if one of the drive fails within the pool you don’t lose all the data in the pool.

You would just lose the data on the drive that failed. Just like it would be if the drive was on its own drive path.

I have run drive pool for years…on my Main Daw machine , my media server and backup server all without issue.

The software is amazing and makes storage and backups a breeze.

 

[Markastellor]

Hi Tim, I used to partition everything about like you are planning to do. The problem I regularly ran into was partitions running out of room....even when I oh so carefully planned ahead and provided twice as much space as I thought I would ever need. At that point my carefully calculated plans would fall apart. You can adjust the partitions, of course, but what a hassle. This was especially true with sample libraries. As SSD's get bigger, the library producers create bigger libraries. I was looking at the Hans Zimmer Strings Library from Spitfire and it was something like 250 GB just for that one library.

I just bought a new 8TB NVME drive as well as some new SATA drives. I sort things out now with folders instead of partitions. When one grows more than expected and one grows less than expected, I still have room on the drive.

We all want all the speed we can get, but it's really not needed for sample playback. Computers today are designed for video production which puts WAY more of a strain than audio. SATA Drives can still work for audio. I usually put my operating system on a really sturdy NVME drive, and I put sample libraries that may strain my system on another. But you can save a lot of money just by putting some on SATA drives. I typically have 50 to 60 tracks of virtual instruments playing in a song and sometimes more than 100. I haven't choked the system yet and I have a really early i9 processor...(about 5 years old).

In theory, writing hurts the SSD more that reading. So if you put all your sample libraries on one SSD then just leave it alone it should last for a very long time...though there are always random failures so you should always back up everything. As a standard practice, I fill a harddrive about 75% full, back it up a couple times a year and just leave it alone. If it does fail I can replace and restore it easily.

 

[TimRideout]

Filling your system drive up with sample data is a bad idea not just from a performance perspective but from a data management one.

The default install for developers is C but this is not because its the best directory to use.

Storing large amounts of data on C drive makes backups more complex and slow, just like storing data on your desktop makes loading times for windows slow.

There literally is no reason to store samples or document data on the system drive in 2022.

I have my C drive for windows and programs and thats it.

Samples are stored on multiple SSD's pooled together into one drive volume labelled X:

I use Stablebit drive pool

This means...Kontakt, Native access, Spitfire player, Orchestral Tools sine juts focus on one drive letter X:

Windows drive becomes corrupt? I don't care... I can format and reinstall windows from an image file and point eveyting to X: in seconds.

I can add SSD's to the pool in seconds and all the sample players are unaffected and X: just gets expanded.

Here is an example Drivepool X:

As said samples on Mine are stored by Vendor

Drivepool X:

Spitfire
Native Instruments
Audio Imperia

etc...

My pool currently is a mixture of 2 x 2TB SSD and 1 X 4 TB for a drive pool of 8TB

Now all you have to do in the future in mange one drive letter X for everything. You just ignore the other drives completely. You can even hide them in windows so you dont see them.

So my a new spitfire library you down load it to Drivepool X: Spitfire

Point Native Access to Drivepool X: Native Access

Point OT Sine to Drivepool X: Sine

etc....

Makes things so easy to manage as all your sample are now on one single volume called Drivepool X:

Backup are super easy as you dont need to remember directories any more and makes backing up super simple.

I backup my Drivepool to a server also running Drivepool

Start running out of space on the pool? No problem just install new SSD in PC add the drive to the existing drivepool and voila the drivepool expands in seconds.

Want to remove a drive from the pool? No problem just remove it as long as there is enough free space in the pool to migrate the data.

PC dies one day? No problem you can access the data from any computer wether it has drivepool installed or not.

I'm down to 705GB free on my Pool and looking to add another 4TB SSD come amazon prime sales time. I dont have to do anything with my samples or data. I just add the 4TB SSD to the pool and the pool expands without me having to faff with moving files etc.

Hope this helps.

I am looking into Stablebit DrivePool:



It truly seems brilliant. Incredibly well thought-out and forward-looking.

I especially like the fact that, if one drive fails, *only the files stored on that drive are affected. This is huge, compared to Windows Storage Spaces.

Have you seen good performance using this system, EasyRider?