# RAID 0 and Kontakt Patch loading



## Tanuj Tiku (Nov 7, 2013)

I have just put RAID 0 for three of my 1 TB WD RED drives.

While smaller samples load extremely fast and normal transfer between the drives through windows is a serious jump, big patches in Kontakt are loading even slower than before. 

Like Embertone violin and Galaxy pianos - ridiculously slow. 

Why is this happenig and are there any settings that I need to be aware of when using a RAID array for sample streaming.

These drives are for sample streaming only. My system and project drives are SSD.


Thanks.

Tanuj.


----------



## JohnG (Nov 7, 2013)

Hi Vibrato,

A long while ago (2006), Sound On Sound magazine published a study of the benefits of RAID for audio, which they said were small:

"While you do see a benefit in performance using a RAIDed volume, you'll notice that the increase is small. You'd get better performance using the drives independently than together in a RAID configuration. I only tried up to three drives, because it's best to keep the system drive separate to the audio RAID, just in case of a drive failure."

Here's that article: 

http://www.soundonsound.com/sos/dec06/a ... s_1206.htm

In a slightly more recent article, they tested RAID with a hardware card. I am frankly not so sure this time exactly what they concluded. They wrote, first, that the performance improvement is not that great:

"As we've discussed in previous Apple Notes columns, using RAID in conjunction with audio software rarely gives a significant performance increase (in terms of the number of tracks), because although the throughput increases, the seek time remains the same. Where RAID excels in terms of performance, is usually with a small number of bandwidth intensive files, such as HD video, rather than a large number of — by today's standards, at least — low-bandwidth audio files."

But elsewhere in the same article they report getting to 250 audio tracks in Logic using a RAID card. The card seems to be the key, but in scanning through the article (I have lost the ability to really read), I couldn't see what exactly the conclusion was -- RAID card or not.

Anyway, here's the article:

http://www.soundonsound.com/sos/jul08/a ... s_0708.htm


----------



## germancomponist (Nov 7, 2013)

Sa far as I remember from the past, the card is the most important thing. It must be a good hardware card, not an emulation via software... .


----------



## chimuelo (Nov 7, 2013)

I saw a single example of Hardware assisted RAID that actually did increase the perfromance but at a huge cost.
It was the NetCell RAID 3 Card made specifically for streaming large amounts of Video which translates to large chunks of data bytes, which could be arranged in sizes from 1MB to 512MB IIRC. But it was like 6 or 7 drives and all were Glyph racks too.
I would use seperate drives as JG mentioned.

But you might want to convert all libraries to NCW format where you end up getting really fast performance boosts. I use an all NCW template where I specifically purchased them as they were developed using this compression. Orange Tree and AudioBro are all made this way. But I seemed to have converted VGP2, Plectrum and other older librairies accidentally when rebuilding my database, as I remember I was shocked to see Kontakts CPU/RAM usage and sizes diminished drastically, and loaded like greased lightning.

I confirmed this in Bidule as it loads templates several times faster than a Host like Cubase SX2, and Reaper which loads much faster than Cubase.

Just a thought. And good luck with the new project.


----------



## Gusfmm (Nov 7, 2013)

Tanuj- as been said, it's critical that it's hardware RAID, not emulated. Not sure what you do since you didn't mention. I personally use LSI RAID controllers.

Second, I love WD, but only use black RD3/4 drives for my RAIDs. No experience with those red drives. Just say this as not all WD drives are suitable for RAID. 

You also dont explain what you are comparing against, when you say they're slower now.

Theoretically, you should be seeing a fair improvement in speed loading larger files. Not so much with smaller, as the smaller they are the less spread among the RAID, thus the less benefit in parallel reading. It is odd the effect you're reporting. Probably has to do with your RAID being emulated, I'd adventure to say.


----------



## rgames (Nov 7, 2013)

Concur with the above - I've never seen any real benefit for RAID arrays for samples. Yes, the read speed goes up, but that's only one piece of the puzzle. The other piece is the seek time (i.e. latency) in starting the data stream. It's kind of like an internet connection - you can have a blazing fast transfer rate but if the transfer takes forever to start, it's a pain.

Read speeds are important for video or recorded audio (e.g. as in a Pro Tools session) but less important for samples. The reason is the number of files being accessed: in video or audio, the number of new files being accessed *per unit of time* is relatively few. For the audio example, even though there might be 100 tracks playing, they're not constantly starting and stopping - you have a relatively constant stream of data compared to a sample player where you're constantly accessing thousands of small snippets of sound. Likewise for the video example: even though the bandwidth requirements are huge, you're accessing only a few clips at a time, so the drive doesn't have to constantly find the locations of new files.

It's kind of like filling buckets with a hose. Let's say you have a hose that can pump out many gallons of water per second (huge bandwidth). If you have to fill up only a few large buckets then you can make good use of that bandwidth. However, if you have to fill up 1000 tiny buckets then you waste a lot of timing just moving from bucket to bucket. In that case, the bandwidth doesn't buy you much - you'd do better reducing the time it takes to move between buckets rather than trying to get faster rates out of the hose.

It's possible that your RAID array produces longer load times because you now have two hoses and have to use both to fill up any one bucket. Again, the time it takes to coordinate that dual-hose stream might negate any increase in overall bandwidth. Of course, if you just have a few huge buckets to fill then having two hoses will help.

I have a range of SSD's with speeds that range from 400 MB/s to 550 MB/s and they all give about the same number of streaming voices. They all perform *much* better than any HDD but I think it has as much (more?) to do with the much-lower seek times in SSD's vis-a-vis HDD's.

I guess that's a long way of saying your results are not unexpected. 

It's also a good example of why read speeds are not necessarily the best metric for evaluating a disk system for the sample streaming application. Seek time plays a huge role for sample streaming.

rgames


----------



## sinkd (Nov 7, 2013)

rgames @ Thu Nov 07 said:


> It's kind of like filling buckets with a hose. Let's say you have a hose that can pump out many gallons of water per second (huge bandwidth). If you have to fill up only a few large buckets then you can make good use of that bandwidth. However, if you have to fill up 1000 tiny buckets then you waste a lot of timing just moving from bucket to bucket. In that case, the bandwidth doesn't buy you much - you'd do better reducing the time it takes to move between buckets rather than trying to get faster rates out of the hose.
> rgames



Extraordinarily lucid analogy. Thanks.

DS


----------



## Tanuj Tiku (Nov 7, 2013)

Thanks guys!

I can confirm that RAID 0 has not presented with faster results. In most cases, its exponentially worse for sample streaming.

My project and system drives are SSD and so there is no problem there.

It seems as Rgames said that because indeed thousands of samples of a single patch are now stored randomly onto three 1 TB drives in the form of a RAID array, the seek times become ridiculously slow for some reason. 

Compared to standard streaming from three different drives, RAID 0 has no benefit on my system. 

It works great for single large files but loading thousands of samples for a single patch does not work well at all.

Interestingly, VSL was the only library that worked very well with RAID 0 because VSL is very efficient and smaller patch sizes. 

Try loading Friedlander violin - it will take upto a minute and even more I think. Same with Galaxy Pianos etc. 

I have disbanded my RAID array as of last night and gone on to the normal system. 

Three 1 TB SSD drives will cost me to the tune of $2200 in India. Not practical for me at the moment!


Thanks for the links John - they make sense. May be for audio, it makes sense. Definitely not for sample streaming.

I have a decent on board RAID controller on the motherboard. On the system itself within windows - it is FLYING!

A single 2 GB file will transfer instantly - windows does not even bother showing the progress bar. But if you load a 2GB patch in Kontakt with hundreds of small samples stored in various places - disaster!


Thanks.

Tanuj.


----------



## Gusfmm (Nov 8, 2013)

Tanuj- your RAID on montherboard is the worst option you can have. That is an emulared RAID controller. I can assure you that your experience is no significative as for what a stand-alone interface can deliver. So unfortunately, your case should not be interpreted as a definite conclusion to RAID performance.

Also, several aspects of your test where unclear as said before.

To Richard's point, what is really odd is that Tanuj experience was contrary to that (which is quite unexpected) where smaller files apparently loaded faster. That should not be the case with mechanical drives.


----------



## Tanuj Tiku (Nov 8, 2013)

Gusfmm,


You may be right in saying that a standalone controller will offer better performance. Why the on-board controller is worse than the normal performance of the drive is not something I understand. 

Tests were not made by myself regarding the stability of the RAID array. It was done by the guy who built this machine. He did many hours of stress test on the drives. The results were amazing. 

Within windows - the transfer rates were amazingly fast. I cannot give you the stress test results as I did not run them. But practical file transfers were made with large files in front of me and the performance was really great.

Unfortunately, when I came back home with the machine and started loading sample patches - the performance was extremely bad even compared to the drives original performance. 

With stress tests and normal file transfers within windows environment being very good indeed. 

Perhaps, it was a case of bad controller but in that case outside Cubase/Kontakt - I should in theory experience the same bad performance.

This was just my experience and after a few hours, I had to start writing a new cue for a score - so patience ran out and the RAID issues were not helping me be creative 

So, in the end I removed the RAID configuration. If anyone here is getting excellent performance with RAID for sample streaming - please post so.

It seems nobody is doing it because there are no replies of people actually using this in their set-up. I was very keen to hear from them. I posted on Scorecast London as well and nobody responded. 


Thanks for all the help guys!


Tanuj.


----------



## germancomponist (Nov 8, 2013)

Gusfmm @ Fri Nov 08 said:


> Tanuj- your RAID on montherboard is the worst option you can have. That is an emulared RAID controller... .



What I said. Makes only problems, CPU hits e.t.c. .


----------



## Tanuj Tiku (Nov 8, 2013)

Gusfmm,

Another thing I want to clarify is that when I say a 2GB patch - it does not mean a single path will have just ONE 2GB file. It is comprised of hundreds of little samples.

So in the end, they are smaller files but lots of them that a single patch has to pull in. this is where what Rgames says, makes sense. 

I am comparing the performance of the RAID array to the performance of the SAME drives as they were before they were in RAID 0. 

So for some reason I actually got much worse performance than I was getting when the drives were just set up as normal three individual 1TB WD RED's. 

If indeed ASUS did put such a bad RAID controller on the Sabertooth that in reality it gives at least 3 times less of a performance than when it was being used as a standard individual HDD then I am shocked at why they would even bother to put it in and that means it is a disastrous controller from ASUS which I have yet to read in any review of this board.


Tanuj.


----------



## germancomponist (Nov 8, 2013)

vibrato @ Fri Nov 08 said:


> If indeed ASUS did put such a bad RAID controller on the Sabertooth that in reality it gives at least 3 times less of a performance than when it was being used as a standard individual HDD then I am shocked at why they would even bother to put it in and that means it is a disastrous controller from ASUS which I have yet to read in any review of this board.
> 
> 
> Tanuj.



Tanuj, this has nothing to do with "Asus". 

These fake RAID controller solutions (in comparison with hardware) are unprofessional. It uses the main processor for computational work and also the internal bus systems are stressed.

I would suggest you to buy fast SSDs or HDs with big cache and maybe 10.000 rpm ... .


----------



## Gusfmm (Nov 8, 2013)

vibrato @ Fri Nov 08 said:


> So, in the end I removed the RAID configuration. If anyone here is getting excellent performance with RAID for sample streaming - please post so.
> 
> It seems nobody is doing it because there are no replies of people actually using this in their set-up. I was very keen to hear from them. I posted on Scorecast London as well and nobody responded.
> 
> ...




Ok, here we go. Let me first clarify:

. I use FOUR (4) RAID arrays in my main PC;
.. I use LSI MegaRAID SAS 9260-4i cards;
... I have a combination of 6 mechanical and 3 SSD drives;

First example: the following is a comparison between my single system SSD and a RAID 0 of two of my sample drives, exactly the same SSD model (tested with CrystalDiskMark v3. x64):


single disk (on SATA II port):
Sequential Read : 273.245 MB/s
Random Read 4KB (QD=32) : 47.8k IOPS

RAID 0 (9260-4i):
Sequential Read : 744.162 MB/s
Random Read 4KB (QD=32) : 49.3k IOPS

These SanDisk SSD's are rated (up to) 520MB/s sequential and 39K IOPS random reads.

I never take these results as absolute, but more as a reasonable representation of reality. It's clear how the RAID 0 on the LSI delivers about 50% faster read times. There are still certain factors influencing some of these results though, just to mention one: the system drive performs below standard, as it gets highly defragmented in time, typical of a system disk, dragging the sequential efficiency of the SSD down. Still outperforming any mechanical drive out there due to its radically faster access time.


Second example, my two WD Raptors 300 in RAID 0, that perform sequentially at 210MB/s, whereas they are factory spec'ed at 120MB/s. Another ~85% higher performance.

To me, the above clearly shows practical proof that there is indeed a performance improvement in using RAID. One must understand setting up a RAID is not only hooking up cables behind drives and clicking a checkbox on the BIOS configuration, there are several factors that come into play to properly try to maximize its advantages (block sizes, how files are written to drive, type of drives and controller, selected bus, motherboard design, ......).

To close, the question of how all this translates into sample performance is a bit more empirical, and open for different opinions and observations (e interpretations), as I've seen over the last few years. And I'm not interested in debating it here at this point, as I have my own theoretical viewpoint and some practical experience that some may not share.

Anyhow, hope this at least leaves you Tanuj with one precedent and opinion, for whatever it's worth it.


----------



## rgames (Nov 8, 2013)

The problem with read speeds on synthetic benchmarks is, well, they're read speeds on synthetic benchmarks. Unless your intent is to run synthetic benchmarks then they're interesting but not particularly useful. Your intent is (I assume) to stream voices in response to MIDI data with no clicks and pops, so that's what you need to measure.

It's kind of like measuring horsepower for your car. Sure, more horsepower is probably better, but you don't just assume that a car with more horsepower will perform better. You still have to put it on the track and see what you actually get! You don't actually care about the horsepower, you care about the time to complete a lap. Likewise, we don't actually care about read speeds, we care about streaming voices, so that's what we need to measure. It makes sense that better read speeds would give more streaming voices, but until you show it, you don't actually know.

The problem is there's no standardized way to measure number of streaming voices. I've created my own benchmark where I run up and down major scales in 16th notes at 120 bpm, modulating at the start of each ascent to try to prevent a caching effect. I then copy that track to bunches of other tracks across bunches of other instruments and bunches of other libraries (winds, strings, perc, VSL, LASS, EWQL, etc) and monitor the voice counts to see when I start to get clicks/pops. For a two-SSD setup, that happens around 1250 - 1500 voices. That number appears to be independent of the read speed for the ranges I've tested (see above, 400 - 550 MB/s), so I have to conclude that the read speed is not a major factor, at least within that range.

Now I'll be the first to admit that my setup might be an exception, but without any way to compare to other setups on the basis of what we actually care about (number of streaming voices), it's impossible to say.

We're stuck comparing horsepower when what we really care about is lap time.

rgames


----------



## Gusfmm (Nov 8, 2013)

Richard, one critical factor to keep in mind is that each of the samplers we use (Kontakt, UVI, Play, VIP, Engine, etc.) employs a particular strategy to handle playback, and disk-streaming. Just this one factor renders any such test as you describe above meaningless, or maybe more elegant to say 'non-universably extrapolative'. I stopped trying last time we talked about benchmarking months ago, after attempting to have a dialog with the VSL folks about buffer sizes and streaming on VIP and getting nowhere with them. 

Many other key factors also come to mind: selection of library (due to size and structure, compression), audio interface, RAM size, disk cache, scale vs. random notes, .......

The fundamental thesis of my reasoning above was that there must be a benefit from using a faster disk when streaming sample files, samples that tend to be larger files (e.g. 64Kb, 256Kb, etc), not smaller (e.g. 4Kb), thus the sequential speed providing a reasonable indication for performance. At the very least, the initial load time must improve. And I will stop here on this as I said before.

Going back to Tanuj observation that RAID offered no speed benefit but on the contrary, according to his very brief test, I simply provided some real test results that show it indeed does. Translating or trying to extrapolate that to somehow predicting the performance of a complex DAW computer goes well beyond such one factor. So utilizing your car metaphor, yes a higher HP car should run faster on a track, but whether or not it performs better overall than others is not just a function of speed. So I agree with you on that.


----------



## Tanuj Tiku (Nov 9, 2013)

Perhaps it was the controller guys. 

Thanks for your help. Thankfully, I will not need RAID in the next 6-months as I am going for a complete SSD system and that is fast enough for now 


Tanuj.


----------



## Dracarys (Nov 9, 2013)

Vibrato,

You say that your project and OS drive is SSD, and your HD's are for streaming in raid 0? If that is the case, I think you may have it backwards. There is no point in raiding HD's for streaming, research the speeds, even with 15000rpm drives. Also, there is no evidence prooving that DAW performance is better from having it on a SSD. 

I have a 1tb WD Black Caviar for my OS and DAW and template folders. I then have two 128 crucial m4's, and another 2 samsung pro 512gb ssds. (If I had more SATA ports I'd have a bunch of 128gb ssds) I have another internal 3tb Seagate with 150mb/sec reads I only stream very small samples from.

I opened a session last night that was roughly 26gigs with very intensive samples, it took less than 30 seconds to load. Your track counts will be higher and kontakt will put out more voices.

All the best!


----------



## Tanuj Tiku (Nov 9, 2013)

Anthony,

Agreed over all!

the OS SSD just makes windows much faster to load and snappy. Installations and everything else is very very quick. Although I agree - the system drive is not doing much else so even a normal drive could do!

The Project drive is because sometimes on a score, I do have to work with 200 audio tracks and above - we work to balance them with the engineer and in case of a 20 GB template, with 25 instances of Diva, Omnisphere, Spitfire, VSL etc etc - UAD, bunch of plug ins loaded - SSD becomes essential. 

Cubase is extremely snappy and tracks load very quickly and its easier to edit them, move them around - quicker bounces and file transfers. 

Also - what is the point of having a superb hardware spec and where the capability of the system is to manage much higher throughput? I even got carried away and decided to spend $2200 to get 3 X 1TB SSD drives but at this time even three of them are not available in Mumbai. 

It will have to be a special order - and I need to write everyday! Projects are getting delayed. I already ended up loosing time with the whole RAID thing. Everything works fine - it just could be better!

I agree - with better thinking, I could have made choices that work better but there are 4 movies lined up and I am almost locked on a daily basis till end of February. Not much time to experiment at this very moment!


Thank you all for your inputs - Perhaps once I finish the run of these movies, I will get to the bottom of this - most likely put in the SSD's. 

Also - The RAID is now removed. I got poor performance with RAID 0 - Some fine folks are saying its because of the controller - may be so or may be it was not set p fine. But the stress tests proved otherwise and within windows it was working very well. But when it came down to sample streaming for some reason it was worse than when the disks were NON-RAID.

Tanuj.


----------



## Dracarys (Nov 9, 2013)

If you have a 1tb for your project, OS, and DAW drive, those 20gb sessions still wont be an issue.

If you're seriously hurting and can afford SSDs atm, you can easily install the new SSDs - remove all your libraries from kontakt - then add them back after dragging them onto the ssds. This won't create any session issues.

Good luck!


----------



## wst3 (Nov 9, 2013)

Mr. Ames has done a pretty good job of describing the problem. And others have filled in some important details as well.

My short version would be something like this:

1) a software implementation of RAID will never work as well as a hardware controller... which of course will eventually become false as CPUs get more powerful<G>, but it certainly is true today.

2) the ultimate value of any storage strategy must take into account not just the hardware, but also the software! And since we can only empirically evaluate something like Kontakt, this becomes difficult.

RAID was conceived to allow the use of lots of smaller drives to provide availability and/or performance - Redundant Array of Inexpensive Drives - and it was a reaction to the high cost and limited capacity of SCSI drives.

In a mirrored configuration write times are usually worse than a single drive, and read times are better, but not remarkably so.

In a striped configuration write times can improve, and read times can improve significantly, but only for a specific range of file sizes... usually large files will show a larger improvement.

As has been described, seek time becomes an obstacle! If one could write to disc sectors in such a way that reads would be sequential then it wouldn't matter, but alas that isn't yet practical.

If you have an interest in arcane mathematics you can read about queuing theory, which is the basis for all this trickery. EMC has some wonderful papers on how they were able to create a multi-level cache strategy that really does work.

The result, today, is that mirrored drives provide tremendous security - right up to the part where they will write the same bad data to both drives<G>! Stripe sets can provide improvements in performance, at a huge cost in terms of security, but the performance improvements will be applicable to a range of file sizes that you likely will have no control over.
A striped and mirrored configuration is currently the optimal solution, but it still suffers from the same performance limits. And RAID 2, 3, 4, and 5 are all geared towards securing your data.

SSDs are changing the face of data storage. When funds allow I'm moving all my sample libraries (approximately 1TB) to SSDs, and manually mirroring them to a spinning drive. I think that's about as good as you can get today.


----------



## rgames (Nov 9, 2013)

Gusfmm @ Sat Nov 09 said:


> Richard, one critical factor to keep in mind is that each of the samplers we use (Kontakt, UVI, Play, VIP, Engine, etc.) employs a particular strategy to handle playback, and disk-streaming.


That's true and an important factor. Some playback engines are just more efficient than others - you can take a powerful machine and cripple it with bad code.

However, I've used my benchmark across a bunch of different libraries (VSL, Kontakt, EWQL) and the performance of all of them appears to be pretty independent of the read speed. Yes, some perform better and some worse, but they're all pretty unaffected by read speeds from SSD's. Again, of course, I throw in the caveat that I've tested only between 400 and 550 MB/s. Maybe if you get up to 1000 MB/s you'll see a difference - I don't know.

But, as Tanuj points out, there's also the "Better is the enemy of good enough" problem. Just because you can get more streaming voices doesn't mean you should if you can get by with what you have. That's where I am right now - my hardware is not producing bottlenecks, so maybe it could perform better but I'm not going to put in the time to see until I need to.

There are still plenty of people running HDD's in non-RAID setups, too. And they work just fine like that. The truth is that any machine with any disk setup from the last 5 years is sufficient to write pretty complex orchestrations.

rgames


----------



## JohnG (Nov 9, 2013)

Great analogy, Richard. Thanks!


----------



## chimuelo (Nov 12, 2013)

http://www.thessdreview.com/our-reviews ... d-storage/

For guys with 128GB Templates.
800,000 Random IOs is most impressive.
5GBs of seq. reads, which I really don't think are the most vital since we use RAM Caching.
To ensure each RAM buffer target finds a home wuickly, 800,000 Random Reads might just work.... :mrgreen:


----------

