# Cubase/VEPro - upgrade to a 10gbit network or no?



## quickbrownf0x (Nov 14, 2021)

Right - got my fancy new DAW pc up and running and added my old one to the VEPro farm. BUT I hit a snag, I think.
Oddly enough performance has gone down. Loads of pops and clicks, watching *all three slave machines* choke on a bit of music that's not that complex, really (although still in surround). I've checked everything, all systems are optimized to the max, so that leaves me with one thing - the LAN network bandwidth/speed, right?

Since all the audio from the slaves is sent to my DAW, maybe a 1 gigabit LAN isn't enough anymore, what do you guys think?
If so, I have to sell my old Cisco switch. Can of worms. 

But you know, maybe I forgot something. Maybe it IS these new Focusrite drivers. In short - who wants to be my tech guy, so I can sleep in?


----------



## AcousTech (Nov 14, 2021)

First, some questions:
1. Are you certain all systems are functioning at 1Gbit full duplex? 
2. Are all NIC drivers current?
3. Which specific Cisco switch are you using?
4. Are all the machines running the same OS, VE Pro software, patch level, etc.?
5. Does the same thing happen with only 1 machine in the VE Pro farm?
6. Two machines? 

Short answer: maybe, it’s the network, but there are some things to tease out first.


----------



## quickbrownf0x (Nov 14, 2021)

AcousTech said:


> First, some questions:
> 1. Are you certain all systems are functioning at 1Gbit full duplex?
> 2. Are all NIC drivers current?
> 3. Which specific Cisco switch are you using?
> ...


Hey, thanks!

1. Yup, full duplex - double-checked, both in Cisco control panel and just by looking at the LEDs. 1Gbps.

2. Yes. Checked

3. It's a Cisco 26-port SG200-26 smart switch

4. All three VEPro slaves are running exactly the same, latest versions of Windows 10 Pro, 21H1 and VEPro7 software and the hardware is (now) identical; three 8700Ks, 64GB RAM, all NVME, M.2 drives. Zero moving parts. Nothing spins, except the fans, lol. Sample and Synth libraries are always separated from the main OS drives to miminize interference. My new DAW pc, however is running the latest version of Windows 11. Maybe that and shitty Focusrite drivers are what's causing it?

5. No, that's just the thing - all of a sudden - ALL my VEPRo slaves are all behaving like they're about to have a heart attack.

6. One DAW, three VEPro7 servers/slaves.

I ordered 3 TPLink 10gBit network adapters just in case, so that means I'll probably have to replace/sell my Cisco router in the future, but you know - rather not (now). Speaking of which, who wants a fancy Cisco 19" rack mounted switch?


----------



## AcousTech (Nov 14, 2021)

What is it that makes you think it’s actually the VE Pro servers that are the issue? Like can you copy large files to from each of them to another of them with no problem? Can you do the same thing from the DAW(Win 11), to the VE Pro machines(Win 10)? If all of those perform roughly the same then this may not be truly a network problem.

Now, on the network front, is the software on the switch current?

Oh, and which NICs are in those systems currently? Just the LOMs(LAN on Motherboard)?


----------



## Jdiggity1 (Nov 14, 2021)

You could temporarily reroute your VEP channels to all use the same outputs, minimizing the number of channels being sent across the network. This will tell you whether it's a bandwidth problem.

Are you able to monitor the resource usage on the slaves, particularly any disk/CPU indicators inside kontakt/spitfire player etc.
I've had a similar issue that I fixed by disabling some instances of HZS.


----------



## quickbrownf0x (Nov 14, 2021)

AcousTech said:


> What is it that makes you think it’s actually the VE Pro servers that are the issue? Like can you copy large files to from each of them to another of them with no problem? Can you do the same thing from the DAW(Win 11), to the VE Pro machines(Win 10)? If all of those perform roughly the same then this may not be truly a network problem.
> 
> Now, on the network front, is the software on the switch current?


That's a good point. I copied a 3gB folder; 

between 2 VEPro machines: 24 seconds. 







between my DAW and both VEPro machines: 31 and 32 seconds






Zero issues. Hmm..., yeah you could be right. Look at that speed difference, though, huh? That's odd. 

Cisco switch software/firmware is up-to-date.


----------



## quickbrownf0x (Nov 14, 2021)

Jdiggity1 said:


> You could temporarily reroute your VEP channels to all use the same outputs, minimizing the number of channels being sent across the network. This will tell you whether it's a bandwidth problem.
> 
> Are you able to monitor the resource usage on the slaves, particularly any disk/CPU indicators inside kontakt/spitfire player etc.
> I've had a similar issue that I fixed by disabling some instances of HZS.


Hadn't thought of that, cool. Will check that. I just bought HZStrings last night btw, haha - go figure. But already had issues before I added that to my template (still working out what I want for HZS);


----------



## Ben (Nov 14, 2021)

Check out the Task Manager on your DAW system. You can monitor not only the CPU, but also the network load in it.


----------



## quickbrownf0x (Nov 14, 2021)

Ben said:


> Check out the Task Manager on your DAW system. You can monitor not only the CPU, but also the network load in it.


Hey Ben, thanks for helping out.

Yeah I did that - For this cue I only had 2 of 3 slaves (called Maxwell and Kelvin) doing all the hard work. The other one basically just sat there. Kelvin had to work the hardest, so its CPU went up to almost a 100% and network was at about 70% peak. It's RAM was at a comfy 37Gb, though.

Maxwell is a newly converted VEPro machine (my old DAW) and only has two synths and HZStrings on it. Same config as Kelvin; tweaked 8700k at 4.8Ghz, 64 RAM, etc. Maxwell also peaked at basically the same CPU/network ranges. Well, eyeballing it, it's around 65%. 

Also noticed the same in the VEPro CPU meter, where both Kelvin and Maxwell choke up at 100% and then go back down to 0 for a split second.


----------



## AcousTech (Nov 14, 2021)

OK. Down the rabbit hole we go...

Ethernet, even switched Ethernet, likely won't get much higher than 75% efficient due to communication overhead. So if you're already hitting 70% you're approaching being capped out. But that doesn't mean there isn't still room to optimize.

Now, are all of these systems on the same network VLAN? If not, then routing is introduced which will slow throughput.

If so, do you have Jumbo Frames turned on:
1. On the switch?
2. On each PC? 
This will cut down the Ethernet communication overhead and allow more data to be sent per frame. Then you need fewer frames to send the same amount of data. 

Are all of these switch ports independent - that is not part of LACP groups etc.?

This is likely already the case, but bears verification:
Do you have AV scanning turned off for the sample folders on all the VE Pro farm machines so the samples aren't being checked when being loaded?


----------



## rgames (Nov 14, 2021)

quickbrownf0x said:


> Since all the audio from the slaves is sent to my DAW, maybe a 1 gigabit LAN isn't enough anymore, what do you guys think?


1 Gbps is ~ 1000 Mbps. A 24-bit stereo stream is about 2 Mbps, so, rough numbers, you should be able to do about 500 stereo streams at 24 bit over a 1 Gbps connection with near-perfect efficiency. Now reality is it's not 100% efficient, so say you only get 80% efficiency, then you're at 400 24-bit stereo streams.

That's still a lot... but maybe you have that many. Easy to count the active tracks.

I've never come close to maxing out my 1 Gbps connection with as many as three slaves. I'm surprised you're measuring 70% load. That has to be ~350 stereo tracks. What kind of music are you writing that requires that many?

I'm also surprised you're maxing out CPU. I always hit real-time bottlenecks long before CPU bottlenecks, at least within the last 10 years or so. So again I'm curious what kind of music you're writing that creates that kind of load. That's really tough to do unless you're running 500 compressors or something like that.

rgames


----------



## quickbrownf0x (Nov 14, 2021)

AcousTech said:


> OK. Down the rabbit hole we go...
> 
> Ethernet, even switched Ethernet, likely won't get much higher than 75% efficient due to communication overhead. So if you're already hitting 70% you're approaching being capped out. But that doesn't mean there isn't still room to optimize.
> 
> ...


Okay, that's good to know. 

- Yes, all are on the same VLAN

1. No, jumbo frames are turned OFF on the switch. 
2. No, turned OFF as well for all machines
3. Yes

I'm wondering if jumbo frames are going to make things better or worse in my case. If you read up on it, it's generally not advised to use them, right?

4. Yup, made exceptions for all sample folders.


----------



## AcousTech (Nov 14, 2021)

rgames said:


> 1 Gbps is ~ 1000 Mbps. A 24-bit stereo stream is about 2 Mbps, so, rough numbers, you should be able to do about 500 stereo streams at 24 bit over a 1 Gbps connection with near-perfect efficiency. Now reality is it's not 100% efficient, so say you only get 80% efficiency, then you're at 400 24-bit stereo streams.
> 
> That's still a lot... but maybe you have that many. Easy to count the active tracks.
> 
> ...


One detail highlight:

watching all three slave machines choke on a bit of music that's not that complex, really (*although still in surround*). I've checked everything...

I still don't think this should max his system out, given what we know currently.


----------



## quickbrownf0x (Nov 14, 2021)

rgames said:


> 1 Gbps is ~ 1000 Mbps. A 24-bit stereo stream is about 2 Mbps, so, rough numbers, you should be able to do about 500 stereo streams at 24 bit over a 1 Gbps connection with near-perfect efficiency. Now reality is it's not 100% efficient, so say you only get 80% efficiency, then you're at 400 24-bit stereo streams.
> 
> That's still a lot... but maybe you have that many. Easy to count the active tracks.
> 
> ...


Hey Richard, how are you? Thanks for helping out. Yeah, that's the thing - I really don't get it either. 
Maybe it's because all slave machines are returning in Quad(.1), not stereo (even though I know most libraries are in stereo)? So yeah, 4 channels per instrument.

Overall, I don't process the audio on the slaves, except for a tiny bit of panning, EQ and gain staging here and there, but that's it. So no fancy plugins. I leave that to my DAW. 

The music is a 20-minute cue, with multi-layered strings (for example 3 different violins (solo, ensemble), up close, in situ), mixed the same, but with a different library) and brass and a bunch of Zebras (like, 5 or something). Don't think it's that big a deal, right?


----------



## quickbrownf0x (Nov 14, 2021)

AcousTech said:


> One detail highlight:
> 
> watching all three slave machines choke on a bit of music that's not that complex, really (*although still in surround*). I've checked everything...
> 
> I still don't think this should max his system out, given what we know currently.


Yeah, me neither.  And it's also the fact that before I switched over to my new DAW pc I had zero issues.


----------



## AcousTech (Nov 14, 2021)

quickbrownf0x said:


> If you read up on it, it's generally not advised to use them, right?


Well, like many things, it is generally "best" if you don't know what you're doing. However, all the video editing suite people use these to enhance the ability to get data to/from their storage arrays - assuming they are using iSCSI. Anyway, it would be a harmless thing to test, and is easy to revert:

1. Enable it on the switch. Probably set for ~9000 bytes.
2. Enable it on the NIC for each PC at roughly the same size. Probably won't require a reboot, but it doesn't hurt. 

If it doesn't improve things, roll back. You can even do the same file copy you did previously to see the difference. So that's something you can test without even playing with the cue. Just see if moving files moves faster. It should. 

Anyway, if you use that DAW to also browse the Internet, etc. then when you start sending packets anywhere other than amongst your audio systems they will need to be fragmented, and that's where you might see slow downs. More on uploading than downloading though.

Anyway, there is still something suspicious here, but I'm not yet sure it's the wire.


----------



## quickbrownf0x (Nov 14, 2021)

AcousTech said:


> Well, like many things, it is generally "best" if you don't know what you're doing. However, all the video editing suite people use these to enhance the ability to get data to/from their storage arrays - assuming they are using iSCSI. Anyway, it would be a harmless thing to test, and is easy to revert:
> 
> 1. Enable it on the switch. Probably set for ~9000 bytes.
> 2. Enable it on the NIC for each PC at roughly the same size. Probably won't require a reboot, but it doesn't hurt.
> ...



1. Done
2. Done

between 2 slaves; 22 seconds vs. 24
between my DAW and a VEPro slave; 28 seconds vs. 31

That's awesome. Internet speed even feels snappier, not sure if it's just me. 

Here's my buddy Kelvin in the middle of chewing through my cue. Still about the same in terms of CPU/network load, BUT I do notice a significant reduction in dropouts. To be sure it's not just because of sample caching and so on, I restarted, reloaded the VEPro server project beforehand to purges all samples.






So lots of thanks for that! It at least did something.


----------



## AcousTech (Nov 14, 2021)

OK. So this:





Concerns me. That's not getting 1Gbps speeds. That's 100Mbps. And that will cause the CPU to backup like it appears to be, too. 

What NICs are you running in that system? 

Would you send a picture of the setting(s) you changed?


----------



## quickbrownf0x (Nov 14, 2021)

AcousTech said:


> OK. So this:
> 
> 
> 
> ...


What the hell? You're right. 

So before I changed Jumbo Frames from disabled to;







Also enabled Jumbo frames on my switch, which took a reboot. 






Look at this - only port 12 run at 100M, but that's because it's my Synology NAS. 


And after reading the 100Mbp/s I manually set this for all machines (which should force it to work at 1Gbp/s right?);






So with all that I run the cue and still: 







What on Earth is going on?


----------



## rgames (Nov 14, 2021)

quickbrownf0x said:


> What on Earth is going on?


It changes the scale to give you better insight into the data. You're only using 65 Mbps so a 100 Mbps scale is perfectly fine. If you plot it on a 1000 Mbps scale it'll be all squashed down at the bottom of the plot. Hard to see. That's what the "Ethernet" plot down below shows - it's plotting % of capacity and you can't really see what's going on because it's all down at the bottom of the plot (which means you're using a small percentage of capacity).

The only problem I see is the huge CPU load in VE Pro. That doesn't make sense. Are you using multiple processors in VE Pro? How many do you have available on that machine?

The network stuff looks perfectly fine.

rgames


----------



## quickbrownf0x (Nov 14, 2021)

rgames said:


> It changes the scale to give you better insight into the data. You're only using 65 Mbps so a 100 Mbps scale is perfectly fine. If you plot it on a 1000 Mbps scale it'll be all squashed down at the bottom of the plot. Hard to see. That's what the "Ethernet" plot down below shows - it's plotting % of capacity and you can't really see what's going on because it's all down at the bottom of the plot (which means you're using a small percentage of capacity).
> 
> The only problem I see is the huge CPU load in VE Pro. That doesn't make sense. Are you using multiple processors in VE Pro? How many do you have available on that machine?
> 
> ...


Ah, right. That makes sense. I did figure out that one of my slaves was using a different NIC (Intel Killer blahblah card), which was capped at 100Mbp/s, so I switched the cable, plugged it into the other Intel port, the one I _actually _set to 1gbp/s.  At least that's working now, although it's weird how that didn't show up in my Cisco switch port overview.

Here's the CPU core overview of one of my slaves, while it's working: 






Kinda busy, huh? 

Here are my preferences for that machine; 






2 Threads per instance has always been the most stable for me. Don't know why. Tried different settings.


----------



## AcousTech (Nov 14, 2021)

rgames is correct. I had forgotten about the dynamic scaling of the Resource Monitor view. Good catch. Another thing you can do to verify is look in Settings for the network card. 

Here's Windows 10:





And here's Windows 11:





As you can see those both show 1000/1000 - which is the confirmation you're looking for and is less "dynamic" a reading. 

From your screenshots, it sure looks like the wire is fine, but VEPro is just clobbering the CPU. Unfortunately for me, that's the deep end of the pool, and I don't swim there. Yet.


----------



## AcousTech (Nov 14, 2021)

Well, maybe I lied a little bit. Or can at least dangle my toes in the water. From the VE Pro manual:

You may change the number of used threads in the Multiprocessing drop-down menu in real-time, and each Vienna Ensemble Pro instance will use the specified number of process threads. So 1 instance on an 8-core computer should use 8 threads, 2 instances should use 4 threads, and so forth. Ideally you should have as few threads as possible while still using all your cores.

More here: https://www.vsl.info/en/manuals/vep7/vep-preferences

That seems to suggest that you might be better off with that Thread count setting at 1 and giving that a go. So instead of putting 2 threads per processor, you only assign 1. Less multi-tasking/time slicing per processor core would generally give you better performance. Assuming you have enough cores for the work you want to get done. 

I don't recall which CPUs you have, or how many cores they have, to be able to comment further.


----------



## quickbrownf0x (Nov 14, 2021)

AcousTech said:


> Well, maybe I lied a little bit. Or can at least dangle my toes in the water. From the VE Pro manual:
> 
> You may change the number of used threads in the Multiprocessing drop-down menu in real-time, and each Vienna Ensemble Pro instance will use the specified number of process threads. So 1 instance on an 8-core computer should use 8 threads, 2 instances should use 4 threads, and so forth. Ideally you should have as few threads as possible while still using all your cores.
> 
> ...



Yeah I remember reading the same thing and experimenting, ended up with 2 threads, but will set it to 1 to see if it helps. So all three slaves are 8700Ks with 6 cores, 12 threads. Maybe it's because 'm running 10 instances on one machine? I've got 5 on the other and just 2 so far on number 3. 

Also checked the link speeds in Win10 and 11; it's all 1Gbp/s, so that's good. Thanks again for all the help. Really appreciate it.


----------



## AcousTech (Nov 14, 2021)

Hmmm. Further down the page there is also this multi-processing note:

vep-preferences

I'd probably have to be running this myself to tinker a bit and figure out those implications. Deep water indeed. I did notice this though:






So whatever changes you make seem like they should be made on all systems.


----------



## quickbrownf0x (Nov 14, 2021)

AcousTech said:


> Hmmm. Further down the page there is also this multi-processing note:
> 
> vep-preferences
> 
> ...


Thank the baby jesus they're at least all carbon copies in terms of hardware, etc. lol
Cheers, I'm out for now. Need to get some sleep before I pass out on my keyboard. It's already 5AM


----------



## AcousTech (Nov 14, 2021)

Hardware, yes, which is good. That note, however, is about the configuration of the Audio & MIDI settings in Windows. That's the important part. Any _OS-level_ AND _VE Pro_ configuration settings are supposed to be identical. Almost like the VE Pro server can't know about distinct configuration settings between systems and so it just assumes they are all the same. If they aren't however, perhaps madness ensues?

Now, this is coming from a guy that has never run that platform, so what do I know...


----------



## quickbrownf0x (Nov 15, 2021)

@rgames Richard, how many instances per slave machine are/were you running? Just curious. I'm thinking maybe I could optimize things more on that side. See if that helps in any way.

Maybe get into track archives as well...


----------

