Have you tried removing the sound cards and running over ethernet? Way back when (7-8 years ago?) I found lower latency over the network than with audio hardware in the slaves.
I run my full template at 128 buffer at 44.1 kHz with 1 buffer for VE Pro - that's 6 ms or somewhere around there (do the math...). Smaller projects can run at 64. My master is an i7 4930k and my slave is an i5 2500k, both on stock Win7 (no tweaks) with all orchestral libraries on SSD.
It can be dependent on the libraries - I use VSL/LASS/EWQL Brass/Cinebrass and all the usual synths. Of course, it's also dependent on voice count - higher voice counts generally require higher latencies in order to prevent clicks/pops. I can get 1250 - 1500 LASS/VSL voices out of either machine (CPU doesn't matter) at that buffer setting. Also, Macs generally run at higher latencies.
Honestly, though, "fast" acoustic instruments have latencies in the 50 ms - 100 ms range, and it's not even constant across the instrument. "Slow" acoustic instruments (e.g. bass instruments) have latencies of hundreds of ms. So trying to get down below 10 ms is fun for geeks but not much practical use. People have been playing acoustic instruments for a few millennia and musicians learned to adapt to the latency. Your brain is good at that type of stuff. Well, most peoples' brains are - I assume you have at least an average brain
.
Latency used to be a problem back in the early days of digital because it was many hundreds and possibly thousands of ms. But it's not an issue any more - pretty much any machine can do better than acoustic instruments. People have just forgotten to stop worrying about it...
HOWEVER, there are plug-ins that add latency that doesn't appear in the values reported by your DAW. I have a lin phase EQ that adds 1000 ms or more sometimes and, of course, that latency is very noticeable even though the DAW says the latency is 6 ms or whatever. So disable all your plug-ins before judging the responsiveness.
For me, somewhere around 15 ms is where I can't reliably tell a difference. But even at latencies higher than that, I can adapt with no issues. My guess is that somewhere around 100 ms is where I would call it "unplayable."
rgames