What's new

The vi-c blinded violins shootout - FINAL UPDATE.

Garry

Senior Member
Ok, the poll results are in from this thread, and as of 12pm CET, an overwhelming majority (86% of the 73 votes cast) voted 'yes', we want to see a blinded shootout between libraries, either just for fun or to help them make decisions about which libraries to purchase. So, your wish is my command...

Thank you to everyone for your helpful comments and suggestions in that thread. As one of the contributors to that thread rightly pointed out, we will not please 100% in exactly how we run this, so since this is our first attempt, let's go with it as described below, and there's nothing saying we can't improve the rules on the next shootout, as we learn from the experience. We now need your contributions, but before doing so, please note the rules.

The RULES:

SOME KEY POINTS: This is a blinded test, so to maintain the blinding, I've highlighted below some key points - please pay close attention to these, repeated here: (i) do not submit files to this thread, submit as PM to me, (ii) do not submit votes to this thread - send to the 'Randomiser' (explained below). To maintain credibility of the competition, do not use anything outside the library (or if you do, 2 versions must be submitted with/without the 3rd party plugins), and do not embellish with additional instruments or harmonic lines.
  • The INSTRUMENT: we'll be starting with violins only. If this round is successful and people find it useful, we can move on to others in subsequent rounds, but only violins for now.
  • The LIBRARY: any libraries are eligible that have violin patches.
  • The PLUG-INS:
    • For 'wet' libraries: (i.e. the library offers multiple different mics, EQ and reverb included within the library): within the library, you are free to set any combination of mic positions, articulations, effects that come provided with the library, that you feel best show off the library's capabilities.
    • For 'dry' libraries: (ie. the library doesn't have this, or is built to offer a really dry sound with no room), then third party-plugins can be used, but you must include (i) one dry version and one wet version separately, and (ii) a screenshot to show settings of all 3rd party plugin settings/presets (credit to @fretti here for a great solution to something many had expressed concern about). Remember, we are not trying to equate the contributions, we are trying to capitalise on the considerable expertise we have in this community to show off the libraries at their very best, or show what is additionally required to make them shine.
    • it would almost seem to go without saying: platform engines, e.g. kontakt, are of course allowed!! Apparently it does need saying, so, 'yup, they are allowed too' ;)
  • The MELODIC LINE: with thanks to @Saxer , we have 7 phrases which provide a perfect variety of tone, tempos and articulations. The MIDI file is provided below. Please do not embellish with additional instruments (even from within the same library) or harmonic lines - this would become more of a test of the composer, and that's for another day!! Please do not write the midi in yourself - we want to make this the constant on which to base comparisons from, however, you can add CCs, velocities, etc, to enrich the performance.
  • The SUBMISSION: please send 1 audio file per library as a PM to me. Please DO NOT submit the file to this thread - this is an important part of the blinding. Please include in the filename, the name of the library used. The submission must include all of the sections of Saxer's midi file in order to qualify. I will collate all the files, and attach pseudolabels (library_A, library_B, etc) to each. I will not vote, as I will be unblinded. You can submit more than 1 library - as many as you choose - remember, 1 audio file per library. Please also consider sending a note to this thread (not disclosing the library name), so that people are aware of the ongoing activity.
  • The RANDOMISATION & POSTING: I ask for 1 other member to volunteer to receive these files from me (please send me a PM), and simply randomise the order, randomly re-labelling the files library_1, library_2, etc, in order that no file benefits from an order position effect. The person randomising the files (let's call this 'the Randomiser') will post all the files to this thread, at the same time. S/he will be allowed to vote, since they remain blinded, and so cannot exert undue influence.
  • The VOTING: votes will be sent by PM to the Randomiser only. Again, please do not post your votes to the thread, as it could easily influence other people's ratings. This too is a key part of the blinding.
  • The RESULTS - the Randomiser will then post the results of the voting (still blinded). I will then post the unblinding (including names of libraries and contributing author for each file (unless you specifically indicate to me that you would prefer to be anonymous), and all will be revealed!
  • The DEADLINE: 1 week today (Wednesday, June 6th)
I've no doubt there are flaws with this method, and some people will have objections, and perhaps someone else could do a better job. But, in the interest of getting things moving, and not letting the excellent be the enemy of the good, let's go with this, warts and all, and any suggestions for improvement can be included in the next blinded shootout. Finally, thanks to @Vik, whose recent, single-handed heroic efforts, comparing violins and cellos, provided a great platform to learn from that exercise, get feedback from the community, and now extend this out to the broad VI-C expertise.

So, now we need your contributions! Please feel free to start sending the audio files to me by PM (remember to label them with the library name).

Let Battle Commence!

[AUDIOPLUS=https://vi-control.net/community/attachments/violinslegatotest-mp3.13651/][/AUDIOPLUS]
 

Attachments

  • ViolinsLegatoTest.mp3
    2.8 MB · Views: 454
  • ViolinsLegatoTest-Midi.zip
    1.8 KB · Views: 85
Last edited:
OP
Garry

Garry

Senior Member
Thread starter
  • Thread Starter
  • Thread Starter
  • #2
In case it helps, here is the notation for the MIDI file, provided by @Saxer:

Violins Legato Test.jpg
 

fretti

Senior Member
Maybe the middle way is the following:(?)
- library offers multiple different mics and a included reverb control: has to/ can be used to the extend of the individual doing the work to his/her liking.
- if the library doesn't have this, or is built to offer a really dry sound with no room (like VSL) then the there should be one version without reverb and one with the reverb the submitter decides to use, so it is in his/her eyes the best result.
But the PM to @Garry should then include:
  1. what 3rd Party reverb was used (product wise; plugin or external etc.)
  2. what preset or settings were used (maybe a screenshot)
so both versions can be included, but the reverb details have to be listed later so we at least can see the difference reverb can make.

Could make it fairer, towards certain libraries. Might also just lead to the opposite of: the included mics sound different than a 800€ reverb plugin on top of a good quality sample library...
 
OP
Garry

Garry

Senior Member
Thread starter
  • Thread Starter
  • Thread Starter
  • #4
Maybe the middle way is the following:(?)
- library offers multiple different mics and a included reverb control: has to/ can be used to the extend of the individual doing the work to his/her liking.
- if the library doesn't have this, or is built to offer a really dry sound with no room (like VSL) then the there should be one version without reverb and one with the reverb the submitter decides to use, so it is in his/her eyes the best result.
But the PM to @Garry should then include:
  1. what 3rd Party reverb was used (product wise; plugin or external etc.)
  2. what preset or settings were used (maybe a screenshot)
so both versions can be included, but the reverb details have to be listed later so we at least can see the difference reverb can make.

Could make it fairer, towards certain libraries. Might also just lead to the opposite of: the included mics sound different than a 800€ reverb plugin on top of a good quality sample library...

Great suggestions, thanks Fretti - I'll update the rules.
 
OP
Garry

Garry

Senior Member
Thread starter
  • Thread Starter
  • Thread Starter
  • #6
Eventually no one's forced to buy the full version, yet if they like what they hear and they're serious about getting same results, then it's still a good thread for them to help them consider buying the full version...for good reasons.
Just to be clear, Kontakt (whether full version or not) is clearly allowed. It was obvious to me, and I thought to others, that we were aiming to preclude the use of third-party plugins that would be used to enhance the sound outside the constraints of the library, and it seems to me an obfuscation to pretend to be unaware that was the case. For the sake of complete clarity, I’ve added what seems an entirely unnecessary clarification that Kontakt is of course allowed!

I hope you can enjoy what was intended to be of mutual benefit to the whole community, and hopefully fun in the process. Let’s see...
 
Last edited by a moderator:

fiestared

Vintage -but- not obsolete
Yes, both good points. I’m not sure I can shorten the original, because I will of course be accused of missing something (ahem...). But you’re right, we just have to hope people are interested enough to read it through. Also, if it’s shorter, then we’ll probably end up with this being a very long thread of clarification questions.

As to your 2nd point, yes I’m concerned about that too. I had planned to periodically update the number of submissions, and perhaps highlight libraries that had not yet been represented. As it is, that’s not yet our problem. We don’t yet have any submissions! So, come on VI-C, let’s get this going.

Thanks for pointing these out Ka00 - perhaps in the next round, the lengthy description will be unnecessary, and we can have another way of submitting. The reason for sending to me is that I then blind them, which means when I send them on to someone else, it becomes double-blind - a point that was made in the other thread as being important. If not sent to me via PM in future iterations, we’d still need an independent way of collating them. I stay independent by not voting. Open to suggestions though.
Why not simply "sending you the file and writing a post to say it to others" ok, two actions, but at least this thread would be active.
 

Casiquire

Senior Member
Well a good way around that issue is to have members post something when they've submitted files. As most of us have more than one string library and we wouldn't know which example came from which member, blind-ness would be retained.
 

ka00

Senior Member
The reason for sending to me is that I then blind them, which means when I send them on to someone else, it becomes double-blind - a point that was made in the other thread as being important.

But does it in fact need to be double blind? Just blind to the thread readers would be sufficient. What sort of bias would you personally introduce as the admin of this test? You are just compiling and posting the resulting sample tracks. Everyone is making conclusions for themselves at the end, no?

Would eliminating your own ‘blindness’ simplify things on your end?
 
OP
Garry

Garry

Senior Member
Thread starter
  • Thread Starter
  • Thread Starter
  • #11
But does it in fact need to be double blind? Just blind to the thread readers would be sufficient. What sort of bias would you personally introduce as the admin of this test? You are just compiling and posting the resulting sample tracks. Everyone is making conclusions for themselves at the end, no?

Would eliminating your own ‘blindness’ simplify things on your end?
Let’s say I preferred library A over library B - if I was trying to bias it deliberately (or indeed subconsciously), I could put library A at the beginning or the end (primacy/recency effect: we tend to bias our preference and improve recall of items that occurr at the beginning or end of a list). So, if I blind the files, and then someone else randomises, I can’t influence the order, and they can’t prejudice one library over another in ordering, because they’re blinded to library name.

In medical science, we do this all the time: the pharmacy prepares the drug or placebo (so is unblinded to the treatment), but it is the statistician that pre-determines the treatment randomisation - the combination protects against bias (my real job is in neuroscience - I’m just a wannabe musician!).
 
Last edited:

ka00

Senior Member
Let’s say I preferred library A over library B - if I was trying to bias it deliberately, I could put library A at the beginning or the end

If it simplifies things, when you get all the files, just list them for yourself here and hit 'randomize', then adhere to that random order. If we can't trust you to do that then the whole thing is null and void anyway. PS: I trust you!

my real job is in neuroscience - I’m just a wannabe musician!

Then you know we're all just going to Yanny and Laurel the results anyway! ;)
 
OP
Garry

Garry

Senior Member
Thread starter
  • Thread Starter
  • Thread Starter
  • #13
If it simplifies things, when you get all the files, just list them for yourself here and hit 'randomize', then adhere to that random order. If we can't trust you to do that then the whole thing is null and void anyway. PS: I trust you!
Yes, tools like that are useful when we want to randomise (although they typically have more structure than you’d think), but transparency is important in any trial, and the elimination of doubt is key to the acceptance of the results. You trust me now, but what about when your favorite library, that you paid top dollar for, comes bottom of the pile!! ;)

Then you know we're all just going to Yanny and Laurel the results anyway! ;)
What are you talking about, there’s only Laurel! ;)
 
Last edited:

Arviwan

Member
Hi everyone,
excellent idea !
I've just downloaded the MIDI file and will try and record examples next week-end.
Just one thing i wanted to say : since the file includes 7 different phrases, it might be relevant to compare phrase by phrase, 'cause i think 1'33 for each bank is a little too long ... or at least is it for my ears ! ;)
Cheers
 
  • Like
Reactions: Vik
OP
Garry

Garry

Senior Member
Thread starter
  • Thread Starter
  • Thread Starter
  • #15
Hi everyone,
excellent idea !
I've just downloaded the MIDI file and will try and record examples next week-end.
Just one thing i wanted to say : since the file includes 7 different phrases, it might be relevant to compare phrase by phrase, 'cause i think 1'33 for each bank is a little too long ... or at least is it for my ears ! ;)
Cheers
Yes, I know what you mean: Saxer's file perfectly addresses a concern that was raised that a 10 second file doesn't give a true impression of a library, but on the other hand, 1'33 makes it a lot harder for people to compare - it becomes a memory test as much as anything! We could compare phrase by phrase, but then people would need to send 7 files, rather than one. Hmmm... I'm not sure - what do people think? We want it to be meaningful, but we want it to be practical as well. If it all just feels like too much trouble, people won't participate.

Also, you mentioning looking at this at the weekend raises another question: we need to set a deadline - does 1 week from today sound reasonable?
 
OP
Garry

Garry

Senior Member
Thread starter
  • Thread Starter
  • Thread Starter
  • #16
+another thing I just realized wich might be a problem: section sizes.
Do we allow all sizes (depending on what the participant thinks sounds best with that exact library) or do we want limits?
What do you all think: limits or just whatever sounds best for the one making the file?
Just asking because with all the libraries out there right now we have like unlimited combinations and section sizes in those libraries (from 4 in SCS to 60 in HZS...)
Personally, I'm ok with this: does it actually sound better with 60 violins, or not? Of course, that may depend on the requirements of the music, and people will have to generalise their ratings there, since we can't have a competition for every type of articulation for every type of music. But I'm not convinced people will pick out the 60 violins - to my ears, they're not sounding as big as one might have expected. If 60 violins sounds awesome, and stands out above the rest, then Spitfire have a winner and will make a lot of money from their library, but I'm eager to see the results, and whether people can actually pick them out in a blind test.
 
Last edited:

Saxer

Senior Member
If I may add my opinion:
Don't complicate everything by rules and methods. If possible: no external plugins. Use the software that is needed to run the library. Use the whole midi file or a part of it. Or make your own music. If you use something else there's probably a reason. If so, it would be kind to tell us why. Send the files to Garry and let him sort the things. I think that should be it. And there are definitely no reasons to drive this thread into the drama zone.

I made the test example with all problems in mind that came across using different libraries. So there will probably problems coming up sooner or later. It would be interesting how the libraries or the users of it (you/me/we) can handle the examples. I could imagine that often more than a single patch is needed to get good results. The borders of libraries will be interesting too. If everything runs fine without problems with most libraries I will have to learn lot how you did it! Probably all my problems were home made then.
 
OP
Garry

Garry

Senior Member
Thread starter
  • Thread Starter
  • Thread Starter
  • #18
If I may add my opinion:
Don't complicate everything by rules and methods. If possible: no external plugins. Use the software that is needed to run the library.
I was originally of this opinion too, but I think it's a fair point that dry libraries would likely not fair well, so it's useful information to know what it takes to make them sound good, but disclosing the plugins used, and comparing with/without, and I still like Fretti's suggestion for dealing with this.
Use the whole midi file or a part of it. Or make your own music.
Hmm... I have to strongly disagree here. This is only a useful test when we hold one thing constant. If even this is variable, then we will be in no better a situation to compare libraries than we are now, just listening to what's randomly out there. Your midi file is the perfect remedy to this: it standardises the melodic line, but provides enough variation and complexity to be meaningful. It was perfect - don't lose me now! ;)

So, with your permission, I'm going to put this in the drawer labelled 'ideas for future tests', and not change the rules of this test, otherwise, I fear we'll never get off the ground.
I could imagine that often more than a single patch is needed to get good results.
I agree, but that is the answer to a different question I think. This test doesn't claim that using only 1 library will get you the best result - it just provides a basis to compare what each library individually is providing.
 
Last edited:

ka00

Senior Member
It’s smart to keep things simple. All I’m personally looking for is the same passages rendered with as many of the top end libraries as possible. That has inherent value whether it’s a secret which library is used or not.

Regardless, in the end, the appreciation of this will be more art than science.
 

Pianolando

Active Member
Cool thread!

I do believe that the "dry" libraries should be entered as dry, as having external reverb will add to many possibilities. Sure, the dry ones will sound dry, but that is not suffer in my book, everyone worth their salt will know how you can use dry libraries and how flexible that is.

Different volume on the recordings will be a problem, but maybe OP can take care of that?

I also think that the example in the OP sounds very good!
 
  • Like
Reactions: Vik
Top Bottom