Menu
What's new
By:
Filters

The vi-c blinded violins shootout - FINAL UPDATE.

@ModalRealist. Also, just a quick answer to your question about science... statistics sit in that odd space between a science and the arts. Normally I wouldn't say they are in and of themselves a science though. You can measure a huge number of things in a scientific way, but the things you are measuring don't have to be easy concepts. In other words, what you are measuring can be people's thoughts, ideas, opinion, emotion, etc., and it can still be measured scientifically.
 
@markleake, and my concern was that you misconstrued what was valuable in the methodology used here. You seem to think we can get a statistical measurement of the libraries qualities, whereas all that's being measured here are expressed responses of individuals' judgments about those qualities. Since the value of a library to a composer (and to the people listening to the composer's work) is ultimately in the gestalt of the finished mockup,* and since the experience of listening to a mockup (or any music) is one in which by default the various qualities are heard as a unified whole, there's no problem in measuring peoples' judgment of, and measuring it in terms of, that gestalt impression.

*Speed and ease of use, etc., are also obviously factors, but not ones that can be investigated through a listening test, blind or otherwise.

Your core claim was that the range of phrases and the indeterminate qualities being assessed:
markleake said:
introduce a whole raft of measurement issues that confound the result.
Click to expand...
But this just isn't the case, unless you are trying to measure the individual aspects. Compare a case in which we are examining oil paintings rather than sample libraries. If we asked "which is the best painting?" it's obvious that the results wouldn't tell us (for example) which painting had the best colours. Similarly, this test won't tell us which example people think has the best rendition of fast passages. The weighting that participants give to colour/fast passages is unknown, as is how they judge what makes for good colour/fast material. But that's part of the point of a test like this: we're not measuring against a standard set down by an individual or a committee, but rather allowing listeners to respond idiosyncratically to the material they hear. At the end of the day, the reason we're blinding here is to remove bias on the listeners' perspective. The double-blinding is borderline superfluous (and if you think it isn't, spell out exactly why, and exactly what the double blind has added).

markleake said:
@ModalRealist. Also, just a quick answer to your question about science... statistics sit in that odd space between a science and the arts. Normally I wouldn't say they are in and of themselves a science though. You can measure a huge number of things in a scientific way, but the things you are measuring don't have to be easy concepts. In other words, what you are measuring can be people's thoughts, ideas, opinion, emotion, etc., and it can still be measured scientifically.
Click to expand...

I didn't ask any questions about science or the status of statistics. I asked a largely rhetorical question about what this exercise is trying to measure: namely, the subjective opinions of VI-C users on what sounds best when it comes to violin recordings (or more precisely, faked violin recordings). What we'll have is objective data about these subjective judgments. There's nothing wrong with the test format for this purpose. Thus, my tongue-in-cheek point that your criticism ultimately targets what we're measuring, and not really the method of measurement. Now, I haven't assumed that you aren't an emeritus professor of statistics; equally, perhaps you shouldn't assume that I need your gestural account of "where" statistics is, or so on. Even moreso if you're not going to distinguish between measuring "people's thoughts" and measuring "people's reports of what they are thinking."

Anyway, I maintain that there's nothing deeply flawed at all in Garry's exercise here. To the contrary, there's more effort than is really necessary being put in, given the aims that were set out. If there's any sticking point of note, it's the volume of submissions, not the passage being listened to or the qualities being judged. A score-based voting system (as opposed to a top 5/10/n system) almost entirely bypasses this issue. If randomised presentation of the entries is impractical (which it surely is) the simplest solution to "too many to listen to" is for Garry to arbitrarily divide the master set into subsets with roughly equal representation of given libraries in each set, and for these to be released as separate batches for scoring. But frankly, ifthe results (average score) can be presented alongside the raw number of responses (not scores, just quantity of responses per rendition) one could pretty safely ignore the issue and just release the whole lot as one batch.
 
@ModalRealist. Well, all that is fine and dandy. But I feel you are a bit defensive here? I think you miss my points by getting lost in the detail. To simply what I am saying further...

The end result may not mean all that much, because it is effectively mashing together lots of different personal tastes about lots of different/competing factors. (Which is fine, BTW, as long as people know.) It's the old question of... OK, so it's the best, but the best at what?

And testing all submissions as one big lot is probably testing people's concentration or willingness to participate as much as anything else.
 
I think we actually get a great feel for the usability of a library of there's more than one entry and they all sound great.
 
** UPDATE **

An update on the ‘competition’. Given all the issues lately that have been raised on VIC, I’ve decided not to proceed personally with the competition, but do want to make sure that the files that were promised are made available for everyone’s use, as originally intended. I also want to ensure there is an opportunity for others to take this over, if they choose to do so. I’ll explain my reasons below, but before that, I’ll outline how I plan to proceed that enables people to continue with the competition, or just to receive the files, as they choose.

PLAN:
  • Tomorrow, I’ll release a Logic file, containing all 78 tracks that I received. These will not be normalised or perfectly lined up in time, but they WILL be blinded, and this should suffice for people's own personal review (you all have volume faders!). I can only release as a Logic file, as that’s the only DAW I have. I’m sure others who have more than 1 DAW will be happy to save in other formats, so that it’s available to everyone. If after this, there are still people whose needs cannot be met, I will make available all of the raw mp3 files (the raw files will be UNblinded, hence the reason for doing this later).
  • If there are people who are particularly keen for the competition/blind voting part of this to go ahead, please contact me, and I will send the Logic file only to you; you can then go ahead with organising the remainder of the competition, with whatever rules/additional criteria you feel would be best.
  • Assuming no one identifies as wanting to take over the competition/voting, etc part, then the day after tomorrow, I’ll release the unblinding. Everyone can then choose to review the files blinded if you wish, or unblinded - whatever you prefer.

REASONS:
  • For me, the competition, with all its flaws, was the least important part of this. The ‘competition’ was just fun to motivate engagement.
  • What was meaningful, was to have as many as possible of the major libraries used to produce a diverse set of melodic lines, that gave the possibility to compare them directly, that gave a meaningful, if imperfect, means for comparison, stripped of prior biases. For this purpose, it matters only what criteria YOU personally are prioritising as important, and it doesn't matter how or if this lines up with the choices others make. We achieved that, and I hope this is useful to people, now and in the future. It can easily be improved: if some people feel a library wasn’t well represented by the entries submitted, there is nothing to prevent anyone, (ideally adhering to the same rules), from sharing a new and better version with this community.
  • I expect this competition to be contentious. People will have their own expectations, biases, requirements, and some members will express these productively. Others will not, and will instead blame everyone involved, will focus on any detail they consider to render the whole process meaningless, or will jump on tribal loyalties from previous battles that have little to do with the ‘competition’. Whilst I would like to think that the competition would stimulate open, transparent discussion, this is the internet, and whilst this forum is no worse than others for flame-throwing (and considerably better than most), the likelihood of it degenerating is high, and I’ve decided that personally, I don’t wish to participate in this.
  • I think this forum needs a contentious discussion right now like it needs Trump to take over as moderator!

THANK YOU

To all those that contributed, in terms of productive discussion of ideas, submission of entries, and general support and encouragement, thank you. I think we have collectively produced something that is useful to the community. I hope it can be the start of something the community can improve upon. Indeed, to those who submitted criticisms, there is now an opportunity to actively redress any of the problems you identified, and act upon them by doing it better. If you are able to do it better, and go on to do so, then I reserve the biggest ‘thank you’ to you.

Peace.
 
Last edited:
Oops... sorry folks and Garry. I didn't intend that at all, if it was me. I'll hang my head in shame and retreat into the corner. :sick:
 
An unfortunate but understandable quasi-end to a fun idea. Hopefully someone will decide to pick up the ball and jog with it. It might be interesting to have the mp3s posted unblinded so we could hear them and make our opinions with maybe less controversy than voting.
 
As a newcomer to both this forum and the world of composing with samples/composing to picture, I'm bummed that this has fizzled out.

However, I completely understand why it's taken this new direction. I was less interested in a "competition" and more interested in learning the sound of different libraries in context.

I'm still very excited for the Logic file. I own very few libraries and I think a blinded test will prove beneficial for both me personally/educationally as well as my wallet.

Thanks for putting this together, @Garry!
 
As a passive observer I agree with you @boxheadboy50. Reading this thread and also looking through all of the other things going on in tandem in the forum has lead me to decide on remaining quiet and just check in every once in a while.

Thanks @Garry for the time you've put into this so far. Too bad the bickering and nitpicking took a nice idea and sucked all the fun out of it.
 
I do feel partly responsible for wearing Gary down, given my comments further above. Maybe? I have to read between the lines on his post to draw that conclusion. After re-reading the thread, I can't see that anyone said anything very controversial. There were some early argumentative posts, but they were removed, so that looks to be all sorted. The rest to me just looks like a friendly exchange of ideas. Of course there's the recent drama threads, but that stuff didn't seem to creep into this thread, so I don't think @rsampaio that you should feel intimidated into not participating, it's just normal forum noise. Anywho... apologies again if I contributed to derailing the thread by suggesting that the ranking of libraries part would be a bit "meaningless" to some people, to quote Garry. I tried my hardest to say I think the exercise itself is NOT meaningless.

I'm very happy to help also. I have limited experience with this stuff, but I'm at least able to help balance the stems, group them into logical comparisons, etc.

+1 to exporting to OMF also.
 
Last edited:
Seems I should clarify my reasoning around the change of direction. Whilst recent comments of how this "just seems like it won't give much useful information to anyone" were mildly irritating coming at this late stage, this wasn't the cause; to my mind, @ModalRealist had already effectively addressed the concerns raised, and needed no further comment from me.

I was disturbed by the recent events on the other thread with Daniel, in which I was directly involved. I completely stand by my comments in discussion with him, and felt that doing what I could to prevent rumours starting against a developer whose contributions to the community I valued, was a worthy cause to try to defend. However, the impact on him and how he responded should be upsetting to anyone, but particularly those directly involved, and it certainly was to me. I'm extremely glad he's received the show of support from the community that he has since, and it was good to see people unite behind this; a difficult situation which I thought @Mike Greene handled well. But the outcome on Daniel still shook me up, and as a result also made me reflect on this thread, and to consider how contentious the results of the competition were likely to be, however well intentioned, and the potential implications of that.

Inevitably, some people would not have been happy with the results once revealed, and at that point, I have no doubt how quickly things could degenerate. Not just because this is the internet and so this is invariably how things go, but because Daniel's situation served as a reminder that there are lives behind these posts; that there can be real impacts, and that our actions, however well-intentioned and individually justifiable, can have consequences beyond what is apparent to us as forum contributors. The potential implications of a developer coming at or near the bottom of any user comparison list is real, and has the potential for tangible consequences in the real world. Given the experience with Daniel (again, I stand by my comments with him, but that doesn't mean I don't regret the impact on him), I decided I don't want to be responsible for lighting the flame that leads to further drama here, but more importantly, further consequences that I can't predict. This was not the purpose of the competition. I find the current situation with developers perplexingly lopsided: for the most part, customers cannot experience the product before purchase, have to rely on vicarious demos, and cannot return or resale afterwards products that can accumulate to thousands of dollars. This was an attempt to redress that balance a little: so that we the customers are armed with a little more information prior to purchase. But on reflection, my feeling is that we can do that, without the unintended effect being to single out individual products, and the stress this could cause those developers and their employees, that recent events sensitised me to.

So, it was with this in mind that I opted to discontinue the competition and voting, and simply provide the information that was collated, so that users can choose to use it, or ignore it, on an individual basis. I felt this was the best way to be true to the initial intention, and the interest and support that was shown by the community in this effort, whilst avoiding any negative consequences, as much as possible. After posting the files, I plan to take a long break from the forum, as I was truly shaken by this recent experience.

Finally, with regards to practical questions about the format, my (limited) understanding is that OMF is not supported by Logic Pro X (according to this), and would anyway lose track names (according to this). I don't see OMF as on option in the export options in my version of Logic Pro X, so, I'll plan to share the Logic file as well as the AAF file. Again, if this isn't enough, I'll later share the raw files, but since these are unblinded, I'll do this after releasing the blinded information, so that those who wish to, can still review the files blinded.

Hope this clarifies.
 
Last edited:
Doesn't a Logic project include a folder with plain audio files?
Where is the problem to import that audio in any other DAW?
The files just have to be named correctly like 01, 02...

Thanks Garry for investing so much time on this.

Chris Hein
 
You must log in or register to reply here.
Top Bottom