# Alternative MIDI Controller Concept



## DaddyO (Oct 1, 2015)

We have breath controllers. I find that personally I just don't have the kind of breath control to get the results I want. What a shame. I would have made a terrible woodwind or brass player.

We have gesture controllers. Okay, sue me. I'm just not steady or fluid enough, nor am I accurate enough in my motions. I'm one of those people who can't dance. What a shame. I really wish I could.

But you know what I CAN do? I can sing, and I can whistle -- don't ask me why I can whistle but I can't successfully operate a breath controller, I simply don't know why. But I can both sing and whistle intuitively and expressively.

So is there a reason why a midi controller can't be developed that analyzes sonic input in various ways and produces useful MIDI data? The most expressive, most natural, most direct musical instrument of all is the human voice. If I sing into a microphone, all sorts of information about the resulting audio can be measured. Pitch. Amplitude. Volume. Timbre. Vibrato. And many more. And if you can't sing, you might be able to whistle or hum.

And there's no need for fancy input hardware. So far as I can conceive of it, it would be software only. You record audio in a track, insert a Vocal MIDI Controller plugin that does the analysis and offers preset data options from which data curves can be extracted and pasted into MIDI tracks.

I offer this as an idea in hopes that someone, somewhere with the desire and the chops to produce it will get it done. I'm sure there's something I'm missing about the concept, but it seems to me an instant winner. Where am I wrong?


----------



## DaddyO (Oct 1, 2015)

Just found this on YouTube, a guy who back in 2011 mentions extracting MIDI Controller data from audio amplitude in Abelton Live.



Makes me wonder if Cubase Vari Audio has some capability I didn't know about? Anyway, it confirms my concept.


----------



## DaddyO (Oct 1, 2015)

aesthete said:


> I am pretty excited about the Roli Rise and Roger Linn Linnstrument
> 
> https://www.roli.com/products/seaboard-rise
> 
> http://www.rogerlinndesign.com/linnstrument.html



I edited the topic title to make it singular, since my interest for this post is strictly the conversion of Audio to MIDI Controller data. Anyway, I understand that what you mention fits with alternative concepts, but neither of these would be any use at all to me. Thanks anyway.


----------



## DaddyO (Oct 1, 2015)

I found a thread from the Cubase forum last December that has a somewhat similar idea, though his specific interest seems different from mine. He uses VariAudio to produce controller data from audio sine waves.

http://www.steinberg.net/forums/viewtopic.php?f=228&t=70062

But the process seems too cumbersome to be practical, unless I misunderstand it.


----------



## DaddyO (Oct 1, 2015)

I posted the following on the Cubase Feature Request forum:

http://www.steinberg.net/forums/viewtopic.php?f=228&t=86229

Currently VariAudio can analyze an audio part and export MIDI pitch data.

The possibilities for expanding the analysis and data export are fantastic and deserve serious consideration. The audio analysis tools already exist in Cubase. What is needed is the programming to capture and convert the results into data from 0 to 127.

The most straightforward and easy example of how this might be used is for MIDI Continuous Controller data. I'm no audio expert, but Imagine if you could record a vocal into an audio track not for purposes of incorporating it into the finished product, but for purposes of generating MIDI CC data. You pick an easily measurable feature of audio like amplitude, then convert it in scale to a number between 0 and 127. You then export that data, and BOOM! You have something that can be assigned to any MIDI Continuous Controller, like Volume, or Breath Controller, or Expression. This allows you to control any parameter of any VST instrument using CC data controlled by the most natural, the most inuitive, the most expressive of instruments, the human voice. It is capable of far more than just pitch.

You would have a revolutionary MIDI controller. Why no one has done this is beyond me. It simply makes too much sense. I can control my voice a whole lot more than I can control a breath controller, or a MIDI fader or knob. Not only that, but there is immediate feedback. I am a VSL user. If I want to control the volume and expression of an orchestral instrument, I turn first to Velocity Cross-fading and use CC2. So instead of trying to record curves using some controller device, I simply sing the part I want to play, including the pitch of course, but also the dynamics. There's no reason I shouldn't then be able to export it to a MIDI track that captures very well the nuances of what I sang.

Other parameters could be captured and put to good use as well.

You could even sell it as a separate, revolutionary new plugin.


----------



## pmcrockett (Oct 2, 2015)

I played around with this idea a while back using a plugin that could convert audio amplitude to MIDI CC. I don't recall being particularly satisfied with the results, but I think that's probably more down to the specific plugin I was using and not the general concept. If I were to revisit the idea, I'd put something together in Reaktor, which would give a lot more control over the way the processing worked.


----------



## thebob (Oct 2, 2015)

You don't necessarily need to be accurate in your gestures and you don't need to know how to dance to use gesture controllers. 
You can walk, clap or bring a glass to your lips don't you ? It is just a question of how you design the relation between motion capture and what it does to your sounds. you could use quantities such as quantity of motion, acceleration, and many many things... depending on your needs. I used wiimote, and I use Hot hand USB, leap motion, and Myo. It is really _useful _for some aspect of music production. 


BUT what you want the most, according to your post, is that : *imitone*.com


----------



## pmountford (Oct 2, 2015)

When I was at university 25 years ago for my final year project I wrote a program what allowed you to do just this - sing into a microphone, record the wav, perform a little analysis to find the pitch content, produce MIDI data and notate it. Interesting project and it worked way back then.. to a degree. (no pun intended ). I thought I read a few years later about this being built into one of the DAW programs. So yes it's possible.


----------



## thebob (Oct 2, 2015)

Back then, there was actually a MICROPHONE that did the tricks. I don't know when it came out, but very early in midi history.


----------



## DaddyO (Oct 2, 2015)

Thanks for the comments, all.

Thanks for the link to imitone. The website says it's designed more as a toy, but it definitely captures the general idea of what I'm talking about. It will be interesting to follow.

My DAW software is Cubase, and it's VariAudio, along with third party plugins like Melodyne, ought to be developer-adaptable to such a purpose without too much trouble.


----------



## ch4rles (Oct 2, 2015)

One way to achieve this could possibly be to hook up a microphone to this expression pedal to USB MIDI converter

A simple analog interface would of course be needed between the mic and the converter to rectify the microphone output into a 0-5V DC signal. The converter then would transform it into USB MIDI and it can be configured to send any CC.


----------



## DaddyO (Oct 2, 2015)

ch4rles said:


> One way to achieve this could possibly be to hook up a microphone to this expression pedal to USB MIDI converter
> 
> A simple analog interface would of course be needed between the mic and the converter to rectify the microphone output into a 0-5V DC signal. The converter then would transform it into USB MIDI and it can be configured to send any CC.



Thanks for the heads up. I'll keep that in mind.


----------



## Udo (Oct 2, 2015)

There are some recent developments which likely could be adapted for midi & music control, e.g. this smart glove which translates sign language into speech: http://tinyurl.com/GestureTranslator


----------



## arwyn (Feb 7, 2017)

DaddyO said:


> But you know what I CAN do? I can sing, and I can whistle -- don't ask me why I can whistle but I can't successfully operate a breath controller, I simply don't know why. But I can both sing and whistle intuitively and expressively.



Having struggled myself inputting expression using a breath controller, I was thinking how cool it would be to be able to sing your phrases into a DAW and convert the dynamics/expression into cc data for each part. After a quick search I found this thread started by DaddyO and was wondering if there have been any further developments with this?
As DaddyO said "_The most expressive, most natural, most direct musical instrument of all is the human voice_".
Now I'm not technically minded by a long shot, but with all the plug ins and functions already available to us that do really clever things, wouldn't this be relatively straight forward in comparison to do? Pitches wouldn't have to be a concern (although we already have this capability) Just the expression etc.
I really think that if this were implemented (well) into any DAW (preferably Cubase as that's what I use at the moment!!) it could be a real game changer for expressive input.
If there is something already available or if anyone has some sort of work around for this, I'd love to know more about it.
Thanks


----------



## DaddyO (Feb 8, 2017)

I feel today exactly as I did when I started this thread.

So far as tools based on the idea of voice as MIDI controller, I am not aware that any significant strides have been taken to make this as easy as other MIDI controllers. My word, all it would take would be a microphone, an algorithm to interpret various sound qualities (and turn them into simultaneous MIDI note and controller values. It seems so obvious. I guess traditional molds are hard to break. With companies looking for every edge to separate their libraries from all the others, I find it baffling, especially for orchestral libraries, that no one has pursued this as a difference-maker.

Doggone I'd like to sing a cello line and have the program take care of pitch, tempo, dynamics, timbre, vibrato, articulations, etc. I'd like to do the same thing with violin lines, but have it capable of transposing what I sing in a range comfortable for me into the upper range of the instrument.

Somebody out there pay attention. I know Cubase has VariAudio, and that Melodyne can do some things. But so far as I know (it's been awhile since I dug into it) their ability to convert audio to MIDI is rudimentary. Someone correct me if I'm wrong.

Thanks arwyn for resurrecting this thread.


----------



## d.healey (Feb 9, 2017)

DaddyO said:


> I feel today exactly as I did when I started this thread.
> 
> So far as tools based on the idea of voice as MIDI controller, I am not aware that any significant strides have been taken to make this as easy as other MIDI controllers. My word, all it would take would be a microphone, an algorithm to interpret various sound qualities (and turn them into simultaneous MIDI note and controller values. It seems so obvious. I guess traditional molds are hard to break. With companies looking for every edge to separate their libraries from all the others, I find it baffling, especially for orchestral libraries, that no one has pursued this as a difference-maker.
> 
> ...


Imitone does this


----------



## DaddyO (Feb 9, 2017)

d.healey said:


> Imitone does this



Despite the fact that I started this thread, I had forgotten this product mentioned above. It looks from their website that they have progressed some.
We'll have to see how well the release version executes. It seems to me that if it succeeds this functionality will over time become a common tool in DAW work, either as a plugin or as an integral part of DAWs.

For those like me who weren't aware of it, here are some links for Imitone.

http://imitone.com/

https://www.facebook.com/search/top/?q=imitone

Here's a YouTube video showing someone (the creator?) trying to apply a 2014 pre-release version to orchestral composition:



Here's more YouTube by the same guy:


----------



## arwyn (Feb 10, 2017)

Thanks for the links DaddyO. Imitone looks interesting. I wonder if they are still developing this? I went to the website and it has a pre order option so hard to tell. Clips are a few years old now. I might try and get in touch to see where they are as regards to development.
The thing is I wouldn't need note recognition.It would purely be expression input like a breath controller or cc fader.
As far as I can tell from the clips it is capturing that through his voice. 
Maybe they could be persuaded to develop that side of things- maybe they already are.
Hopefully I'll get an answer and we'll see


----------



## DaddyO (Feb 10, 2017)

arwyn, this FB post, just over a month old, makes it sound like the are in the final stages of development and expect to release this year (they say "several months," but in software-development-speak, who knows when).



I do agree that it's been three years since the Kickstarter that allowed him to get it going, so it's anybody's guess how long and how well the product will be supported. I'd say his best bet will be to release a product that works well enough to draw purchasing interest from an established company.


----------



## arwyn (Feb 10, 2017)

Ah great thanks. I'm one of the few that doesn't do facebook. I emailed them with some questions.I'll see what they say and wait for a release.
Cheers


----------



## Deelanee Orchestra (Mar 13, 2019)

Hello there. We are interested in the same things. See this recent post of mine: https://vi-control.net/community/threads/vochlea-dubler.80448/


----------



## Deelanee Orchestra (Mar 13, 2019)

Also, my assumptions right now is that Imitone only detects pitch and doesn't look like a "playable" instrument. Vochlea Dubler is much more advanced in terms of playability, but as of today, I mostly see very simple pitch detection and excellent beatboxing to drums conversion.
https://www.vochlea.co.uk/


----------

