# Violin Bow MIDI Controller using the Apple Pencil



## Audio Motion (Mar 28, 2018)

Hi everyone

I always felt that we lack a device for controlling bow movement for strings that is as intuitive as a breath controller. This is why I wrote a new iOS App called Pen2Bow, a violin bow MIDI controller using the Apple Pencil on the iPad. It takes full advantage of the sensors in the Apple Pencil and generates 4 continuous MIDI controllers: velocity, force, tilt and orientation. These can be assigned for example to expression, vibrato, bow pressure and position on the bridge to drive a virtual violin - or to any other parameter to drive synthesizers.

But the key innovation is that it recognizes linear as well as circular pencil motion to generate bow movement from the user input. Circular motion enables the infinite extension of the bow that otherwise would be limited in length. This frees the user from constantly thinking about performing bow changes in order to prevent the bow from running out. I believe it is one of the very few, if not the only practical MIDI input device for bowed instruments.

For more information please watch "Pen2Bow: Violin Bow MIDI Controller using Apple Pencil on iPad" on youtube: 
I hope you will enjoy this new device.

Update:
There are various ways to connect the iPad to your computer and transmit the MIDI information. You can use a MIDI interface like e.g. "iConnectivity iConnectMIDI2" to connect through a cable or you can use a wireless MIDI connection over WiFi. To set this up please check this blog post: http://musicappblog.com/wireless-network-midi-ipad-to-mac/ (There is also a link at the top that shows how to achieve this using a pc). Please note, depending on the performance of your WiFi network this may add a few milliseconds of latency.

Kind regards
Dimos Gaidatzis
Switzerland


----------



## Garry (Mar 28, 2018)

This is incredible!!! What an amazing idea, and the perfect controller for SWAM instruments! I downloaded immediately, and can't wait to try it out, but please could you let me know how do I link the app to my computer as a midi controller?


----------



## devonmyles (Mar 28, 2018)

Amazing....Great stuff.


----------



## Audio Motion (Mar 28, 2018)

Garry said:


> This is incredible!!! What an amazing idea, and the perfect controller for SWAM instruments! I downloaded immediately, and can't wait to try it out, but please could you let me know how do I link the app to my computer as a midi controller?



Thanks a lot, I am glad you like the idea! Please check the update in the first post.


----------



## Silence-is-Golden (Mar 28, 2018)

This is making me considering swam violin again.
Interesting!

Btw: Apple just released a new ipad at a low price range that can also use the pencil


----------



## pmcrockett (Mar 28, 2018)

I don't have any iDevices so I can't try this out, but I've experimented with a homebrew system that works similarly to this (using a hacked drawing tablet), and I can confirm that the concept is really cool. I'm glad someone is taking the idea and making it commercially available.


----------



## Garry (Mar 29, 2018)

Hey, it works!!!! I set it up using the blog you mentioned, and I have 0ms latency! *Works perfectly!* 

In using the app, particularly with SWAM instruments, it really does add another layer of expression. I use multiple midi controllers to get the most out of these highly expressive instruments, but with the TEControl breath controller and the Korg Nanokontrol2, I'm not able to achieve the bowing action that you can get with Pen2Bow. The other controller that I use is Leap Motion, which uses hand gestures, and this does allow a similar bowing motion, but the Pen2Bow is definitely an interesting option that I will play with, and since it now DOES NOT require additional hardware (as the Leap Motion does, well it's not _additional_, it is itself hardware based), then Pen2Bow is considerably cheaper.

*Thank you Dimos*. I hadn't understood that this would work over wi-fi, but, *I was wrong - the wi-fi based MIDI connection *_(on my system at least, can't vouch for others) _*works extremely well *- I really wasn't expecting 0ms latency!

Minor downside: since you have to set up a local network, you are no longer connected to the internet. A minor point, and not the fault of the app of course, just the nature of the connection, but just some user feedback for others, in case this is relevant for them.

My only suggestion would be to add some description to your app about the wi-fi midi connection and setup instructions (or just the link you previously provided), and that if the latency on their specific system does not allow it to be used, then there is the additional hardware route as a failsafe (but more expensive) option. Then people will know what they are getting from the start. But if everyone has the same experience as me, there is absolutely no need for the additional hardware - the wi-fi connection is flawless.


----------



## Garry (Mar 29, 2018)

Silence-is-Golden said:


> This is making me considering swam violin again.
> Interesting!
> 
> Btw: Apple just released a new ipad at a low price range that can also use the pencil



Works GREAT with the SWAM instruments!


----------



## Audio Motion (Mar 29, 2018)

Garry said:


> Hey, it works!!!! I set it up using the blog you mentioned, and I have 0ms latency! *Works perfectly!*
> 
> ...



I am glad it works and that you enjoy using it. I added an update to the original post and on youtube to include the info on how to connect the iPad to a computer and will also update the App Store info as soon as possible. Thank you very much for your input.


----------



## lelepar (Mar 31, 2018)

I'd like to say thank you to @Audio Motion for Pen2Bow App. It is a cleaver and useful tool for controlling the SWAM Solo Strings with a natural gesture for the bowing gestures.
Note that this finally gives more sense to the "Bipolar" Bow Gesture we have introduced from the beginning, waiting for a suitable controller... now it has found its perfect match!

As for the connection between the iPad and macOS, now iOS11 and macOS Sierra and above support direct MIDI over USB.
Please watch this video showing how to control SWAM instruments from GeoShred. Starting at 00:38 it explains how to connect the iPad to macOS over USB: http://www.swamengine.com/2018/03/using-geoshred-with-swam/

BTW: we have recently released an update of all our SWAM products. Login to the Audio Modeling Customer Portal (https://my.audiomodeling.com) to get it.

Emanuele
_CTO
Audio Modeling_
_www.audiomodeling.com_


----------



## Piano Pete (Mar 31, 2018)

Pretty neat. 

This sort of problem is why I have stayed clear of many solo string libraries, and while I am always an advocate of recording real solo strings, I understand that not everyone has the resources to do this. A step forward towards better mockups!

Audio Motion, do you have any plans on making a port of this prepackaged for the roli environment? While the ipad ecosystem is great, not everyone is sold on them. I do not know what the hassle would be to do this, but it may expand your potential market. I guess my thought is that people on a budget may go for an mpe controller like a roli block over an ipad pro with an apple pen.

Nice job.


----------



## Audio Motion (Apr 1, 2018)

Piano Pete said:


> Pretty neat.
> 
> This sort of problem is why I have stayed clear of many solo string libraries, and while I am always an advocate of recording real solo strings, I understand that not everyone has the resources to do this. A step forward towards better mockups!
> 
> ...



Thanks a lot for your encouragement. I thought about many ways of achieving a bow controller with different available hardware. I experimented also with regular touch control, but I found that a finger touching a glass or plastic surface causes too much drag. It makes it virtually impossible to measure force and velocity simultaneously as the finger starts to jump on the surface. I came to the conclusion that only a device like a pencil with minimum drag allows for precise and smooth control of a virtual bow. Moreover, I don't see how to measure additional controllers like tilt and orientation as easily when using the finger. If a platform offers a tablet-like device with accurate stylus support I would definitely consider it as a potential option.


----------



## puremusic (Apr 5, 2018)

This is almost enough to make me buy an iPad. All I have is Windows 7 here.


----------



## Audio Motion (Apr 9, 2018)

Pen2Bow 1.0.1 update was released

updates:

- Added smoothing option for the pencil velocity. Pen2Bow is very sensitive and picks up the smallest fluctuations in pencil speed. When used with string instruments, this greatly adds to the realism. In the case of synths however, those micro fluctuations might be less desirable. Therefore now there is an option to dampen those fluctuations (see help).

- Support for iOS 11.0 and higher (before it was 11.2 or higher)

- Pen2Bow now produces force values from 1 to 127 when the pencil is attached to the surface. Before it was 0 to 127. Zero is now only emitted when the pencil is detached from the surface. This allows a downstream App to detect pencil attachments and detachments by reading the force continuous controller.


----------



## Audio Motion (Apr 22, 2018)

Thank you all for your interest and feedback! We just released update 1.0.2 that includes some of the features that users requested to make Pen2Bow more powerful when used with synths. Here is a video introducing what's new: 

Updates in detail:

- Added a settings page. It includes sensitivity and range adjustment for the 4 MIDI controllers.

- Added a touch piano keyboard. Virtual instruments can now be played without the need for an external MIDI keyboard.

- Added a refresh button for the MIDI device list. Pen2Bow no longer needs to be restarted in order to detect new MIDI devices.

- Added smoothing option for the pencil velocity. Pen2Bow is very sensitive and picks up the smallest fluctuations in pencil speed. When used with string instruments, this greatly adds to the realism. In the case of synths however, those micro fluctuations might be less desirable. Therefore now there is an option to dampen those fluctuations (version 1.0.1 and higher).


----------



## Audio Motion (Apr 27, 2018)

New Video is out! Part 1 of a 4-part series of videos, showing the interplay between the MIDI controller App Pen2Bow and the virtual string instruments by Audio Modeling. This part is about the SWAM Violin.


----------



## Audio Motion (May 12, 2018)

New video is out! Part 2 of a 4-part series of videos, showing the interplay between the MIDI controller App Pen2Bow and the virtual string instruments by Audio Modeling. This part is about the SWAM Viola.


----------



## Rodrig Vourot (May 12, 2018)

It is interesting to visualize the perspective of "realism" of this can generate, curiously it is what most concerned in the 'library' industry and 'samples' a realism in what refers in the joints, but finally we continue talking about the same is a sequence, program. To me at least anyway I find it very interesting.


----------



## Audio Motion (May 25, 2018)

Here is a new video: Part 3 of our 4-part series on the interplay between Pen2Bow and the virtual string instruments by Audio Modeling. This part is about the SWAM Cello. It also includes a comparison between a breath controller and Pen2Bow.


----------



## Audio Motion (Jul 7, 2018)

Last video of the series is out: Part 4 on the interplay between Pen2Bow and the virtual string instruments by Audio Modeling. This part is about the SWAM Double Bass. It also includes a summary of my three month experience with Pen2Bow and the SWAM string instruments:


----------



## Audio Motion (Jul 19, 2018)

Here is a live performance of the piece Sicilienne by M.T. Von Paradis using Pen2Bow and the SWAM Cello:


----------



## dflood (Jul 19, 2018)

Please watch this video showing how to control SWAM instruments from GeoShred. Starting at 00:38 it explains how to connect the iPad to macOS over USB: [URL said:


> http://www.swamengine.com/2018/03/using-geoshred-with-swam/[/URL]



This is working great for me. With the iPad Pro connected as an audo/midi device as per the above instructions there are no latency issues at all. Using the pencil to simultaneously manage bow changes and expression seems very natural and adds a much more authentic dimension to solo string performances with the SWAM instruments. Now I just need to work on my live performance technique...


----------



## Audio Motion (Jul 20, 2018)

Since the Leap Motion Controller (LMC) has come up multiple times, here are some thoughts on the similarities and differences between Pen2Bow and the LMC. The fundamental bowing principle is the same but when it comes to practicality there are several differences:

Hand movements in mid-air on the LMC can be taxing on the arm when used for more than just a few minutes. This is not an issue with the iPad as it can be placed at the level of the keyboard or even below. Attaching and detaching the bow with the LMC in mid-air is far more difficult due to the lack of a reference point. On the other hand, the iPad offers a surface to attach and detach the pencil. Engaging vibrato with a separate hand gesture or axis is difficult to achieve with the LMC. For that purpose, Pencil force is more intuitive. The hand recognition algorithm of the LMC is not perfect. In my experience it works well about 95% of the time. In the remaining 5% (every minute or so) however, it messes up heavily. The hand can disappear, the position of the hand can jump or a tremor can appear. Pencil detection on the iPad is far more reliable. To be honest, I did not experience many advantages of the LMC other than perhaps the absence of battery charging which is required for the Apple Pencil. And the iPad requires you to clean the surface once in a while, which is not the case with the LMC.


----------



## puremusic (Jul 20, 2018)

I think you meant you have to clean the surface of the LMC.


----------



## Audio Motion (Jul 20, 2018)

You are right @puremusic, apparently also the LMC needs to be cleaned! I didn't even know that. I didn't touch the front surface of mine so it was always clean, but if it becomes dirty indeed it needs to be cleaned too, good catch!


----------



## Audio Motion (Jan 29, 2019)

We just released a major update to Pen2Bow. In version 1.1, Pen2Bow can now be used as an AUv3 plugin within Audio Unit hosts. This allows you to open multiple instances of Pen2Bow, to wire each instance with a different instrument and to save and recall the parameters as part of a whole session. Watch the video for more details: 

Hope you enjoy the update!


----------



## rudi (Mar 31, 2019)

Amazing!!! So expressive


----------



## chimuelo (Apr 1, 2019)

I’m using Leap Motion but needed an excuse to upgrade from Air 2 iPad to Pro.
Now I’ve got one.


----------



## molemac (Apr 30, 2021)

chimuelo said:


> I’m using Leap Motion but needed an excuse to upgrade from Air 2 iPad to Pro.
> Now I’ve got one.





Audio Motion said:


> We just released a major update to Pen2Bow. In version 1.1, Pen2Bow can now be used as an AUv3 plugin within Audio Unit hosts. This allows you to open multiple instances of Pen2Bow, to wire each instance with a different instrument and to save and recall the parameters as part of a whole session. Watch the video for more details:
> 
> Hope you enjoy the update!



Dont suppose anyone has invented a midi violin that can do this ? ie play a midi violin that inputs the notes +possibly the vib as well as the bow movements ?


----------



## Markrs (Apr 30, 2021)

Wow, I wasn't aware of this app. I love using an app called musikraken which uses lots of detectors within the iPad but at this time it doesn't utilise the pencil like this app does. With some of the SWAM Instruments now on iOS, this could be very useful.


----------



## sekrit_studios (Sep 2, 2021)

pmcrockett said:


> I don't have any iDevices so I can't try this out, but I've experimented with a homebrew system that works similarly to this (using a hacked drawing tablet), and I can confirm that the concept is really cool. I'm glad someone is taking the idea and making it commercially available.


I'm hoping desperately that by now, something is available right now in September 2021 for PC/Android?


----------



## terminalgarden (Oct 6, 2021)

Has this application been tested on the very newest iOS, running on a M1 ipad pro?


----------



## Jan-Erik (Oct 30, 2022)

Hi, I've downloaded the app. Eager to try out out! I'm trying to connect my Ipad to my Macbook. All the links on how to do it seems to lead nowhere. Would be very happy if someone could help me out.


----------



## branshen (Oct 31, 2022)

Jan-Erik said:


> Hi, I've downloaded the app. Eager to try out out! I'm trying to connect my Ipad to my Macbook. All the links on how to do it seems to lead nowhere. Would be very happy if someone could help me out.


Thanks for bumping this thread! I just found out about this really cool app. are you using it with SWAM v3?


----------



## Vebjörn (Nov 2, 2022)

Audio Motion said:


> We just released a major update to Pen2Bow. In version 1.1, Pen2Bow can now be used as an AUv3 plugin within Audio Unit hosts. This allows you to open multiple instances of Pen2Bow, to wire each instance with a different instrument and to save and recall the parameters as part of a whole session. Watch the video for more details:
> 
> Hope you enjoy the update!



Wow, great job. I guess you have a background in programming? I'm curious how to go about learning to create apps like this.


----------



## JacksonTree (Nov 3, 2022)

This is cool! Is there anything similar for a Wacom?


----------



## timbit2006 (Nov 3, 2022)

JacksonTree said:


> This is cool! Is there anything similar for a Wacom?





L i v e l a b . d k - Products - Tablet 2 MIDI


I dont know how well it works but I have a Wacom tablet with the pen broken and when I replace it I intend to use this


----------



## JacksonTree (Nov 3, 2022)

Nice! I’ll have to try that out!


----------



## AudioXpression (Nov 4, 2022)

Interesting!!!
Very good work. 
Intuitive, practical and useful.


----------



## nibor (Yesterday at 8:00 AM)

Hi there Pen2Bow fans. I'm having trouble getting my daw to recognize pen2bow. I have followed all of the tutorials and am using RTPmidi. But no matter what I do I can't get Studio One to recognize RTPmidi. I have tried to geet more information from rtpmidi's developer but he's not responded. Pen2Bow seems like a simple, elegant and musical solution to making physical modeled instruments so much more expressive. 

I'm on Windows 11 with a ipad pro and running Studio One. 

Any help would be much appreciated.


----------

