How about resynthesis sampler for acoustic libraries?

[Dewdman42]

Harmor has resynthesis but I don't know if I would say the result is indistinguishable. its a start towards something you can mangle as a synth really.

I wrote a magazine article about resyntheis for a high tech music store I was working for around 1990 and have been waiting for tech to catch up to maybe do it someday ever since..but the simple truth is the resynthesis of acoustic instruments would take orders of magitude more data and CPU demands then samples..for a marginal return really. In 1990 it seemed more interesting because sample technology was nowhere close to what it is today. Today I think sample tech is quite good...to move beyond that with resynthesis will require a lot more computing resources then people realize.

For example, for an acoustic instrument, has a timbre that changes over time. If you use additive synthesis, which is probably the most "generic" way to resynthesize, then it could take dozens or maybe hundreds of partials to create the resynthesized sound...but if that sound changes over time, then you'd be talking about dozens or hundreds of data points...potentially on a per rendered sample basis...with way more data and cpu demand then samples. Might not be necessary o add up partials once per rendered sample, might be able to use interpolation and other tricks to spread out the partial intervals, that end up rendering sample rate output...... Hard to say, but to do this effectively to sound like real instruments, you still basically would be sampling the input to get this same kind of data...jsut to resyntheize it again...and not really save anything in terms of hardware demands...

So what is the point? Good question, and probably why nobody has done it. The only advantage I can think of is the potential to modify in real time the way the partials are combined, for less of a "sampled" sound...however, the reality is that this would not be a very intuitive process to figure out how to do that exactly... possibly...would probably lead to unrealistic acoustic instrument results.

In my opinion when you look at other kinds of modeling like Pianoteq and other things..they are doing similar things, but rather then "generically" resynthesizing partials, they use very focused algorithms which can be optimized for the particular instrument they are modeling. Thus we see many products already which can do that and are getting better every day.

 

[lychee]

Harmor has resynthesis but I don't know if I would say the result is indistinguishable. its a start towards something you can mangle as a synth really.

I wrote a magazine article about resyntheis for a high tech music store I was working for around 1990 and have been waiting for tech to catch up to maybe do it someday ever since..but the simple truth is the resynthesis of acoustic instruments would take orders of magitude more data and CPU demands then samples..for a marginal return really. In 1990 it seemed more interesting because sample technology was nowhere close to what it is today. Today I think sample tech is quite good...to move beyond that with resynthesis will require a lot more computing resources then people realize.

For example, for an acoustic instrument, has a timbre that changes over time. If you use additive synthesis, which is probably the most "generic" way to resynthesize, then it could take dozens or maybe hundreds of partials to create the resynthesized sound...but if that sound changes over time, then you'd be talking about dozens or hundreds of data points...potentially on a per rendered sample basis...with way more data and cpu demand then samples. Might not be necessary o add up partials once per rendered sample, might be able to use interpolation and other tricks to spread out the partial intervals, that end up rendering sample rate output...... Hard to say, but to do this effectively to sound like real instruments, you still basically would be sampling the input to get this same kind of data...jsut to resyntheize it again...and not really save anything in terms of hardware demands...

So what is the point? Good question, and probably why nobody has done it. The only advantage I can think of is the potential to modify in real time the way the partials are combined, for less of a "sampled" sound...however, the reality is that this would not be a very intuitive process to figure out how to do that exactly... possibly...would probably lead to unrealistic acoustic instrument results.

In my opinion when you look at other kinds of modeling like Pianoteq and other things..they are doing similar things, but rather then "generically" resynthesizing partials, they use very focused algorithms which can be optimized for the particular instrument they are modeling. Thus we see many products already which can do that and are getting better every day.

I cited 3 examples of resynthesis-based instruments earlier in this thread, which consume little computer resources (Wivi, Synthful, and Sampleson).
The first two are old and even if they sound correct for my taste it is true that it is not perfect, but the technology or the tools to be able to create them have surely evolved since.
Sampleson's instruments are newer and of good quality (to my ears).
All this to say that some did not stop at the disadvantages of this technology but tried to circumvent them with more or less success.

We can make libraries with lots of stacked high definition samples, but that does not make them ultra realistic playing instruments.
We have to evolve the sample towards something more modern, maybe resynthesis is not the best approach, but I remain convinced that the current one is not either.

This is apparently what 8dio understood with Soundpaint, they have apparently recreated a new audio format that gets rid of the inconveniences of our old samples of yesteryear.
It only remains to wait to see what it will give.

 

[Dewdman42]

I cited 3 examples of resynthesis-based instruments earlier in this thread, which consume little computer resources (Wivi, Synthful, and Sampleson).
The first two are old and even if they sound correct for my taste it is true that it is not perfect, but the technology or the tools to be able to create them have surely evolved since.
Sampleson's instruments are newer and of good quality (to my ears).
All this to say that some did not stop at the disadvantages of this technology but tried to circumvent them with more or less success.

We can make libraries with lots of stacked high definition samples, but that does not make them ultra realistic playing instruments.
We have to evolve the sample towards something more modern, maybe resynthesis is not the best approach, but I remain convinced that the current one is not either.

This is apparently what 8dio understood with Soundpaint, they have apparently recreated a new audio format that gets rid of the inconveniences of our old samples of yesteryear.
It only remains to wait to see what it will give.

Technically those are not “resymthesis”Instruments, they would most likely be categorized as “modeled” which I made reference to in my last paragraph.

Harmor, for example, is a synth capable of resynthesis. This means you can feed it a sample, it can analyze this sample and break it down into its partials, and then re-synthesize it by building up the replication of the sample using partials, at least an approximation.

 

[muratkayi]

I know this is not exactly adhering to OPs aim of automatically analysing audio into additive synthesis (and also, I find the discussion highly informative)...

...but if you forego the initial idea for a second you might find the sounds of both Razor and Prism really intriguing. Razor is a synth truly devoted to additive synthesis as each effect/voice/modulation is completely built on manipulating count and shape of the sines added to the signal, be it filters or reverb.
Prism can spit out pseudo-acoustic instruments which keytrack nicely (which is nice for influencing formants).
Now, this is not resynthesis, but it might be inspiring on your journey.

Apart from that: Reaktor features a pre-programmed sine bank for additive synthesis. IIRC, Benn Jordan used it on one of his earlier videos on Pianoteq and modelling to demonstrate how to model the sound of a harmonica in Reaktor based on a previous analysis of its frequency spectrum.

 

[lychee]

I know this is not exactly adhering to OPs aim of automatically analysing audio into additive synthesis (and also, I find the discussion highly informative)...

...but if you forego the initial idea for a second you might find the sounds of both Razor and Prism really intriguing. Razor is a synth truly devoted to additive synthesis as each effect/voice/modulation is completely built on manipulating count and shape of the sines added to the signal, be it filters or reverb.
Prism can spit out pseudo-acoustic instruments which keytrack nicely (which is nice for influencing formants).
Now, this is not resynthesis, but it might be inspiring on your journey.

Apart from that: Reaktor features a pre-programmed sine bank for additive synthesis. IIRC, Benn Jordan used it on one of his earlier videos on Pianoteq and modelling to demonstrate how to model the sound of a harmonica in Reaktor based on a previous analysis of its frequency spectrum.

Thank you for these suggestions muratkayi, and sorry for the late reply.

I know Reaktor Prism and also Steampipe2 which seem to be excellent synths dedicated to acoustic type of sounds.
Strangely, I never used these instruments (which I own), probably because of the fact that at the time I didn't have enough machine to support Reaktor's resource gluttony.

For Razor, not being presented as a potential "acoustic" instrument, and not knowing the possibility of recreating this type of sound with synthesis at the time, I completely missed this plugin.

I also saw Benn Jordan's video you mention, it makes me want to see how to create my own instruments using this technique.
I also created another thread to find out what would be the best tool for this:

Physical modeling instrument: do it yourself?​

But in the end what I want is a tool that directly captures the sound source as a sampler rather than starting from approximations and then modifying the whole thing to try to imitate the desired sound, such as a Prism, Razor, Pigment...

So I kept looking for synths capable of additive resynthetization.
Candidates like Vertigo4, Cube2, Loom and Icarus2 came to me.
It remains for me to know if these candidates are capable of fulfilling my desires.
Maybe one of you has the answer?

 

[lychee]

Bee_Abney and jesussaddle

You had a long and interesting discussion (nearly off-topic ) among other things about spectral morphing and additive resynthesis on the thread about a Soundmagic product.

Knowing that you have some knowledge on the subject, maybe you can help me in my quest?

 

[Jrettetsoh]

Reakto’s Razor can be reprogrammed in Reaktor. It has 2 oscillators but one can be switched to a vocoder—-

Can one of you mod Razor so that both osc slots are vocoders, so you can vocode an input with itself?

This would allow applying all the cool additive effects to it.

 

[Jrettetsoh]

Also consider SpecOps.

 

[jblongz]

Synful orchestra was the best demonstration of modeling memes instruments of its time. I’m not sure why they discontinued, but their patent is still on file at the USPTO. Perhaps someone with deep pockets can acquire and revive such a promising technology.

 

[lychee]

I just discovered Kyma X, which made me want to dig out my old thread.
Damn, the sampling industry really needs to move towards an "elastic" format that would allow this kind of thing and revolutionize our plugins.

 

[Bruhelius]

I found that each additive, physical modeling or sample modeling platform had its own weaknesses, so by combining them together i was able to achieve a sound that I am much happier with, mostly with respect to the sound of orchestral string ensembles. The key is to build a really strong control layer stack that amplifies the strengths of each respective platform.

Recently, I compared the combined modeling sound to a very reknown sample library whose name I won’t mention here. The difference was striking. I was shocked how much the library sounded like a bunch of recordings poorly strung together. And i was really riding all the controllers pretty hard. I refuse to use any sample libraries for orchestral instruments…it just sounds very uncomfortable to my ears. I hear all the crossfades and the poor vibrato LFOs. Anne-Katrin Dern recently posted a video about this, and nailed it IMHO.
The industry simply must move to hybrid sample/modeling tech, because it is the only tech which is truly timeless, especially if you combine it with evolving tech like impulse response morphing. Any thoughts on this actually?

 

[drjee]

My guess is that the V Series of Acousticsamples (at least to some extent) relies on resynthesis. Otherwise the low size footprint is hard to explainable and one can adjust harmonics to shape the sound.

 

[quietmind]

I'm late to this thread.... somehow missed. But I have dreamt of resynthesis taking more of a role in VIs for 50 years. I did some early work at Stanford in the 70's on analysis/resynthesis with my colleague Andy Moorer, based in additive synthesis. I sampled a set of orchestral instruments. The potential power of the approach was clear: the ability to recreate complex natural sounds but being able to manipulate the analyzed data before resynthesis.

I did some psychoacoustic studies on how much data reduction you could get away with in the analyzed data that would be used to resynthesize. Here's an example... the figures on the left are the initial complete analysis, from which resynthesis would create an identical waveform to the original. The figures on the right represent highly reduced data for resynthesis, and they produced indistinguishable sounding results, up to a certain limit of simplification. But back then it was clear that the amount of data reduction one could do could take the data needed down significantly.



Another thing I played with briefly was morphing between the instruments by interpolating the respective harmonics by degree from one instrument to another. I just recently ran across some ancient sound files digitized from old tapes... and know that even digital back then was 12-bit, 25K SRate... so this is embarrassingly the total epitome of lo-fi, plenty of distortion, high frequency degradation (50 year old tape source), and tape warble. Nonetheless, you might get some feel for what's possible were one to pursue this approach with today's tech. This example morphs I believe between horn, clarinet, oboe, violin and back to horn (or trombone?)...

View attachment Morph1974.mp3

The power of the analysis/re-additive-synthesis approach I still hope could be realized. Knowing a bit about the difficulties of large scale implementation, it will probably come down to a commercial AI based approach. I recently posted a bit more about some of the power of this approach in offering high quality vibrato control over instruments that also have the sonic quality of samples here:

Peteris Vasks Strings vs. Berlin First Chairs (vs. Others) for Songwriting

Sure, ideally they would have done both. That is why I am considering JB violin as a potentially valuable addition to my tool box. But I prefer the Cremona library the way it is (completely controllable vibrato and dynamic layer blending without phasing) rather than having another limited...

vi-control.net

 

[d.healey]

Since the original post Synful has been released for free and is due to become open source too, so that's good.

HISE now includes FAUST which is great for physical modelling and additive synthesis, as well as lots of other purposes.

HISE also includes Loris which is a fantastic re-synthesis toolkit. I made a video about the Loris command-line tools, but the interface in HISE is much more practical for building VIs.

 

[d.healey]

I should also mention this free plugin, Paraphrasis, which uses Loris and is fun to play with - https://virtualanalogy.com/

And of course keymap pro

 

[lychee]

What I don't understand is that today we're still at the same point, whereas Kyma X, Redmatica Keymap, or worse, quietmind's work is not new.
The only plugin I know that is innovating in this direction today is Tomofon.
In view of all the examples above, I remain convinced that our current tools are capable of this revolution.

 

[jbuhler]

I'm late to this thread.... somehow missed. But I have dreamt of resynthesis taking more of a role in VIs for 50 years. I did some early work at Stanford in the 70's on analysis/resynthesis with my colleague Andy Moorer, based in additive synthesis. I sampled a set of orchestral instruments. The potential power of the approach was clear: the ability to recreate complex natural sounds but being able to manipulate the analyzed data before resynthesis.

I did some psychoacoustic studies on how much data reduction you could get away with in the analyzed data that would be used to resynthesize. Here's an example... the figures on the left are the initial complete analysis, from which resynthesis would create an identical waveform to the original. The figures on the right represent highly reduced data for resynthesis, and they produced indistinguishable sounding results, up to a certain limit of simplification. But back then it was clear that the amount of data reduction one could do could take the data needed down significantly.



Another thing I played with briefly was morphing between the instruments by interpolating the respective harmonics by degree from one instrument to another. I just recently ran across some ancient sound files digitized from old tapes... and know that even digital back then was 12-bit, 25K SRate... so this is embarrassingly the total epitome of lo-fi, plenty of distortion, high frequency degradation (50 year old tape source), and tape warble. Nonetheless, you might get some feel for what's possible were one to pursue this approach with today's tech. This example morphs I believe between horn, clarinet, oboe, violin and back to horn (or trombone?)...

View attachment Morph1974.mp3

The power of the analysis/re-additive-synthesis approach I still hope could be realized. Knowing a bit about the difficulties of large scale implementation, it will probably come down to a commercial AI based approach. I recently posted a bit more about some of the power of this approach in offering high quality vibrato control over instruments that also have the sonic quality of samples here:

Peteris Vasks Strings vs. Berlin First Chairs (vs. Others) for Songwriting

Sure, ideally they would have done both. That is why I am considering JB violin as a potentially valuable addition to my tool box. But I prefer the Cremona library the way it is (completely controllable vibrato and dynamic layer blending without phasing) rather than having another limited...

vi-control.net

This is a very neat example. It sounds like you were involved with some interesting research back in the day. I imagine there’s quite a lot of good stuff that was uncovered back then but then lost because the state of the tech at the time made it impractical.

 

[timprebble]

What I don't understand is that today we're still at the same point, whereas Kyma X, Redmatica Keymap, or worse, quietmind's work is not new.
The only plugin I know that is innovating in this direction today is Tomofon.
In view of all the examples above, I remain convinced that our current tools are capable of this revolution.

I owned a Kyma system for a few years, and analysis and resynthesis was one of the reasons I bought it. KYMA is both a software environment and a hardware platform, and while it was capable of resynthesis at no point did I hear an acoustic sound from it. Everything it created had a digital grain or pallor, like it was not high enough resolution to convince my ear that what it synthesised had the complexity and continuity of an analog sample. (I felt the same about the plugin Iris. It never transcended its own processing)

The only conclusion I came to is that it doesn't matter how much you spend, nothing can recreate the complexity of the real world. And it seems a high resolution recording of an acoustic sound is as good as we can get. These tools are better purposed for sounds that can't or don't exist.

 

[richhickey]

I think one of the challenges is that there is a huge leap from using something like Loris to fiddle with a sample to getting a functioning full-range emulative virtual instrument with expressive vibrato control, varied articulations, proper formants etc. Also things like legato are out of scope of the analysis of individual samples.

A second challenge is that most of these models and algorithms are best at dealing with and morphing the sustain portion of a sound (where they garner praise for their "expressiveness"), but have trouble modeling/capturing and retaining (never mind morphing) the critical onset part of the sound, from which a lot of our sense of what the sound is derives.

Every time I go back from SM/AM/VH/Infinite to samples I detect the enormous lack of nuance in the onsets of the (sample-)models vs that captured by actual samples. And the ramping etc techniques they use for faking it are totally inadequate and weird sounding IMO.

Thus I'm not sure yet the tech fully exists. But I wish it did

 

[David Cuny]

A second challenge is that most of these models and algorithms are best at dealing with and morphing the sustain portion of a sound (where they garner praise for their "expressiveness"), but have trouble modeling/capturing and retaining (never mind morphing) the critical onset part of the sound, from which a lot of our sense of what the sound is derives.

The Kawai's K1 is a good example of that. It used 8-bit PCM sampled sounds for the instrument's attack, with the rest of the sound supplied with additive synthesis.