# Composing for video games - adaptive vs edited



## Daniel James (Nov 15, 2013)

Hey guys, 

I have been think and talking about this subject a lot recently and was curious to hear other peoples opinion on the subject.

Composing a video games is a unique and challenging form of composition which generally fall into one of two categories either at adaptive music system where the music adapts to what's going on on-screen or an edited music system where the composer composes a full track which is then edited by the sound designers and sound team in the studio. 

I've always thought that as composing is an art or I should say more of an art than it is a science, and to me adaptive music systems, with all its technical restrictions makes composition end up being more of a science based piece of work as opposed to an artistic one. Of course one can be artistic when writing, to a certain degree however I feel like the artistic element to the music suffers a little as you are filling the composition with more of the scientific technical aspects.

Edited music pieces on the other hand I feel make for much more artistically developed pieces of music. The composer writes an extended albums worth of music that suits the tones and situations which is then edited to the picture, now this may not always be the tightest 'cut to edit' way of putting the music in bit I think that having the correct tone spoken in a really musical voice helps more than the adaptive system which hits the edits better.

I can think of a few examples where I felt edited music showed its power... Infamous , The Last Of Us, Modern Warfare 2 (correct me if I'm wrong their Hans)

I am interested to hear what you guys think on the subject. Agree? Disagree?

-DJ


----------



## G.R. Baumann (Nov 15, 2013)

- Music for video games app
- Cloud based library of every sample under the sun
- Random creation of effects, trailers, motifs, hooks, from epic to porn
- Composing by numbers (beta version)
- tagged browser library of emotions linked to random creator
- automatic registration of final composition with library of congresss

:wink: 

sitting back... Popcorn


----------



## Walid F. (Nov 15, 2013)

This is a really interesting topic. I don't have much experience in this, but it seems to me that you can do much more with adaptive music? If the music is, in real-time, adapted to the picture and the actions of the game, it would provide a much deeper and interactive experience than if it's all edited and looped before hand. I don't think that would take away any artistic aspect of it?

Please do keep this discussion going, very interesting since I am currently working on several games and would love to find the best approach for certain types of games!



G.R. Baumann @ Fri Nov 15 said:


> - Music for video games app
> - Cloud based library of every sample under the sun
> - Random creation of effects, trailers, motifs, hooks, from epic to porn
> - Composing by numbers (beta version)
> ...



I wish I could understand anything of that :(


----------



## Daniel James (Nov 15, 2013)

Walid F. @ Fri Nov 15 said:


> This is a really interesting topic. I don't have much experience in this, but it seems to me that you can do much more with adaptive music? If the music is, in real-time, adapted to the picture and the actions of the game, it would provide a much deeper and interactive experience than if it's all edited and looped before hand. I don't think that would take away any artistic aspect of it?
> 
> Please do keep this discussion going, very interesting since I am currently working on several games and would love to find the best approach for certain types of games!
> 
> ...



I used to be in a similar boat but I nowadays I feel that the adaptive systems are adding almost unnecessary amounts of restrictions on the composer. Sometimes the adaptive soundtracks sound a little forced as they have to account for so many potential factors and make it all sound kind of seamless. Granted that can offer some tighter edits and more specific moment to moment music however I feel the more creative and musical scores tend to be the ones where the composer was told to do what they do best and just write some awesome music. Then you have some clever people work over the stems to make it fit into the game, that way the music has its original creative and artistic intent maintained.

Like I said, an recent example is The Last Of Us. Such a beautiful score, its dark, threatening, sad, hopeful etc and you feel this while playing and its not been scored to the system. It was just written and then edited to the game.

Infamous is another example, they got the composers to just write some awesome music then took the stems and made an adaptive system out of it.

Now I understand that composers these days have to just suck it up and learn to do it, but it just seems like forcing a technical practice on a person you really want to be focusing on art seems to me a little like shooting yourself in the foot.

For the record I have done both before 

-DJ


----------



## Walid F. (Nov 15, 2013)

I don't really see where the line goes though. We have soundtracks like Shadow of the Colossus where Koh Otani wrote long beautiful pieces, but they were still adapted to the game mechanics. Like you would climb a colossus - One cue, once atop him, another cue, but much more heroic. You fall down - yet another one that should show that you have to start over and the action has gone down a bit.

I feel edited music can do this too? I'm not sure where the distinction is made, really. Can edited music be used with things like Fmod? 

Where can I read more about these different approaches and how they are utilized? :D


----------



## KEnK (Nov 16, 2013)

Daniel James @ Fri Nov 15 said:


> ...to me adaptive music systems, with all its technical restrictions makes composition end up being more of a science based piece of work as opposed to an artistic one...


I'm not a gamer so perhaps I can only guess at what it is you mean here.

I think you mean the adaptive process is where the music would have to 
continuously unfold according to the plot or visuals.

If that's correct, I wouldn't find writing such a piece to be any more "scientific"
or less "artistic" than composing w/o any visual reference in mind.
Personally I would enjoy the challenge of myriad possibilities.
I'd see it as writing more pieces rather than a restriction on one piece.

I started a thread a while ago about algorithmic composition.
So perhaps my orientation is more naturally a marriage of math and music.
In that discussion, I mentioned once writing for a 10 piece jazz/funk band.
I experimented a bit w/ the concept of "spontaneous composition".

I used something called a Markov Chain as a model.
Simply stated a Markov Chain says "If A, then B".
Slightly extrapolated you can say,
If A, then B or C
If B, then C or D
If C, then A or E
If D, then A
If E, then C or B

(Letters represent structures, or parts)

It seems like this kind of formula would be particularly suited to the adaptive approach.
When I did this w/ the group, I dispensed w/ the "rule framework".
The "head" was played, then during solos anyone could direct one or more people
to play a set part. It worked. 
The piece always retained it's identity yet was different each time. 
And that group was Funky as Hell- not scientific at all.

k


----------



## Daniel James (Nov 16, 2013)

KEnK @ Sat Nov 16 said:


> Daniel James @ Fri Nov 15 said:
> 
> 
> > ...to me adaptive music systems, with all its technical restrictions makes composition end up being more of a science based piece of work as opposed to an artistic one...
> ...



Yeah adaptive where the music changes depending on the situation...on of the more common systems is a layered one where you basically write a 1 minute or so loop which will then be layered on with different elements which happen depending on the state of the game. Each layer has to work with the others...so the main reason I think it puts a slight strangle hold on the artistry of a composer is it is essentially forcing them to have one musical idea that cant venture too far because, on a technical level, it still has to work with itself. Of course you can still be creative here, but I tend to feel the better 'music' is written when the composer isnt worrying to much about if it will technically work when the situation changes, they are doing what they are good at. Writing music.

-DJ


----------



## dannthr (Nov 16, 2013)

So, in order to clarify my discussion point, I'd like to establish a few vocabulary terms as I use them upfront. I've gone back and forth with Richard Stevens a bit about these terms as we're both involved in education and being concise with language is something we're both really interested in achieving--and I think it's important to begin accepting an established terminology if we're to start incorporating music for interactive entertainment into the wider musical aesthetic theory.

Nonetheless, we couldn't quite agree.

So here's how I drop it:

_*Procedural Audio *is a methodology for designing audio events or systems (or a description of an audio event or system resulting from an application of design or method)._ 

This term more describes the method by which we create and design dynamic audio events.

_*Generative Audio *is an audio event (or system) resulting in the synthesis of new audio product._ 

This is audio, or in our case, music that is created by a designed system, resulting in something completely new.

_ *Reactive Audio *is an audio event (or system) that changes its output according to a design utilizing information from an external system through a notification pathway. _

So, in our case, this is a musical design that is designed to respond to changes in the game state or by some information passed from the game to the audio engine. A lot of game music falls under this category and you can visualize the relationship like this:

Game ==> Music

_* Interactive Audio *is an audio event (or system) that changes its
output according to a design utilizing information from an external
system and informs that system’s processes through a notification
pathway._

So, in our case, this would be a musical design that changes based on notifications from the game, but also, it feeds that information back into the game so that the game can be informed by the music as well. You could visualize this relationship like so:

Game <==> Music

Wonderful examples of this would be the SSX Tricky Soundtrack. When the player gets air on their snowboarding tricks, the Game engine feeds altitude information to the Music engine and the Music engine applies a high-pass filter on the music; when the player is about to land, the Music engine sends BPM information back into the Game and the Game engine time-delays the animation so that the player always sticks their landing on the beat.

_*Adaptive Audio *is an audio event (or system) that changes its design
based on heuristic analyses of input through a notification pathway._

And here we are at Daniel's word, which as you can see here, I interpret as a system by which the music actually changes based on an analysis of player input, providing a musical experience that is specifically suited toward a player's play style--thereby providing an extremely unique and personal musical experience.

I have never played a game that did this. But the relationship would look something like:

Player ==> Game <==> Music <== Player (or if I could do triangles).

_* Performative Audio *is an audio event (or system) that determines the
outcome of an external system based on feedback through a notification
pathway._

This is an example of your standard Rock-band or Music-puzzle game (or action puzzle or whatever you want to call it). The changes to the music are very reactive, but the player has direct input on the music output (because the game emulates playing a musical instrument).

_ *Dynamic Audio *describes an audio event (or system) design that results
in a changing or evolving product, usually based on communication with
an external system._ Which, we can agree, encompasses almost all of this.

This is pretty much all the audio I just described--or audio that is essentially non-linear or in some way changes over time based on something.

------

With that stated clearly, my personal opinion is that rejecting the "science," as you put it, is focusing on the art is a false premise.

This would be like rejecting harmony, counterpoint, recording technology, digital audio workstations, etc.

At some point you have to accept that the science IS the art and instead, I am of the opinion that the artist, if they wish to embrace the potential of their medium, that they should embrace the technology, embrace the systems, and dive in--immerse themselves, if you will, so that it doesn't encumber their creativity, but instead inspires it.

I think Sony has had some amazing working relationships with their composers. I absolutely love what Greg Edmonson did for Uncharted, and I am an Amon Tobin fan. But they're really protecting these people from the evolution of their art-form by calling it technology or complicated or whatever.

I think, as a composer, you need to create in your way, the way that you work and operate; but if you're young (or young at heart) and you're willing to learn, then I think being inspired by the possibility is by and far the better way to go.


----------



## Daniel James (Nov 16, 2013)

> With that stated clearly, my personal opinion is that rejecting the "science," as you put it, is focusing on the art is a false premise.



I didn't quite follow this sentence my brain is having trouble going round the grammar of it, could you please say it a different way for me 



> This would be like rejecting harmony, counterpoint, recording technology, digital audio workstations, etc.



Totally disagree with this point, they all with except maybe recording technology allow a composer to create music...they are not restrictions of a medium.



> At some point you have to accept that the science IS the art and instead, I am of the opinion that the artist, if they wish to embrace the potential of their medium, that they should embrace the technology, embrace the systems, and dive in--immerse themselves, if you will, so that it doesn't encumber their creativity, but instead inspires it.



Of course every artist is free to pick and choose what projects they do and as I mentioned there are some great examples of the edited music style working fantastically in games (I remember hearing that Metal Gear Solid 2 wasn't score to picture, it was written and edited later...and I LOVE that score.) So I am saying that I personally think you will arrive at a more musically unique and voiced end product if you are not restricting where the composer can take the music. When you attach an adaptive musical system to the composition process the composer isn't solely focused on where the music can go but more on if it will work correctly. Of course you can get good at adaptive but I still maintain the opinion that by restricting a composer like that in the first place you are taking attention away from what he is hired to do...write kick ass music 



> I think Sony has had some amazing working relationships with their composers. I absolutely love what Greg Edmonson did for Uncharted, and I am an Amon Tobin fan. But they're really protecting these people from the evolution of their art-form by calling it technology or complicated or whatever.



They are protecting the artists from the evolution of the art form by calling it technology? I don't quite follow that sentence, unless you mean to say that they are letting the composer just focusing on what they do...which is write music. And thats my point, there are perhaps some of my favorite scores out there across all media.



> I think, as a composer, you need to create in your way, the way that you work and operate; but if you're young (or young at heart) and you're willing to learn, then I think being inspired by the possibility is by and far the better way to go.



I agree, you can never learn too much and I would encourage everyone to learn to write for all the different musical systems but I still believe that a composers job should be totally focused on writing music without the restriction of a technical system dictating how far they can take some ideas.

-DJ


----------



## dannthr (Nov 16, 2013)

Daniel James @ Sat Nov 16 said:


> > With that stated clearly, my personal opinion is that rejecting the "science," as you put it, is focusing on the art is a false premise.
> 
> 
> 
> I didn't quite follow this sentence my brain is having trouble going round the grammar of it, could you please say it a different way for me



Suggesting that ignoring the systems available to create dynamic and responsive music allows the composer to focus on creating music is an assertion I disagree with.



> > This would be like rejecting harmony, counterpoint, recording technology, digital audio workstations, etc.
> 
> 
> 
> Totally disagree with this point, they all with except maybe recording technology allow a composer to create music...they are not restrictions of a medium.



Why do you see interactive systems as a restriction? They are designed specifically to open up possibilities--to expand potential, not restrict it.

I can only imagine that you feel it is a restriction because you feel encumbered by the technology that allows you to achieve this--in which I suggest that learning how to exploit a system that provides expanded potential is no different than learning any other tool--technological or theoretical.



> So I am saying that I personally think you will arrive at a more musically unique and voiced end product if you are not restricting where the composer can take the music. When you attach an adaptive musical system to the composition process the composer isn't solely focused on where the music can go but more on if it will work correctly. Of course you can get good at adaptive but I still maintain the opinion that by restricting a composer like that in the first place you are taking attention away from what he is hired to do...write kick ass music



The composer is only restricted by their lack of insight into the functionality of a system.

If I may illustrate an example:

Recently, I spoke with a composer on this very topic--he had just come off of a fairly big title and he had written some great music for it. However, he lacked insight into the technology and systems that drove the interactivity of his score and two very awkward things happened as a result:

1) The client/studio requested a dynamic music structure that the composer felt was inappropriate to their writing method--so he rejected it on principle. Unfortunately, the studio was in error--they asked for a dynamic music structure because it was popular and what people did, and the structure they requested did not serve the gameplay. 

2) The composer's instincts where spot on in rejecting their request, but the composer was not able to then turn that around and exploit the system to create something more appropriate. Because the technology was new on both sides, neither used it effectively or could offer strong options, what resulted was a score that served the overall feel of the game, but totally ignored the nuance or flow of any individual gameplay run through.



> > I think Sony has had some amazing working relationships with their composers. I absolutely love what Greg Edmonson did for Uncharted, and I am an Amon Tobin fan. But they're really protecting these people from the evolution of their art-form by calling it technology or complicated or whatever.
> 
> 
> 
> They are protecting the artists from the evolution of the art form by calling it technology? I don't quite follow that sentence, unless you mean to say that they are letting the composer just focusing on what they do...which is write music. And thats my point, there are perhaps some of my favorite scores out there across all media.



In the Sony case, they had musically strong integrators working to adapt pieces of a linear score to a system that allowed really appropriate interactivity--but these integrators served as a buffer between the systems and the composers.

This allowed the composers to take very general direction from an informed music supervisor and the pieces were produced and recorded with interactivity in mind.



> > I think, as a composer, you need to create in your way, the way that you work and operate; but if you're young (or young at heart) and you're willing to learn, then I think being inspired by the possibility is by and far the better way to go.
> 
> 
> 
> ...



At what threshold do you say it is writing music or not writing music? At what point do you say this is not writing music, it is arranging? At what point do you say, this is not writing music, it is orchestrating? At what point do you say, this is not writing music, it is production? Sound design? 

It is all composition--every element is part of your composition, every timbre, every hit, every tone, these are the choices you make as a composer (lest you believe that composers deal solely in fundamental tones and tiny motives and themes and nothing else)--this is the next blossoming moment in composition--when we, as composers, don't just write to what is, but write to what could be. 

Unleashing possibility, potential, and the exciting opportunity to connect to our listeners in a unique and beautiful way--to interact with them and have them interact with our music.

This is not restricting, it is the opposite.

They say that creative people are merely experts at navigating endless possibility.

So let's do that--let's make that journey a part of our art form.



This is why I call myself a game composer and not a composer currently working in games.

You have to connect to the medium if you want to create something amazing and to me, that means understanding it.


----------



## Daniel James (Nov 16, 2013)

> Suggesting that ignoring the systems available to create dynamic and responsive music allows the composer to focus on creating music is an assertion I disagree with.



Fair enough.



> Why do you see interactive systems as a restriction? They are designed specifically to open up possibilities--to expand potential, not restrict it.



I see the system as a restriction as its focusing the composer firstly compose music to a very strict set of rules...ie it has to loop, it has to musically make sense in a loop, it has to make musical sense at every point should the music have to fade elsewhere (I am speaking to the layered approach there) etc. And while these may not seem like big deals...its the combination of all the little rules that forces the composer into a much narrower corridor of flexibility as to the different directions and musical elements they can develop in the music. Of course you can be creative _within_ but that doesnt change the fact you are restricting the flexibility of the composer to write without a boundary.



> I can only imagine that you feel it is a restriction because you feel encumbered by the technology that allows you to achieve this--in which I suggest that learning how to exploit a system that provides expanded potential is no different than learning any other tool--technological or theoretical.



I had this happen in a different thread earlier...so please let me just say this now, that lets not make the thread about me personally. And please don't presume to know my understanding or ability of the field simple based on my opinion of it. 

I have written a few scores that used very rigid adaptive music system and I have written the edited way... I don't think its about your ability to exploit the system because at the end of the day you are stlll hitting the border eventually. At somepoint you may have to sacrifice a musical idea or movement because it won't work in relation to a later that may or may not happen in the gamers play through...which to me is reason enough to make me believe, that even though one can be creative and artistic withing the boundaries...the boundaries still exist. 




> This would be like rejecting harmony, counterpoint, recording technology, digital audio workstations, etc.




Totally disagree with this point, they all with except maybe recording technology allow a composer to create music...they are not restrictions of a medium.




> Why do you see interactive systems as a restriction? They are designed specifically to open up possibilities--to expand potential, not restrict it.
> 
> I can only imagine that you feel it is a restriction because you feel encumbered by the technology that allows you to achieve this--in which I suggest that learning how to exploit a system that provides expanded potential is no different than learning any other tool--technological or theoretical.



So I am saying that I personally think you will arrive at a more musically unique and voiced end product if you are not restricting where the composer can take the music. When you attach an adaptive musical system to the composition process the composer isn't solely focused on where the music can go but more on if it will work correctly. Of course you can get good at adaptive but I still maintain the opinion that by restricting a composer like that in the first place you are taking attention away from what he is hired to do...write kick ass music Smile




> The composer is only restricted by their lack of insight into the functionality of a system.



Completely disagree. Having an intricate and detailed insight into the functionality of the system only allows you to see where the borders are, it doesn't allow you to disregard them.



> This would be like rejecting harmony, counterpoint, recording technology, digital audio workstations, etc.



Totally disagree with this point, they all with except maybe recording technology allow a composer to create music...they are not restrictions of a medium.



> Why do you see interactive systems as a restriction? They are designed specifically to open up possibilities--to expand potential, not restrict it.
> 
> I can only imagine that you feel it is a restriction because you feel encumbered by the technology that allows you to achieve this--in which I suggest that learning how to exploit a system that provides expanded potential is no different than learning any other tool--technological or theoretical.



So I am saying that I personally think you will arrive at a more musically unique and voiced end product if you are not restricting where the composer can take the music. When you attach an adaptive musical system to the composition process the composer isn't solely focused on where the music can go but more on if it will work correctly. Of course you can get good at adaptive but I still maintain the opinion that by restricting a composer like that in the first place you are taking attention away from what he is hired to do...write kick ass music 




> I think Sony has had some amazing working relationships with their composers. I absolutely love what Greg Edmonson did for Uncharted, and I am an Amon Tobin fan. But they're really protecting these people from the evolution of their art-form by calling it technology or complicated or whatever.



They are protecting the artists from the evolution of the art form by calling it technology? I don't quite follow that sentence, unless you mean to say that they are letting the composer just focusing on what they do...which is write music. And thats my point, there are perhaps some of my favorite scores out there across all media.



> In the Sony case, they had musically strong integrators working to adapt pieces of a linear score to a system that allowed really appropriate interactivity--but these integrators served as a buffer between the systems and the composers.
> 
> This allowed the composers to take very general direction from an informed music supervisor and the pieces were produced and recorded with interactivity in mind.



And that is what I am saying, this way of working I see as much more creatively flexible for the composer. He keeps in mind the fact that at some point the music might change but he isnt locked into....ok if I put this note here I can no longer use note x,y,z because in layer 6 it does this which will clash. He can just focus on and write the music, then allows someone to work that into the game later. That way the music is creatively free and open musically.



> At what threshold do you say it is writing music or not writing music? At what point do you say this is not writing music, it is arranging? At what point do you say, this is not writing music, it is orchestrating? At what point do you say, this is not writing music, it is production? Sound design?



I think that is slightly outside the point, I fully accept that you still have to write music in adaptive systems. I am saying that the restrictions of the system will inevitably lead you down a certain, less flexible, route.



> It is all composition--every element is part of your composition, every timbre, every hit, every tone, these are the choices you make as a composer (lest you believe that composers deal solely in fundamental tones and tiny motives and themes and nothing else)--this is the next blossoming moment in composition--when we, as composers, don't just write to what is, but write to what could be.



I agree. Totallly agree. And what I am saying is that forcing a composer to compose into the system is restricting them from writing in a more open and flexible music way. You can still write the music that could be in the edited system...it just in a non adaptive way you are not always constantly scrutinizing each note to make sure it will work with a layer that could potentially happen. You write the greatest music you can, then someone takes that and makes it work in the system.



> Unleashing possibility, potential, and the exciting opportunity to connect to our listeners in a unique and beautiful way--to interact with them and have them interact with our music.



This works just as well in either system.



> This is not restricting, it is the opposite.



Restricting the composer...does not free the composer. Far from it. Having to constantly make sure what you are writing this second will work musically down the road, making musical sacrifices to ensure it does, is not a good example of freeing the composer.



> You have to connect to the medium if you want to create something amazing and to me, that means understanding it.



Again just because I believe a composer should be musically free to compose the best music he can without restriction (as was done very successfully in the titles I posted previousl) doesn't mean I lack the understanding of how the systems work.

-DJ


----------



## dannthr (Nov 16, 2013)

All of your quote tag mishaps aside--I acknowledge that where you see restriction, I do not.

Where you see forcing someone into a box, I do not.

You talk about clashing notes, how is that different from orchestration or functional harmony?

The only difference is that they don't talk about it in traditional music theory.

Music has rules, music rules can be broken, but to be effective as a composer is to understand how music functions.

Interactivity is no different.

You can ignore the rules of music and edit your music later on to adhere or you can hire other people to do that for you--and in this way you can claim to be a free musician, composing whatever you like and letting your creativity run wild. If that's what you're talking about, then fine, all systems set before you are a restriction--along with every tool of your chain from idea to product.

That is a choice you make in your perception of the process. You are deciding to see the tools as something that restricts you.

Fine. I'm not going to argue against that.

But at the end of the day, your linear composition is just a linear composition, and interactive music is so much more.


----------



## Daniel James (Nov 16, 2013)

> All of your quote tag mishaps aside--I acknowledge that where you see restriction, I do not.
> 
> Where you see forcing someone into a box, I do not.



I fixed the quotes  and I guess if thats your stance we will have to agree to disagree on those points.



> You talk about clashing notes, how is that different from orchestration or functional harmony?



I was speaking very specifically here. And for a a random example, lets say you are in bar 2 of a loop and your arrangement forms a minor chord but in loop 6 bar two it would have been perfect to have the tonality shift to a diminished chord (basic examples) You have to make a sacrifice somewhere so that you can have both... I mean thats the best example I can come up with at 6am after being awake over 30 hours XD



> Music has rules, music rules can be broken, but to be effective as a composer is to understand how music functions.



_Music_ has rules that can be broken...adaptive music systems on the other hand are not quite as black and white...for example try having layer 3 at a separate tempo the 1 and 2. xD




> You can ignore the rules of music and edit your music later on to adhere or you can hire other people to do that for you--and in this way you can claim to be a free musician, composing whatever you like and letting your creativity run wild. If that's what you're talking about, then fine, all systems set before you are a restriction--along with every tool of your chain from idea to product.



I wouldn't put an adaptive music system in the categories of tools that enable you to write music, yes its technology but software like DAWs and plugins don't restrict you from writing free flexible music where as an adaptive music has musical rules and restrictions in order for it to work seamlessly. Its the output destination of the end result.



> That is a choice you make in your perception of the process. You are deciding to see the tools as something that restricts you.



Adaptive music system is a destination for the music, not a tool of musical creation in my eyes.



> But at the end of the day, your linear composition is just a linear composition, and interactive music is so much more.



Well I disagree here, thats the whole point of editied music, its made to work with the game system....sometimes ironically it even works like an adaptive system, however that is not the composers focus. They write the music as musically as they can, then the technical wizards come in and make it work to the game...just like Infamous, The Last Of Us, Modern Warfare 2 etc

-DJ


----------



## Walid F. (Nov 17, 2013)

Having finished reading all your discussion points, it seems to me that you are arguing opinions on separate definitions of what it really means to write to adaptive music systems. One side of the argumentation is saying that the tools presented by this system actually restricts the music in itself simply because it forces the composer to make sacrifices in order to make it seamless. The other side says that the tools presented by the system actually expands the possibilities, and does not restrict because... I am not sure. I'm trying to understand it and that's where I am now :D Need examples to really understand your point, Dan!

I want to look at this:



dannthr said:


> Why do you see interactive systems as a restriction? They are designed specifically to open up possibilities--to expand potential, not restrict it.



DJ gave great examples of what he meant with the restriction, and I agree completely with it. If the different loops and streams of music do not blend well together, it imposes problems that you don't have to solve if you're simply writing what would fit, and then edit it afterwards to make it seamlessly fit. 

Could you give some concrete and well explanatory examples of how the adaptive approach actually expands the potential of writing music for games? This way it will be much easier to see why you can't agree with the example DJ just gave, because it really is the definition of restriction if you look at that exact example - If you can't make it fit, it is a problem you have to solve, you have to get rid of notes, have to think much more on every loop needing to fit together if they need to blend seamlessly, don't you agree?

This is extremely interesting and educational! Thanks for this. :D Oh also, I want to reinforce DJ's point of not making it about a person, but more on the actual thoughts, opinions and deductions - that is what makes solid discussions solid. I am not experienced with these approaches myself, but I do have a brain!! 

Oh! also, DJ - the 11th post of this thread is really messed up with quotes. Some repeat themselves 2 times I think even :D 

W


----------



## dannthr (Nov 17, 2013)

Well, Walid, I suppose it's because I did not see it as an interactive music problem. What he described was specifically a linear music problem:

How do I go from A to B?

You draw a line.

I don't know Daniel very well, so it's very possible that I can misinterpret him--I've only spoken to him once over the phone, and even then, he didn't know it was me.

But Daniel described in that example a very common linear film problem:

I have theme A and theme B and they're separated by x number of film seconds--how do I connect these?

Now, unless Daniel is specifically coming in here to advocate for through-composed only music, then I have a hard time understanding what _his_ argument is--because the problem he described is exactly the problem you run into when you write the music first and you approach the interactive scenario later.

You have to "take away notes?" Well, that sounds like editing to me. Any time you take away notes, I guess I would have to call that editing. Unless you're specifically writing through from front to back without deviation, then yes, you will have to edit the shit out of that score.

Because that doesn't sound like someone who is writing interactive music to me.

I'll give another Sony example (where the previous Sony examples were given to describe how the "editing" method only worked _because_ of the exceptional work done by all the bad ass integrators at Sony), but I hesitate because I haven't spoken directly with the Composer on this topic (actually, I may have never spoken to him, I'll have to think about that):

Journey.

Journey, a score by Austin Wintory, is a great example of a composer who was deeply connected to the design process throughout the game's production, who worked closely with an audio integrator and always pushed the potential of the system--always engaging the "what if" possibilities. Because of this, you have a score that is brilliantly executed, brilliantly responsive--a score that is touching and personal and one that only plays out exactly like that for the individual player (so many moments that do not occur because they can only happen if the player does certain things), and one that was nominated for a Grammy against film and music titans.

He didn't engage the system through its restrictions, he explored the freedom of possibility.

We're talking about a perception problem.

So, back to the specific example?

I have a [Loop A] and I need to transition to [Loop B]. This is the problem that he's described. And he's frustrated by the fact that Loop B starts with a harmony that doesn't transition well from Loop A and suggests that his problem is that he wrote a harmony for Loop B that doesn't transition well.

That may be true, but it sounds like you wrote Loop B with no regard for the system that you adopted.

The macro solutions are this: Edit Loop B (which was the suggestion, I think) or change the system (which you can't do because, for whatever reason, you're not the person in charge of designing the system).

What other solutions might we try? Just about anything.

If this were happening in film, and we had some time between these two themes, then perhaps we'd write a connective piece of music to smooth the transition between these elements. If we had no time, perhaps we would change the harmonic language and utilize a variant of the theme (we do this a lot in film).

At this stage, you're only limited by your imagination and your engagement with the interactive systems.

If you're not the person integrating the audio, if you're not the person designing the system, then maybe you should be--it's important to identify where the actual restrictions are coming from in any creative endeavor.

Going back to my original example (from previous posts where I mentioned talking to a composer who had just come off of a big title and underutilized his interactive music tech): The composer, who used fully live instruments in his score, said he used a certain trick of repeating the first two measures of his loop at the end of his loop in order to preserve the continuity of the spatial tail from the church where he recorded his orchestra.

And I was like: you didn't have to do that. The technology he was using allowed overlapping tails and anacrusis, which meant he didn't have to edit loops like that at all. He was dumbfounded. I was dumbfounded that neither he nor the people he was working with knew.

All of that was extra work for performers, extra work for him, extra work for his editor (or him, which ever did the editing)--all the time thinking that they were engaging a restriction when in fact they were ignoring a feature.

Daniel James suggests that knowing ("knowing the boundaries") gives you restrictions and as a life-long learner, as an educator, and as an artist, I think that's absolutely absurd.

He's just moving the work around, either he's offloading the work to a process he feels is not creative (editing) or he's giving the work to someone else to edit. But that wouldn't fly in linear music either.

So yeah, I don't really see how his examples are problems specific to adaptive, reactive, or interactive music.


----------



## Marius Masalar (Nov 17, 2013)

Without wishing to interrupt the excellent discussion going on between two of my favourite Dans, I just wanted to put in my two cents since I'm actually in the midst of a project right now where the question of musical interactivity and its implementation was a major concern.

As I can't discuss this too openly, I'll just give a birds-eye view of the situation because it was an interesting process for me to go through and much of the discussion was based around what's being said here.

Initially, I was approached by the developer with a very specific implementation system in mind that was based primarily on after-the-fact editing of musical cues to fit a predetermined structure.

This structure was fantastic—on paper. It was responsive to in-game events, it looked great if you drew it as a flowchart, and it made a lot of logical sense.

When I looked at what was being proposed though, I realized that it was fundamentally overcomplicated and would impose inevitable but unnecessary restrictions on how I was able to write the music so that the various transition scenarios remained as transparent as possible.

I brought this up in a discussion with the developers and it led to a series of meetings between myself, my bosses, and the implementation engineer to see if we could find a better solution.

I'm very much of the opinion that interactive music is a fascinating challenge that represents the epitome of what makes game music so thrilling for those of us who focus on it. 

Crafting "interactive" music by writing linear cues and adapting them into non-linear implementation systems seems like a broken approach to me on a very basic level. The most skillful of composers paired with the most skillful of implementation teams will pull it off admirably because the sheer force of their respective talents will make it work, but ultimately the end result will not be as natural.

Back to my situation, I proposed to them a new system that was based on layering vs. a myriad of separate cue chunks, with some additional functionality specific to this title. What I proposed was simpler, with less implementation hassle, and a more seamless ability to transition between situations.

There is certainly a science to good game audio—that's not really up for discussion—but like any good use of the scientific method, the simplest solution is often best. The initially proposed system was designed from an editor's perspective and so it made the creative angle suffer.

In the end, what we're doing now is more responsive and more natural sounding despite requiring fewer separate audio assets. It's designed and developed from the ground up to offer the best player experience without sacrificing musical sophistication or integrity. My hands are not tied. 

Together, we found the best solution for *this title* and that's the other half of what makes game audio so much fun—getting from A to B isn't a straight line where we just decide what colour it is, it's a wild thing that needs to be able to turn on a dime to hit point F and R along the way. And it's different for every game.

I love my job :D

Now back to reading everyone else's responses I go—you guys are always a treasure trove of experiences and knowledge.


----------



## dannthr (Nov 17, 2013)

Yes!

But Marius, if you didn't know what COULD be done, how could you propose an alternative solution?


----------



## Marius Masalar (Nov 17, 2013)




----------



## germancomponist (Nov 17, 2013)

Very interesting to read here!


----------



## dannthr (Nov 17, 2013)

Heheh!

And that's why in my first post I brought up a vocabulary.

Because dynamic musical design is not just about this way or that way--but about function and how we can design a system that is meaningful, creative, expressive, and specific to the game's needs.

Also, I'm an English nerdo!

(I have a BA in English: Creative Writing  )


----------



## Daniel James (Nov 17, 2013)

> I don't know Daniel very well, so it's very possible that I can misinterpret him--I've only spoken to him once over the phone, and even then, he didn't know it was me.



Wait wut! we did? 



> Now, unless Daniel is specifically coming in here to advocate for through-composed only music, then I have a hard time understanding what his argument is--because the problem he described is exactly the problem you run into when you write the music first and you approach the interactive scenario later.



I'm not really advocating in either, I am just seeing how others feel on it. 



> You have to "take away notes?" Well, that sounds like editing to me. Any time you take away notes, I guess I would have to call that editing. Unless you're specifically writing through from front to back without deviation, then yes, you will have to edit the [email protected]#t out of that score.



Maybe the word I am using for the style is confusing you here.. I mean every bit of music gets edited at some point, so thats not what I am saying. The taking away notes part is in reference to the adaptive music system where each layer has to seemlessly interact with every other one....and say mid way through bar 1 of layer one you have a cool harmonic idea, but then later in layer 8 bar 1 you think it would be awesome to have a totally different and unrelated harmonic idea you will have to sacrifice one of them because layer one and layer 8 have to seamlessly work together. Of course that doesnt happen every time but the potential for it to happen looks alot like a musical restriction to me.



> Journey, a score by Austin Wintory, is a great example of a composer who was deeply connected to the design process throughout the game's production, who worked closely with an audio integrator and always pushed the potential of the system--always engaging the "what if" possibilities. Because of this, you have a score that is brilliantly executed, brilliantly responsive--a score that is touching and personal and one that only plays out exactly like that for the individual player (so many moments that do not occur because they can only happen if the player does certain things), and one that was nominated for a Grammy against film and music titans.
> 
> He didn't engage the system through its restrictions, he explored the freedom of possibility.



Journey was a fantastic score! And I am not saying that you cant still be very musical in adaptive systems, just that they restrict the composer. I believe journey was based on an adaptive system however the game itself doesn't seem to have many changeable moments, I mean musically and within gameplay. The game itself lends itself nicely to allowing the music to evolve over time as the pace is rather slow and unchangeable. I imagine Journeys score would have sounded a little more 'rigid' if it was a fast paced and changeable game, where situations change within seconds of each other...at that point we would have heard the restrictions of the system come into place.



> I have a [Loop A] and I need to transition to [Loop B]. This is the problem that he's described. And he's frustrated by the fact that Loop B starts with a harmony that doesn't transition well from Loop A and suggests that his problem is that he wrote a harmony for Loop B that doesn't transition well.
> 
> That may be true, but it sounds like you wrote Loop B with no regard for the system that you adopted.



And here is exactly my point. The fact that you have to musically tie loop B to loop a directly means that loop b is restricted by the ideas you had in loop a. You are not free to musically write loop b however you want, even if you think that loop b would benefit from a different musical structure. Which means that you have to sacrifice and compromise to make them work together



> If this were happening in film, and we had some time between these two themes, then perhaps we'd write a connective piece of music to smooth the transition between these elements. If we had no time, perhaps we would change the harmonic language and utilize a variant of the theme (we do this a lot in film).



But what if the passage between loop A and loop B is a sudden transition without the time to set it up. Layer B might work better with a whole new harmonic structure, but if it is tied to layer A you are confined to the musical footprint already established.



> If you're not the person integrating the audio, if you're not the person designing the system, then maybe you should be--it's important to identify where the actual restrictions are coming from in any creative endeavor.



But thats where we are REALLY going into the scientific aspect. I still think a composer should be the guy who focuses on writing the music, not having to worry about building the musical system itself to implement it. Thats dividing both your time and focus.



> Daniel James suggests that knowing ("knowing the boundaries") gives you restrictions and as a life-long learner, as an educator, and as an artist, I think that's absolutely absurd.



No I am not saying that knowing the boundaries give you restrictions. I am saying that the restrictions *are*there, whether you know it or not, whether you accept it or not. They exsist. The examples I gave earlier with Loop A and Loop B above, showing that there are _musical_ sacrifices one would have to make in the interest of the system are, I believe, proof of a restriction...I can't see how having to change your musical idea is more liberating.



> He's just moving the work around, either he's offloading the work to a process he feels is not creative (editing) or he's giving the work to someone else to edit. But that wouldn't fly in linear music either.



I'm not fully sure what the accusation here is. But to answer the more direct wording...yes of course I would move the work around, A composers job should be to compose music, thats our job, thats our art, thats what we train to do. Editing music to a video game system is a more technical aspect in my opinion. So of course it would be ideal to allow someone else to place it for you, the tech wizards, someone whos job is to solely focus on that.

-DJ


----------



## dannthr (Nov 17, 2013)

Daniel James @ Sun Nov 17 said:


> > I don't know Daniel very well, so it's very possible that I can misinterpret him--I've only spoken to him once over the phone, and even then, he didn't know it was me.
> 
> 
> 
> Wait wut! we did?



Ahahahah!

We called you at GDC last March--I'm the one that said this:

DANYO CHAYMZ 'ERE TA TOK AHBOU' SHEENEESAMPOZ SHEENEEBRAAHZ!


----------



## Daniel James (Nov 17, 2013)

dannthr @ Sun Nov 17 said:


> Daniel James @ Sun Nov 17 said:
> 
> 
> > > I don't know Daniel very well, so it's very possible that I can misinterpret him--I've only spoken to him once over the phone, and even then, he didn't know it was me.
> ...



hahaha Ciineeeesamples siiiiineeeeesamapes.....here Daniel talk to Skrillex...BWAAAAHHHHHWUBWUBWUB

-DJ


----------



## dannthr (Nov 17, 2013)

Daniel James @ Sun Nov 17 said:


> Maybe the word I am using for the style is confusing you here.. I mean every bit of music gets edited at some point, so thats not what I am saying. The taking away notes part is in reference to the adaptive music system where each layer has to seemlessly interact with every other one....and say mid way through bar 1 of layer one you have a cool harmonic idea, but then later in layer 8 bar 1 you think it would be awesome to have a totally different and unrelated harmonic idea you will have to sacrifice one of them because layer one and layer 8 have to seamlessly work together. Of course that doesnt happen every time but the potential for it to happen looks alot like a musical restriction to me.



Why the hell are you using an 8+ layer vertical transition system?!

Maybe the emotional mood of Layer 8 is best served through a horizontal transition.

Not all dynamically changing music scores have to feature parallel layers. Geez!

This is a restriction you are imposing upon yourself (or someone is imposing it upon you externally) because I highly doubt 8 layers of parallel composed music is appropriate for more than maybe 0.5% of game scenarios.

Your only restriction is your imagination. You could transition to a different part of the layer, you could add a transitioning segment of music that would feel instantaneous in transition but allow you to maneuver into a very different harmonic language--or you could evoke all the millions of different ways that composers solve these problems in linear music. Try transposing the layer to be more effective (or use an alternative layer that is transposed) or create some alternative variant.

You're talking about an every day linear compositional puzzle and you're assigning it the label of adaptive music when it's not a problem specific to non-linear forms.



I really can't say more without addressing your perception and insight specifically, DJ, and I don't want to come off insulting because I respect you, man!

8 layers of vertical transition--it's just craziness! Who would design that? Like if you're going to use 8 layers, then your method should be additive if anything, and it should feel like you're composing stems--they should feel like the same cue.

If you've got Loop A and Loop B and you want to go from Bar 1 Beat 2 of Loop A to Bar 1 Beat 2 of Loop B, then yes, they have to harmonically compatible, but who the hell is telling you that you have to do that?

Because you don't. You could leave Loop A and enter ANYWHERE YOU WANT in Loop B, anywhere you feel is the best, musically speaking.

The ONLY reason you should leave Bar 1 Beat 2 of Loop A and go to Bar 1 Beat 2 of Loop B is because you feel it's the best musical way to move from A to B.

Tell me who has been feeding you this nonsense, I will set them straight! :D

Is it Mick Gordon? I will smack that dude around! Smackity smackity smack!


----------



## dannthr (Nov 17, 2013)

Daniel James @ Sun Nov 17 said:


> hahaha Ciineeeesamples siiiiineeeeesamapes.....here Daniel talk to Skrillex...BWAAAAHHHHHWUBWUBWUB
> 
> -DJ



Ah, dude! Skrillex! Ahahahaha! I forgot who was Skirllex--but I did not expect how awesome that dude was going to be!

Are you going to GDC next year?! :D


----------



## germancomponist (Nov 17, 2013)

Hey guys, don't forget to let this discussion going on! Very interesting!


----------



## Daniel James (Nov 17, 2013)

dannthr @ Sun Nov 17 said:


> Daniel James @ Sun Nov 17 said:
> 
> 
> > Maybe the word I am using for the style is confusing you here.. I mean every bit of music gets edited at some point, so thats not what I am saying. The taking away notes part is in reference to the adaptive music system where each layer has to seemlessly interact with every other one....and say mid way through bar 1 of layer one you have a cool harmonic idea, but then later in layer 8 bar 1 you think it would be awesome to have a totally different and unrelated harmonic idea you will have to sacrifice one of them because layer one and layer 8 have to seamlessly work together. Of course that doesnt happen every time but the potential for it to happen looks alot like a musical restriction to me.
> ...



Haha my terminology may have been off abit. because I am pretty much referring to the parallel layers system. And yes lol I am on a game right now which goes up to an 8th layer!

There are a few dynamic systems which I think work better than the layered approach. I heard Joris De Man talking about the score to the first 3 Killzones where they would compose in chunks that would not crossfade in layers but change between a few mini compositions. There is still a little restriction as to how much you can develop a musical idea that way BUT its more flexible than parallel layer....but not quite as flexible as writing whatever works for you.

I'll comeback in with a few more points if the post develops but I have to finish something up quick! Please continue 

-DJ


----------



## KingIdiot (Nov 17, 2013)

8 layers is nuts. But truthfully it all depends on gameplay, triggers, rate of change, repeatability, thematic approach.. blah blah

I once did everything from parallel layers to section changes on beat with ifs (what a pain in shuffle/triplets back in the day), to synchronized to animation all within the same game. Intense, but fulfilling in the scope reached within the limitations. I kinda get bored with basic layer/additive heirchy approach I hear in games, and personally I like things to be abrupt sometimes.


----------

