# The compiler in KScript Editor released as open source



## kotori (Feb 9, 2011)

Hello everyone,

I hereby release the script compiler that is part of KScript Editor as open source. The :arrow: source code (63 kB) is up for download. It is licensed under the GNU General Public License v2 which means that you can do pretty much anything with it but if it's incorporated into other programs those need to have their source code released publicly too. In order to run the code one needs Python (version 2.5-2.7) and PLY. Just let me know if there is anyone who wants to try it out and need help to get started.

I do this
 as a gift to this wonderful community and thanks for all the encouragement and feedback I have received.
 in order for everyone to be able to have full confidence that the compiler will stay around and be updated.
 in case it would be possible to gain more contributor(s) to the project - something that could benefit everyone.
Btw. I should probably note that this is a new version of the compiler where perhaps half of the code is newly added. It now has support for functions with return values using the syntax outlined in this post and the compilation speed is typically twice as high as before.

Cheers,
Nils


----------



## Big Bob (Feb 9, 2011)

Hi Nils,

Let me be the first to thank you again for your invaluable contributions to the Kontakt community. I simply hate to think of what it would be like to go back to the days of KSP coding PRIOR to when you first released your editor. I dimly still remember those days.

Making it open source is truly a wonderful gesture. Of course I hope you won't just leave us on our own with it :lol: Personally, I don't think I will want to dig into the 'gears and pulleys' because I doubt I possess sufficient ability to understand it anyway. :oops: But, I'm sure there will be others who will jump at this chance.

I think my workload is finally easing up a bit and boy do I have the catching up to do :( I haven't even begun to use K4 or any of the wonderful improvements you have made to your editor in the last year or so. I'm sorry I haven't been able to contribute much feedback but hopefully that will soon change.

Thanks again Nils.

God Bless,

Bob


----------



## Big Bob (Feb 9, 2011)

> Oh, no - I wouldn't dream of it. Before I leave you will of course also get the 349 page manual titled "Entering the mists of the KSP compiler" with a special appendix covering language grammars in Backus–Naur form and LALR(1) parser construction inspired by the level of detail found in none other than your very own elaborate manuals.



Now there's a left-handed compliment if ever I saw one :lol:


----------



## tonewill (Sep 12, 2011)

Hello Nils,
Are you still going to continue working on KScript yourself? I'm wondering if you will be releasing a version with the new features you mention here, especially functions with a return value.

Failing that, has anyone else taken up the challenge with the source code, or, at the very least, compiled a version with the new features?

I thank thee,
Barry.


----------



## ScoringFilm (Sep 13, 2011)

Another vote of thanks Nils for all your work,

Justin


----------



## kotori (Sep 14, 2011)

Hello

yes, I plan on continuing the development. 

If someone is willing to help out that's very welcome however. For example, I haven't had time to update the internal KSP reference. One can do that by modifying the ksp_doc.txt file included in the installation. Btw. I just uploaded a newer version of KScript Editor for Windows on my web page with support for K5 functions/ variables/callbacks.

/Nils


----------



## Big Bob (Sep 14, 2011)

Hi Nils,

It's a big relief to see a post from you. I was beginning to wonder if something bad had happened. Of course I've been hoping that you were just extremely busy but it's always nice to see some sign that you are OK. :D 

I don't have K5 yet so I don't think I could help with the ksp_doc update. I'm interested in knowing if anyone is using the full version of K5 under Windows XP.

God Bless,

Bob


----------



## Big Bob (Sep 14, 2011)

Hi Nils,

I just downloaded V1.4.8 and I can't get anything to work unless I check 'Use old compiler version'. Here's a simple example of the kind of thing that no longer works right.

*function* format 
``*declare* global fmt.val
``*declare* global fmt.div
``fmt.val := fmt.div + 2
*end function* _{ format }_

Apparently, using a period in a name is no longer allowed? That would indeed be bad news!

Then, trying to understand your macro foo comment for the new compiler, I tried a simplified macro like this which also didn't fly.

macro declare_switch(#name#,x,y,caption,auto)
``*declare* ui_switch #name# 
``*declare* #name#.cx 
````#name#.cx := get_ui_id(#name#) 
*end* macro _{ declare_switch }_

Does this mean that I can no longer use this kind of construct? That would be horrible because I have written tons of macros based on this theme. How would I have to rearrange this to get the same effect?

BTW I notice, even with the old compiler option checked, that the keyword macro is no longer syntax highlighted.

Rejoice,

Bob


----------



## kotori (Sep 14, 2011)

Hi Bob,

I'm sorry for not getting in touch with you sooner.
Both code examples that you posted compiles here. For some reason that I have not understood yet it's not possible to switch from the old compiler to the new compiler without restarting KScript Editor. Please try to restart it and compile again to see if that helps.

Btw. the old version is the compiler that was used in all previous KScript Editor versions but with added support for K5 features. The new compiler is needed in case one wants to use functions with return values.

Cheers,
Nils


----------



## tonewill (Sep 15, 2011)

Thanks Nils! Just downloaded it, made a small contribution, and look forward to trying out the new return function feature. I'm glad this app exists, makes scripting far less painful. Don't have K5, only just upgraded to K4 when they did that offer a couple of months ago. May get K5 when K6 comes out :wink:.

Thanks again,

Barry.


----------



## Big Bob (Sep 15, 2011)

> For some reason that I have not understood yet it's not possible to switch from the old compiler to the new compiler without restarting KScript Editor. Please try to restart it and compile again to see if that helps.



Hi Nils,

That did it, and I'm greatly relieved. :D Now as soon as I find some quiet time I want to check out the return value feature (if I can remember how it was supposed to work :roll: ). 

*I notice however, that the 'macro' keyword is still not syntax highlighted (old or new compiler option). *And, speaking of syntax highlighting, there has been a problem for some time with losing syntax highlighting after making a syntax error. Usually it shows up when there is an error in the host script code that causes the KSE to point to a line in an 'imported' support script. After such an error is reported, much of the syntax highlighting in the imported module is lost. And, the only way I have been able to restore it is to close and then reopen the script. Is there some other quicker way to restore syntax highlighting, maybe like some magic hot key?

Anyway, glad to see you hanging around again.

Rejoice,

Bob


----------



## Big Bob (Sep 15, 2011)

Hi Nils,

I just started playing around with the new function return value feature and I must say that this is all very exciting. o-[][]-o I can't wait to see what I can all do with this to simplify the source for some of my scripts.

Here's some preliminary questions and observations.

Questions:

1. Evidently, the name 'result' can be any name of our choosing? At least it seems to work with several names that I tried (including the name of the function itself).

2. For one-line functions (used primarily as a shorthand) when there are no parameters involved, apparently we need to use something like exp() rather than just exp for the function name?

3. Is this version of the compiler fairly stable and would it be safe to use it for something ambitious like the Math Library?

Observations:

It's probably going to take a certain amount of getting used to how this works 'under the hood' to avoid certain potential problems. For example, consider the following:

*function* min(a,b) -> result
``result := b
``*if* a < b
````result := a
``*end if*
*end function*

If this is referenced with something like Z := min(X,Y) it will work as expected but if referenced with X := min(X,Y) it won't.

*function* exp() -> result
``result := (x + y)/z
*end function*

I can see some advantages to requiring that all functions that return values have to have a () suffix but, for more compact notation, it might be nice if parameterless functions could be written without the (). Would it be difficult to make this optional?

Anyway, I for one am just thrilled to see this happening and I'll continue to provide feedback as I get deeper into it.

God Bless,

Bob


----------



## kotori (Sep 15, 2011)

Hi Bob,

I have already fixed the macro syntax highlighting - it'll be part of the next update.
What you say about loosing syntax highlighting I have noticed occasionally too. An easy way to fix it should be to press Ctrl+A, Ctrl+X and Ctrl+V. I'm not sure why it happens, but it seems like a quite rare problem I haven't bothered to dig very deep into it.

Btw. large parts of the code base for the new compiler is new. For people with complex scripts who are near a release I would suggest using the old compiler version (available as an option in the Settings menu also in the new KScript Editor) since it has been tested for a longer time.

Cheers,
Nils


----------



## Big Bob (Sep 15, 2011)

> I have already fixed the macro syntax highlighting - it'll be part of the next update.



Hi Nils,

Since my last post I noticed that 'on pgs_changed' is also not highlighted. I examined the styles.ini file but saw only minor differences plus a few extra line breaks that weren't in V1.4.7. However, I tried replacing the styles.ini file for V1.4.8 with that of V1.4.7 but the problem persists. Did your fix also cover 'on pgs_changed'?



> An easy way to fix it should be to press Ctrl+A, Ctrl+X and Ctrl+V.



Thanks for the suggestion, I'll try that the next time it happens. This problem is only a minor annoyance and probably not worth much effort to correct. It probably occurs more for me than for you, primarily due to 'cockpit' problems. :oops: For example, one pothole I fall into a lot is, after correcting some problem, I hit F5 with with the import module when I meant to F5 the host script. If the import module is a library that's using certain conditional compilation tricks, it often can't be compiled by itself without producing KSE errors.



> Btw. large parts of the code base for the new compiler is new. For people with complex scripts who are near a release I would suggest using the old compiler version (available as an option in the Settings menu also in the new KScript Editor) since it has been tested for a longer time.



That's sort of what I thought. However, maybe I'll try generating an experimental Math Library version that takes advantage of the new function features (but I won't dump the version that works well with V1.4.7 :lol: )

BTW I haven't checked yet but does the new compiler address the problem I reported of not removing unreferenced KN functions when using code optimization?

Rejoice,

Bob


----------



## Big Bob (Sep 15, 2011)

Hey Nils,

This new KSRV function feature is really terrific o=< 

The more I play around with it the more impressed I am. For example, I notice that you provided for string return values as well as integers. This will indeed be very useful. Of course we can also easily 'fake' a boolean return as long as we can write a single-line function with an expression that evaluates to zero or non-zero and then use

```
if foo() # 0 
  { do something }
end if
```
This editor just gets better and better. I sure hate to think of what it would be like without it :cry: . Time for another Pay Pal visit me thinks. :D 

God Bless you my friend.

Bob


----------



## kotori (Sep 16, 2011)

Hey Bob

alternatively I believe you can make the "value # 0" comparison inside the function and then just write

```
if foo()
  { do something }
end if
```

Or, assuming that you keep your original foo, you could use something like this if you find it easier to read and more general purpose (more general since the return value of foo can also be stored if necessary):

```
function true(x) -> result
  result := x # 0
end function

{ ... }

if true(foo())
  { do something }
end if
```


----------



## andreasOL (Sep 16, 2011)

Hi

must have missed it...

Where is this new return value behaviour documented?

Cheers,
Andreas


----------



## Big Bob (Sep 16, 2011)

Hi Nils,

That's even better. I didn't even try to see if 'result' could be a boolean expression, I just assumed it would have to be of type integer. I was just thrilled when I discovered it could also be of type string. Now I'm even more impressed :o 

BTW Is it my imagination or is this new compiler smarter at handling argument expressions? I notice that the new compiler seems to automatically provide parenthesis as needed (and I don't think the old compiler used to do that did it?). I have always been providing parenthesis with argument expressions in order to resolve evaluation ambiguities. Now it appears we no longer have to do that? This is really super duper. 8) 

Rejoice,

Bob

Andreas, here's the only posting I know about where the proposed syntax was outlined.

http://www.vi-control.net/forum/viewtop ... 613#252613

BTW Is this forum getting slow as molasses in January for anyone else or is it just that way for me?


----------



## kotori (Sep 16, 2011)

Big Bob @ Fri Sep 16 said:


> Hi Nils,
> 
> That's even better. I didn't even try to see if 'result' could be a boolean expression, I just assumed it would have to be of type integer. I was just thrilled when I discovered it could also be of type string. Now I'm even more impressed :o


Return values are similar to function arguments.



> BTW Is it my imagination or is this new compiler smarter at handling argument expressions?


No, it should be smarter as long as we're talking about functions. Macro inlining takes place before the AST is built though, so it works more or less the same as before.



> Andreas, here's the only posting I know about where the proposed syntax was outlined.


That's an accurate description of the syntax supported by the new compiler.


----------



## Big Bob (Sep 16, 2011)

> Return values are similar to function arguments.



Yes, evidently it was allowable to pass booleans as function pararmeters even with the old compiler and I didn't know about it :oops: 

Why then did we lose the ability to pass a function name as a parameter? I'm sure going to miss that. Since you mentioned that we were losing it, I been sitcking to using macros when I need to pass a procedure name but, since macros can't be nested, it's not nearly as nice. I suppose there is a good reason why we lost it, but, it sure would be nice to get it back :roll: 

BTW I notice that whenever I use V1.4.7, right after that V1.4.8 is reverted back to the 'use old compiler' option. It would be nicer if once set to the new compiler option it would stay put unless overtly changed again. I suppose this must be a parameter in a common .ini file or something but it would be nice if you could decouple the two versions for this option switch.

Rejoice,

Bob


----------



## andreasOL (Sep 17, 2011)

Big Bob @ Fri 16 Sep said:


> ...
> 
> Andreas, here's the only posting I know about where the proposed syntax was outlined.
> 
> ...



Hi Bob,

many thanks! And to Nils, for implementing this! The whole function "thing" uses the concept of substitution and thus it can be used as the boolean expression in an if statement. This can help writing much more readable and maintainable code.

Cheers and have a nice weekend,
Andreas


----------



## Big Bob (Sep 21, 2011)

Hi Nils,

I more or less finished torture testing the new compiler so I thought it might be useful if I summarized my observations in this one post. That way if you ever find some time to look into these, you won't have to read all my scattered posts in this thread.

Problems: 
1. Syntax highlighting doesn't work for 'macro' and 'on pgs_changed'.
2. Still not removing unreferenced KN functions
3. Have to close and relaunch when unchecking the 'use old compiler' option.
4. SET_CONDITION isn't recognized.
5. local overide doesn't work in on_init functions.

Questions:
1. Would it be difficult to allow function names without the () suffix when there are no parameters? (I'm of course refering to RV functions).

2. Would it be difficult to regain the ability to pass function names as parameters, perhaps by enclosing them in #proc# or something similar? I realize we can still do this with macros but, since macros can't be nested, it's not nearly as powerful.

3. At least until the need to close and relaunch is corrected, could you decouple V1.4.7. As it is, if I load V1.4.7, the next time I load V1.4.8 it always reverts back to the old compiler and therefore has to be changed, closed, and then relaunched.

Overall, I think this is a beautiful upgrade to the KSE and will greatly improve the readability of the KSE source code. Thanks a million for this update. o-[][]-o 

Of course I will keep testing and let you know if anything else pops up but, so far so good.

God Bless,

Bob


----------



## tonewill (Sep 22, 2011)

I've been testing this too, great to have functions with returns.

One problem I've found so far is the new compiler gives "unknown function" error with this:

* SET_CONDITION(NO_SYS_SCRIPT_GROUP_START)*

A horizontal scroll-bar in the editor would be nice also if you ever get the time  .

Al the best,
Barry.


----------



## Big Bob (Oct 11, 2011)

Hi Nils,

I found another problem with the new compiler. In *on_init_xxx *functions which are of course implicitly global, the *'local'* keyword overide doesn't seem to work anymore. This still works OK with the old compiler option on.

I've added this to my summary post (2 posts back in this thread) and also included the SET_CONDITION problem that Barry reported.

Do you have any idea when some of these problems might be addressed? The SET_CONDITION thing work-around isn't too much fun. :lol: 

God Bless,

Bob


----------

