Hello! I thought I'd try to (slightly) clean up and post some utility functions I've found useful while working on RP-8. Maybe they'll be useful to others, maybe not. Let's see!
There are three possibly-useful functions here, plus their support code. Note that in the name of token conservation, all of these functions have essentially no error checking and can blow up in various exciting ways if you feed them bad data (or if they have bugs). The code also isn't the cleanest - maybe I'll tidy it up eventually, but I don't think it's completely unreadable right now. Anyway, we've got:
- stringify(), 114 tokens. This serializes structured data into a string - RP-8 uses this to save songs. Escapes binary. Both the string format and the binary escaping are completely nonstandard, but the string format at least looks vaguely similar to Lua literals.
- parse(), 286 tokens (can be cut to 246 if you don't want eval). This takes a string and transforms it into structured data. RP-8 uses this to load songs, as well as to set up some of its internal data. Uses the same weird format and binary escaping as stringify(), although it also supports some variations and syntactical sugar. You can probably cut more tokens if you don't need to support large input strings or binary unescaping.
- eval(), two versions - 428 tokens for interpreted, and 556 for "compiled", plus you need parse(). (Very loose use of the word "compile" here...) This evaluates a script in a vaguely LISP-formatted mini-language with Lua-ish behavior, and returns the result. RP-8 uses this to save tokens by encoding bulky logic that doesn't need to run fast, like UI init. This can help save tokens. Note that the token costs here are pretty squishy, since they depend on what builtins you want to define.
While this eval may help you save tokens, if you're really serious you should probably consider external build tools and bytecode. (For more on this, check out what @carlc27843 did with Picoscript in his unbelievable Nebulus cart.) RP-8 uses a condensed version of the interpreted eval - 382 tokens, but that small token savings causes some questionable behavior with multivals and closures, so I'm posting something better behaved, if less well tested, here.
The cart contains these three functions, some tests for the two versions of eval (none for parse or stringify, sorry), and a small and highly artificial performance comparison between the two versions of eval and native Pico-8 Lua code.
Examples
Here is a real-world example of parse()-formatted data, including using backticks to embed data from eval():
{
pat_store={},
tick=1,
playing=false,
base_note_len=750,
note_len=750,
drum_sel=bd,
b0_bank=1,
b1_bank=1,
dr_bank=1,
song_mode=false,
samp=(0),
patch={},
pat_seqs={},
pat_status={},
tl=`(timeline_new $default_patch),
pat_patch=`(copy $default_patch),
} |
And here is an example of eval()-executable script:
(set paste_state (fn () (audio_wait 2) (let pd (stat 4)) (if (not (eq $pd "")) (seq (set state (or (state_load $pd) $state)) (@= $seq_helper state $state) )) )) |
... which is equivalent to:
paste_state=function() audio_wait(2) local pd=stat(4) if pd != "" then state=state_load(pd) or state seq_helper.state=state end end |
Script usage
The script has mostly LISP-ish syntax, though the behavior is more Lua-ish, including (hopefully correct) support for multivalues in function argument lists and returns.
The biggest syntactical quirk is that you must prepend the character $ to each variable name you want to be looked up instead of treated as a literal string. Since the parser operates independently of eval and does not pass any side information about the strings it finds, the alternative (all strings are treated as variable lookups) would require double-quoting of some strings, like "'hello world" or (' "hello world") ... I found this even more distasteful, so the $ prefixes are the fix for now. The one place where $ is not required is function invocations: you can't call a string, so it's clear that a lookup is required. If you find this confusing, I don't blame you, but perhaps some of the tests in the cart will make things clearer?
Possibly-incomplete list of script forms/builtins/keywords/whatever (I'm not a LISP person...):
- (e1 e2 ... en) - if e1 evaluates to a function or the name of a function, call that function with the values of e2 through en as its arguments. For example,
(print "hello world" 0 0 8)will print "hello world" in red. The argument evaluation also respects Lua's multival semantics, or at least it mostly should. - (fn (a1 a2 ...) e1 e2 ... en) - define a function. The args list is implicitly quoted. The function returns the value of en, the last expression.
- (seq e1 e2 ... en) - evaluate expressions sequentially and return the value of the last one, en.
- (' e1) - quote a value. e1 will be returned literally, without being evaluated at all. Use this to keep a table from being interpreted as an expression.
- (if e1 e2 e3) - if e1 evaluates to a truthy value, evaluate and return e2, else evaluate and return e3.
- (for e1 e2 e3) - executes
for i=e1,e2 do e3(i) end - (set e1 e2) - sets the global named by the value of e1 to the value of e2. (Maybe a better name for this one?)
- (let e1 e2) - sets the local variable named by the value of e1 to the value of e2. Note that this behaves like Lua's local assignment and not like
letin many LISP dialects. - (@ e1 e2) - property access. Returns e1[e2]. There is an alternative form (@ e1 e2 e3) that returns e1[e2][e3].
- (@= e1 e2 e3) - Sets e1[e2]=e3. Does this name make any sense with
setandlet? No, it does not. - (~ e1 e2) - returns e1-e2. Uses ~ instead of - because - looks like a number to the parser.
- (cat e1 e2) - returns e1..e2
- (len e1) - returns #e1
- +, *, eq, gt, or are all present. Many other operators and keywords are missing. (There's no
and, nowhile, etc.) They're not hard to add if you need them! If you care about tokens you should also delete anything you're not using.
What is this "compiled" nonsense?
Ok, yeah, I maybe should have a better word for this - but I don't. Essentially, the script can be evaluated in two different ways.
First, it can be interpreted straightforwardly by walking the parse tree on each execution, dispatching to different functions or builtin operators as required. This can be done in relatively few tokens (especially if you ditch multival support), and mostly doesn't care what's a builtin vs. an external function.
Second, it can be executed in two steps - first, the parse tree can be walked to produce a closure. This closure invokes other similar closures, each of which takes a dictionary of locals as an argument. The reason I've been using the word "compile" for this step is that it is a transformation on the code to a new form based only on its statically observable properties, with no knowledge of runtime values. The second execution step is then to invoke the top-level closure (this may be done many times with different locals).
The "compiled" method was motivated by @carlc27843's Picoscript, I definitely wouldn't have thought to try it otherwise.
This second method is much faster than the naively-interpreted version - about 4x as fast in the small test that's in this cart. How much faster it might be in any other case depends on how much the script can use builtins vs. function calls. Function call overhead will be very similar - most of the arg and return packing/unpacking logic is essentialy identical. So if you're mostly just dispatching function calls, both versions should perform about the same, however, if you can mostly stick to builtins, you could potentially see bigger speedups.
Requests
If you have clever speedups, bug reports, bug fixes, or ways to save tokens, let me know! There have to be all kinds of horrible bugs lurking in here.
Caveats
The script eval functions are larger and slower than the ones I use in RP-8, in order to give them more consistent behavior with regard to variable scopes and multivals. If you're feeling adventurous you can modify the code to take some shortcuts here.
[Please log in to post a comment]
