Log In  


Would you mind tweaking the token counter so that these two statements...

x = -1
y =  1

...use the same number of tokens each?

It's frustrating that we lose a token to the unary minus on number literals.

I could rewrite that code with hex to save a token...

x = 0xffff
y = 1

...but that's ugly, non-portable code.

Basically, just elide any unary minus token if it's being applied to a number literal.


P#49267 2018-02-13 22:19 ( Edited 2018-02-14 19:12)


The problem is that the tokenizer doesn't know yet if the minus is unary or part of a subtraction. It's the parser who checks it, but this is the step after tokenizing.

P#49273 2018-02-14 06:14



I know the original Lua tokenizer will do this regardless of what I want, but @zep doesn't actually count every token that's produced. Some are ignored for the sake of pragmatism within the fantasy console.

P#49287 2018-02-14 17:23


PS: I believe the current version of Lua actually folds constant expressions if there's no loss of information, e.g. x=-2*-5+2 will produce x=12. However, @zep built PICO-8 on an earlier version of Lua (5.1, I think). This is why we have to use functions for binary operations like BAND() instead of using operators like current Lua can do.

P#49288 2018-02-14 17:26 ( Edited 2018-02-14 17:28)


As an aside, parens probably ought not to be tokens either, since really all they're doing is indicating order of operations, which means the expression needs to be evaluated in a certain order, but they don't change the number of operations performed at all.

Functionally, these two statements perform exactly the same math:


Ideally those would both reduce to the same number of tokens.

That's kind of a gray area, though, since tokens in a truly tokenized language would probably include parens, but Lua is semi-compiled and does some very simple optimizations, so it's hard to say where to draw the line. I guess that's up to @zep.

I guess what I'm saying is, if we were writing games on a real retro platform, we'd probably be using assembly, in which case parens have no meaning. There are no opcodes for parens. ;) It's all down to instruction ordering. But if we're following PICO-8's higher-level language format, then we'd be using BASIC, in which case the tokens would include parens.

My thing is that sometimes the token limit seems to get pushed for reasons that don't make a lot of sense. The token limit itself makes sense, but it really ought to depend on the amount of work the code does, not the amount of source code. That's why, sensibly, comments take no tokens, even though in a real BASIC-style computer, they would.

Maybe I should say anything that qualifies as syntactic sugar should count in terms of its actual function, not its original source text. By and large, that does seem to be the sentiment for most existing token counts. Just not in these examples.

P#49291 2018-02-14 19:12 ( Edited 2018-02-14 19:18)

Log in to post a comment


New User | Account Help
:: New User
About | Contact | Updates | Terms of Use
Follow Lexaloffle:        
Generated 2018-02-18 01:02 | 0.124s | 1572k | Q:13