4

so i'm starting to get near the often mentioned but never explained token limit in my project. and while I think I might fit everything i want into this project, I think I need to keep tokens in mind from the get go of my next project.

so I have some questions:

what is a token? what counts towards the token limit? can i look at my existing code and try to optimize it with token limit in mind? and also do you have any tips & tricks while developing with this in mind?

P#23272 2016-06-20 03:39 ( Edited 2016-06-20 15:58)

each word is a token a pair of parens,brackets,braces count as one token, a string is one token, punctuation is generally one token. "local" does not count as a token. "end" does not count as a token

tip 1: don't repeat yourself (there are some exceptions)

find similar code you use frequently and make a function and use that instead, the more you use it the better the return.

tip 2: if you use lots of attributes, it might be worth making a local first to save tokens:

 ```foo = a.x + b.x + c.x + d.x bar = a.x + b.x + c.x + d.x baz = as.x + b.x + c.x + d.x = 39 tokens local ax,bx,cx = a.x,b.x,c.x foo = ax + bx + cx + dx bar = ax + bx + cx + dx baz = ax + bx + cx + dx = 37 tokens ```

if you use a lot of vectors you can save a heap of tokens by making a vector class with operator overloading

 ```pos.x += vel.x pos.y += vel.y ```

you can do

 ```vec = {} vec.__index = vec function vec:__add(b) return vec.new(self.x+b.x,self.y+b.y) end -- any other vector functions you need function vec.new(x,y) local v = {x=x or 0,y=y or 0} setmetatable(v,vec) return v end pos += vel ```
P#23273 2016-06-20 04:17 ( Edited 2016-06-20 08:17)

There's a really good write up in one of the Pico 8 fanzines about tokens (the first one I think?).

Basically a token is one programming element in Pico 8 LUA. For example, keywords like "FOR" "FUNCTION" "IF" "END" all constitute one token each. Variable names you create are all one token, no matter how long they are. Operators like "+", "=", are all one token. Operators written with multiple characters like "*=" count as a single token too.

The easiest way to learn about them is to write code while paying attention to the token counter in the bottom right of the code editor.

P#23274 2016-06-20 04:22 ( Edited 2016-06-20 08:22)

thanks for the input!
@impbox your tips will be of great help, especially the first one!
@mrh i write all my code in atom. i should check to see if there is any way to display token count in it :)

also this reminds me i should read the fanzines...

P#23276 2016-06-20 05:36 ( Edited 2016-06-20 09:36)

Like the other replies said, a token is basically "something that the text parser considers functional". Think of a token as a delimiter or an instruction. One line of code such as "playerPos = { x=10, y=20 }" could consist of about 10 tokens or so.

Tokens are a feature of Lua, not Pico8. Remember that by design, you're limited to 8192 tokens, which must all be under 65536 characters in length total.

P#23288 2016-06-20 11:47 ( Edited 2016-06-20 15:47)

Speaking of operator overloading... does anyone know if Lua does return value optimization?

Where RVO would be something like taking this:

 ```function x() return {x=0} end v = x() ```

And, under the hood, producing code that actually works like this:

 ```function x(rv) rv.x=0 end x(v) ```

The goal being to avoid constructing temporaries and moving them around.

P#23289 2016-06-20 11:58 ( Edited 2016-06-20 16:09)