Log In  

so i'm starting to get near the often mentioned but never explained token limit in my project. and while I think I might fit everything i want into this project, I think I need to keep tokens in mind from the get go of my next project.

so I have some questions:

what is a token? what counts towards the token limit? can i look at my existing code and try to optimize it with token limit in mind? and also do you have any tips & tricks while developing with this in mind?

P#23272 2016-06-20 03:39 ( Edited 2016-06-20 15:58)

each word is a token a pair of parens,brackets,braces count as one token, a string is one token, punctuation is generally one token. "local" does not count as a token. "end" does not count as a token

tip 1: don't repeat yourself (there are some exceptions)

find similar code you use frequently and make a function and use that instead, the more you use it the better the return.

tip 2: if you use lots of attributes, it might be worth making a local first to save tokens:

foo = a.x + b.x + c.x + d.x
bar = a.x + b.x + c.x + d.x
baz = as.x + b.x + c.x + d.x

= 39 tokens

local ax,bx,cx = a.x,b.x,c.x
foo = ax + bx + cx + dx
bar = ax + bx + cx + dx
baz = ax + bx + cx + dx

= 37 tokens

tip 3: (advanced) use operator overloading

if you use a lot of vectors you can save a heap of tokens by making a vector class with operator overloading

so instead of

pos.x += vel.x
pos.y += vel.y

you can do

vec = {}
vec.__index = vec
function vec:__add(b)
   return vec.new(self.x+b.x,self.y+b.y)
-- any other vector functions you need
function vec.new(x,y)
    local v = {x=x or 0,y=y or 0}
    return v

pos += vel
P#23273 2016-06-20 04:17 ( Edited 2016-06-20 08:17)

There's a really good write up in one of the Pico 8 fanzines about tokens (the first one I think?).

Basically a token is one programming element in Pico 8 LUA. For example, keywords like "FOR" "FUNCTION" "IF" "END" all constitute one token each. Variable names you create are all one token, no matter how long they are. Operators like "+", "=", are all one token. Operators written with multiple characters like "*=" count as a single token too.

The easiest way to learn about them is to write code while paying attention to the token counter in the bottom right of the code editor.

P#23274 2016-06-20 04:22 ( Edited 2016-06-20 08:22)

thanks for the input!
@impbox your tips will be of great help, especially the first one!
@mrh i write all my code in atom. i should check to see if there is any way to display token count in it :)

also this reminds me i should read the fanzines...

P#23276 2016-06-20 05:36 ( Edited 2016-06-20 09:36)

Like the other replies said, a token is basically "something that the text parser considers functional". Think of a token as a delimiter or an instruction. One line of code such as "playerPos = { x=10, y=20 }" could consist of about 10 tokens or so.

Tokens are a feature of Lua, not Pico8. Remember that by design, you're limited to 8192 tokens, which must all be under 65536 characters in length total.

P#23288 2016-06-20 11:47 ( Edited 2016-06-20 15:47)

Speaking of operator overloading... does anyone know if Lua does return value optimization?

Where RVO would be something like taking this:

function x() return {x=0} end

v = x()

And, under the hood, producing code that actually works like this:

function x(rv) rv.x=0 end


The goal being to avoid constructing temporaries and moving them around.

P#23289 2016-06-20 11:58 ( Edited 2016-06-20 16:09)

[Please log in to post a comment]

Follow Lexaloffle:          
Generated 2023-01-28 07:47:51 | 0.013s | Q:16