Log In  


Hello, I am hitting token count with my game so often - it is irritating. I feel like I am fighting a war of having clean code vs having less tokens.
(I understand that pico8 is for small games and first step should be cutting features though)
I am not sure if something is wrong or am I expecting too much and trying to use pico8 like modern framework.

In the docs is stated:
One program can have a maximum of 8192 tokens. Each token is a word (e.g. variable name) or operator. Pairs of brackets, and strings count as 1 token. commas, periods, LOCALs, semi-colons, ENDs, and comments are not counted.

Let's look at this code:
(taken from my game, but shortened and altered - don't expect this to work! I am not saying that provided code is greatest piece of clean, descriptive code!)

It draws (on local space - screen) piece of text, npc will say to the player - after delay, with black background to ensure readability.

  draw_say=function(this,x,y)
    x=this.x-x
    y=this.y-y
    palt(0,false)
    foreach(this.draw_message_queue,function(obj)
        local deltatime=gl_app_time.timeelapsed-obj.time
        local text_len=#obj.m
        local text_size=text_len*2
        local text_time=text_len/4.0
        local x1=x-text_size-2
        local x2=x+text_size+2
        local y1=y-12
        local y2=y-4

        if deltatime<text_time then
          rectfill(x1,y1,x2,y2,0)
          print(obj.m,x1,y1+2,obj.color)
        end
      end)
    palt()
  end,

This code isn't perfect, there are still magic numbers or non descriptive piece of code like y1=y-12 - what this '-12' means?
But most important things (to me) are explained in local variables like: text_size (how much space string will take on screen) and text_time (how long text should be printed based on length of screen)
All of this pasted into empty cart gives token count: 100

If I change lines:

        local text_len=#obj.m

        local text_size=text_len*2
        local text_time=text_len/4.0

to:

        local text_size=#obj.m*2
        local text_time=#obj.m/4.0

I get token count: 99

This wasn't bad change as '#obj.m' still can be quite descriptive (even more if this was '#obj.text'), but shows that using locals is quite wasteful if your value isn't reused more. So going with that logic, let's change more code:

If I change lines:

    x=this.x-x
    y=this.y-y

        local deltatime=gl_app_time.timeelapsed-obj.time

        local text_len=#obj.m
        local text_size=text_len*2
        local text_time=text_len/4.0
        local x1=x-text_size-2
        local x2=x+text_size+2
        local y1=y-12
        local y2=y-4

        if deltatime<text_time then

to:

        local x1=this.x-x-#obj.m*2-2
        local x2=this.x-x+#obj.m*2+2
        local y1=this.y-y-12
        local y2=this.y-y-4

        if (gl_app_time.timeelapsed-obj.time)<#obj.m/4.0 then

I get token count: 95
This comes from removing text_size, text_time and deltatime, but didn't get anything from removing lines 'x=this.x-x' and 'y=this.y-y'. (which make x and y local space btw - yep bad naming from my part)

Code isn't so bad, yet, but let's compress this even more:

  draw_say=function(this,x,y)
    palt(0,false)
    foreach(this.draw_message_queue,function(obj)
        local x1=this.x-x-#obj.m*2-2
        local y1=this.y-y-12

        if (gl_app_time.timeelapsed-obj.time)<#obj.m/4.0 then
          rectfill(x1,y1,this.x-x+#obj.m*2+2,this.y-y-4,0)
          print(obj.m,x1,y1+2,obj.color)
        end
      end)
    palt()
  end,

And now we have token count: 89!
(Have fun decoding this if you didn't wrote this/came back after long break.)

11 tokens.. now we are talking!
But why the f*k you only removed x2 and y2? Because those are only used once. But lets remove all those:

  draw_say=function(this,x,y)
    palt(0,false)
    foreach(this.draw_message_queue,function(obj)

        if (gl_app_time.timeelapsed-obj.time)<#obj.m/4.0 then
          rectfill(this.x-x-#obj.m*2-2,this.y-y-12,this.x-x+#obj.m*2+2,this.y-y-4,0)
          print(obj.m,this.x-x-#obj.m*2-2,this.y-y-12+2,obj.color)
        end
      end)
    palt()
  end,

I get token count: 99!

So this shows that if you have more than one operation and you reuse it at least once, you should use variables for that.

You may ask, well isn't X tokens worth more readability? Answers is yes, but this is only one function... This starts to add up when you have 50 functions (50x11=550 saved tokens!) and you hit token count and want to finish feature or at least run your game to debug... And saving means that your code will get more and more ugly.

The thing that irritate me, even more than non descriptive code, is the inconsistency in such actions.

I think there is a rule of thumb: if you use some number (especially with operator and variable) more than 3 times it is worth it to put it in separate variable. If you have few operations for one variable, use locals if you use result more than once. If you have even more operations use variables.

Which leads to functions that sometimes have local variables and sometimes don't and sometimes have only some variables for magic numbers...

So what do you think? Is this is me trying use pico8 wrong?

P.S. I understand that pico8 is modelled after old hardware, but do we seriously need to make everything hard because of old times? Embedded code editor, while still pain to use, is reminiscent of old editors and somehow gets modern features to make writing code simpler - why can't this happen with token count as well?

P.P.S. Ability to run game with having more tokens would also allow me to write clean code before I finish everything. Which means I would just refactor code before release. This isn't perfect, but would be good start.

2


IMO simply ditch OOP and use globals and so on. Size (and other) optimization is never about the details anyways.

Makes sense to me that the code should be similarly challenging as the audio and graphics are. At least I find if very satisfying when you have trouble with code size and then find a way to fit one more feature in.


Pico8 was never intended for what you are trying there.
If you plan something big in this environment you achieve the most if you throw out good coding standards or just start to "cheat".

When i wrote DungeonExplorer (CSDb DungeonExplorer) for the Forum64 BASIC compo back in 2010 i wrote my own reliner and compactor to stay in the limits as my goal was a bit ambitiousy. The Compo was, in a nutshell, "Write a game for the C64 in BASIC without using ASM in 160 blocks. You can use any amount of files but you have to stay in that limit."
I squeezed a game engine and an editor into the same program, you have ~38 kByte for BASIC on a C64 which means programm AND variables mind you, and after loading everything had ~4 kByte left. Sadly never got around to create a bigger and better version.

So, the point is, if you want to stay with pure Pico8 and your project still is in theory too big you will have to cheat a bit. With 1.10 even the web exported version can now use several carts for data. And you can surely find some Lua compressor, so that you can write good code outside of the builtin IDE.

Or change the framework. ;)


Thank you for your input.

@kometbomb:
Makes sense to me that the code should be similarly challenging as the audio and graphics are. At least I find if very satisfying when you have trouble with code size and then find a way to fit one more feature in.

Yeah, if we look at the code as a challenge that this makes sense. I guess I don't like challenging myself that much in this region. Your advise is good though.

@bastedfurry: I think you are right. At this point I guess I will prolong 'cheating' as long as I can. :p
Wow! Impressive feat (with DungeonExplorer).

At first I loved idea that, not only graphics and cpu but, also code limit will force me to leave only necessary features, but it happens to be two-sided sword for me. :p I think I still overscoped and getting salty about it. Will cut features till I get to point where I can't cut anything and then go with cheat and compressors route.

P.S. Still little salty about not having 'dev preview' which would allow you to start game with exceeded token count.


I guess if you have a 10000 token game and want to "dev preview" it, it would still mean it will be too big even after optimizing. I have been at nearly 8000 tokens in our latest game for a while, constantly adding stuff and optimizing stuff and I feel more alive than ever. :)

IMO it's not a shame if you have big plans but need to cut back because of reasons. That's a thing I have learned while making games and PICO-8 is a godsend in that it forces you to say things like "ok, the cart is now full, I guess it's finished".


You have a point. I guess it depends on view, but I am really against premature optimization, and reducing token count see like this. But considering pico8 limits, it isn't premature anymore - so no problem here. ;)
I prefer to try out idea first, without worrying about performance (or token count in this case), as I only try maintain how code looks - so it is easier to read, and later clean up if its working... But this isn't language for that. I made peace with that.

I also like a feeling of reducing token count and is a good break from something, but don't like to be forced to do that - like ringing alarm.

rriiiing stop doing what you are doing, you crossed token count by 3. ring ring debugging feature? f*k you, go clean some code.

I know, I know I

edited:should
shouldn't get to situation where i have 8182/8192 tokens... I guess I don't like remembering about this...

Anyway. Obviously this is my first game in pico8, probably with next one such problems would be non existent as I would write more token count friendly or constantly cleaning.

I wasn't prepared for all of this. Yeah, cutting features isn't something to be ashamed of.


The thing that has seemed to work best for a few other folks on the BBS over the last year and a half that I've been following them is to, for any sort of really ambitious project, treat PICO-8 as a protoyping phase with the expectation that you can go over to something like Love2D to do a full unfettered version after you've worked out some ideas.

So: embrace some messy code and some shortcuts and some limiting compromises while you try to get the core bits worked out. Strip down, not as a gut-wrenching design concession, but as a choice to defer that bit of design for the next iteration in a similar but less constrained context.


Thank you. I guess you phrased better, what others told already.
You are right of course, all of you.

Well, my plan was to check some ideas I had in short but functional game and then move on to something bigger with strong typed language (prefer those actually) to make next part of story or remake with flesh-out ideas... I made good assumptions with graphics/sound, cpu and memory conspution.. But not with mechanics itself.

As said earlier, I overscoped even though this was heavily cutted version. (I don't have, yet, skill to make realistic scope) I guess this is why I got so angry at pico8, as at one point I looked at this as doomed cause... but as emotions calmed I am welcoming this as it forces me to really evaluate stuff and only keep the most important things of all. And if I only check 1 thing out x I planned, this isn't end of the world.

Btw. does any of you have fav. compressor that is good with pico8? I will of course scourge this bbs, but maybe you have experience with any of those.


There's not really such a thing as a "compressor" that will reduce token count. Tokens are meaningful units to a language, and in general you can't remove tokens without changing the meaning. You might be thinking of a "minifier," of which picotool has one. A minifier by definition does not change the token count. It can reduce the uncompressed char count and may or may not reduce the compressed char count. It's very rare to hit the uncompressed limit before you hit the token limit or the compressed limit, and in all other cases it just makes the code unreadable for no benefit.

There are a few automated transformations that can reduce the token count. The simple ones are uninteresting and don't save much. For example, any function call that passes exactly one argument whose type is a string or table can omit the parens for a savings of one token. Pico-8 discounts certain tokens to avoid these little tricks becoming ugly habits. (That's why "local" is free, for example.)

The fancier ones are more fun: constant inlining, build-time literal expression evaluation, dead code elimination. That'd allow you do to something like:

use_final_title = false

if use_final_title then
  title = "the fall of rome"
else
  title = "rome: that one place that fell"
end

title_xpos = (128 - #title * 4) / 2

in your source file and end up with

title = "rome: that one place that fell"
title_xpos = 4

in your final cart. I have plans to add such features to the picotool build tool, but I'm not making promises as to a schedule. :)


... Incidentally, the only reason Pico-8 has a token limit in the first place is that it started with a character limit and people were minifying their code by hand to pack in more code. It was a bad habit that Pico-8 didn't want to encourage, hence tokens.

The compressed char limit still matters to the cart PNG size, and it discourages egregious workarounds with strings (though string packing within the limits is fun and encouraged :) ). I can't say if the uncompressed limit still has a purpose, but it's been increased enough that tokens and compressed chars are the dominant limits in nearly all cases. See the chart in the picotool announcement thread.


I agree the token limit can be frustrating. It's nice that local is free, but I really wish "self" would be free. Using tables as objects really keeps my code more maintainable. In my current project "self" appears about 450 times in functions like this:

function Ship:is_visible(player_ship_position)
  self.screen_position = self.sector_position - player_ship_position + screen_center
  return self.screen_position.x < 128 and
         self.screen_position.x > 0 and
         self.screen_position.y < 128 and
         self.screen_position.y > 0 
end

I also use pretty descriptive variable names which now requires a minifier to get it under the compressed character count.


@dddaaannn:
You could also have a look after if(foo==bar)then constructs that get created out of habit writing stuff in C-Style languages.



[Please log in to post a comment]