Log In  
Page:
1
2

I'm pleased to announce picotool, a set of tools and Python libraries for manipulating Pico-8 game files. picotool can read .p8.png and .p8 files, and can write .p8 files.

To get picotool and learn how to use it, visit the picotool Github repo.

The p8tool command is a multi-tool with multiple functions. You can run it like this:

% p8tool stats helloworld.p8.png
hello world (helloworld.p8.png)
by zep
version: 0  lines: 48  chars: 419  tokens: 134

picotool currently includes the following tools:

  • build: builds a cart from pieces of other carts
  • stats: a statistics reporting tool
  • listlua: prints the Lua region of a cart
  • writep8: converts a .p8.png cart to a .p8 cart
  • luafmt: writes a new .p8 cart with Lua rewritten with regular formatting, to make it easier to read
  • luamin: writes a new .p8 cart with Lua rewritten to use a minimal number of characters, to make it difficult to read
  • luafind: searches for lines of Lua code containing a string or matching a pattern
  • listtokens: prints a visualization of Lua tokens
  • printast: prints a visualization of the Lua parser abstract syntax tree (AST)

The Python library includes a hand-written Lua parser with support for Pico-8 features, with API access to the token stream and abstract syntax tree. Additional modules provide access to the graphics, map, sound, and music data. All game data can be transformed or created with Python code and written to a .p8 or .p8.png cart file.

I've included the Lua minifier (luamin) as an example tool, though I don't actually think it's a good idea to use it. Statistically, you'll run out of tokens before you run out of characters. The Pico-8 community benefits more from published carts with code that is easy to read and well commented. All luamin does is make the code difficult to read, without much benefit.

Here is a chart of the character count, minified character count, and token count of 496 carts that have been posted to the BBS, relative to the Pico-8 maximums:

(Here's the spreadsheet.)

As shown, even un-minified code tends to stay below the token count, percentage-wise. Minifying reduces the (uncompressed) character count to 65% of the original size on average.

The inspirational use case for this library is actually luafmt, which re-formats the Lua code of a cart to be easier to read. The current version of this tool is simple and only adjusts indentation. There is much more luafmt can do by analyzing the AST, but this is not yet implemented.

As of today, this is very much a v0.1 early release. There are known issues regarding parsing and token counting, and the library APIs are incomplete and messy. This project has already gotten away from me a bit, so I'm not sure how far I'll take it. But if you do want to build against it, patience with non-backwards compatible changes will be appreciated. :)

Let me know what you think! Thanks!

Update 2015-10-29: Added luafind tool. Refactored AST walking code to make it easier to build tools that read or transform the parser tree.
Update 2016-10-17: Added build tool. Added support for updating and creating .p8.png cartridge files.

P#15984 2015-10-29 04:20 ( Edited 2018-10-02 06:46)

That is very handy! Thanks man!

P#16024 2015-10-30 06:32 ( Edited 2015-10-30 10:32)

Do you know if compacting the code saves on tokens? Sort of like javascript compression, where the code becomes basically unreadable/usable but saves on space?

P#16032 2015-10-30 13:49 ( Edited 2015-10-30 17:49)

hseiken: By definition, minifying the code as luamin and JavaScript minifiers do does not change the token count. They only reduce the number of characters used to represent long tokens, such as by renaming variables. Pico-8's token count already excludes comments and whitespace, and only counts each name as a single token, regardless of its length. (Try the listtokens tool for an illustration of how tokens are defined and counted.)

There may be a few token optimizations that a tool could do automatically, but they'd be pretty smart compiler-style optimizations that actually rewrite the semantic structure of the code to a shorter equivalent. I'm thinking about building a tool that does dead code elimination, for example. (No promises, I might not have time, but it's the kind of thing you can do with access to the AST.)

P#16037 2015-10-30 17:28 ( Edited 2015-10-30 21:31)

Hey this is pretty useful. I added it to the Awesome PICO-8 list. :)

P#16195 2015-11-03 18:50 ( Edited 2015-11-03 23:50)

felipebueno: Thanks!

P#16200 2015-11-04 00:47 ( Edited 2015-11-04 05:47)

@dddaaannn: bravo! Great work.

P#16285 2015-11-05 19:20 ( Edited 2015-11-06 00:20)

matt: Thank you!

P#16295 2015-11-06 02:08 ( Edited 2015-11-06 07:08)

I'm pleased to announce that picotool now supports writing .p8.png files, including code compression.

I've also started a new build tool. Its features are modest for now: it can replace sections of a cart (Lua, spritesheet, sfx, etc.) with sections from other carts, and it can also read Lua code from a .lua file and add it to a cart. It can also create carts from scratch and erase sections.

I have plans for more compelling features for the build tool (require() support, dead code elimination). My summer sabbatical is coming to an end so I might not get to it right away, but I thought I'd push what I have so far.

The new .p8.png code includes the code compression routine, the dual of the decompression routine that we've had for a while. Thanks to zep for providing the last few missing pieces!

Enjoy!

P#31209 2016-10-19 01:29 ( Edited 2016-10-19 05:33)

[Update: fixed] Oof, just noticed picotool's compressor is compatible but not efficient, and is not producing the same result as Pico-8's compressor. It's bad enough that I don't think this is quite ready for the build workflow of a real (large) game. I'll update again once I've fixed it, but I'm mentioning it in case I don't get around to it right away. (My vacation is over! :( )

Tracking bug: https://github.com/dansanderson/picotool/issues/7

P#31486 2016-10-22 18:33 ( Edited 2016-10-23 21:45)

I made a quick fix by temporarily replacing my compression code with a direct port of the Pico-8 version. It'll be worth going back and looking for differences, if only to speed it up. But this'll do for now.

P#31584 2016-10-23 17:46 ( Edited 2016-10-23 21:46)

Dear dddaaannn.

I am trying to use your picotool to minify my game. Currently I have:
65302/65536 char count and 8136/8192 token count.
When I am trying to use luamin on my cart I get:

C:\Python34>python p8tool luamin saveyourself.p8
saveyourself.p8 -> saveyourself_fmt.p8
warning: token count 29022 exceeds the Pico-8 limit of 32768Traceback (most rece
nt call last):
  File "p8tool", line 8, in <module>
    sys.exit(tool.main(sys.argv[1:]))
  File "C:\Python34\pico8\tool.py", line 512, in main
    return args.func(args)
  File "C:\Python34\pico8\tool.py", line 349, in do_luamin
    return process_game_files(args.filename, luamin, args=args)
  File "C:\Python34\pico8\tool.py", line 210, in process_game_files
    procfunc(g, out_fname, args=args)
  File "C:\Python34\pico8\tool.py", line 242, in luamin
    lua_writer_cls=lua.LuaMinifyTokenWriter)
  File "C:\Python34\pico8\game\game.py", line 746, in to_file
    self.to_p8_file(outfh, *args, **kwargs)
  File "C:\Python34\pico8\game\game.py", line 659, in to_p8_file
    for l in self.gfx.to_lines():
  File "C:\Python34\pico8\gfx\gfx.py", line 95, in to_lines
    yield bytes(newdata).hex() + '\n'
AttributeError: 'bytes' object has no attribute 'hex'

I have ton of text packed into loooong strings if that helps.
Please help me, as I can't even export my game to html at this point.

Thanks in advance!

P#34133 2016-12-25 16:37 ( Edited 2016-12-25 21:37)

@dagondev: Can you share the cart with me so I can troubleshoot?

It's likely that the cart just doesn't compress under the limit and there's not much to be done, but I'd like to clean up the error messages at least. If we're lucky, minification will get it just under the wire, but no guarantees. :)

P#34283 2016-12-27 04:03 ( Edited 2016-12-27 09:03)

Hey, I wrote followup at issue page in github: https://github.com/dansanderson/picotool/issues/9

Here is the code: https://bitbucket.org/dagondev/save-yourself/src

I ended up minifing my code by hand, so no pressure now... But if you get this working, that would mean I could fix bugs on non minified version - so I would be grateful regardless. :)

P#34284 2016-12-27 04:18 ( Edited 2016-12-27 09:18)

Replied on the bug but will summarize here for reference: I need to add support for glyphs and labels before I can properly parse your cart. These are likely unrelated to the symptom you're seeing but I can't repro otherwise, and they're notable shortcomings regardless. I'll keep the bug open and revisit once I've had a chance to make these changes. Thanks for the report!

P#34319 2016-12-27 17:59 ( Edited 2016-12-27 22:59)

@dagondev:
what os are you using?
Bacause I have the same problem on andriod's qpython and I solve it.
Maybe you need to use new python 3 version.

P#34425 2016-12-29 06:56 ( Edited 2016-12-29 11:56)

I'm having a strange problem; I'm attempting to use writep8 on Mac OS Sierra/Python 3.5 to convert p8.png carts to .p8 carts (to be edited in Sublime Text). However, every time I've attempted to do so, p8tool reports:

../carts/cartname.p8.png -> ../carts/cartname_fmt.p8.png

and sure enough, when I check the resulting output file, it's still a *.p8.png cart, and attempting to read it with cat or Sublime only yields byte data. Any ideas what could be wrong?

EDIT: The most recent cart this failed on was the TinyTV Jam base cart, found here: https://www.lexaloffle.com/bbs/?tid=28480

It also failed on the Emily Quest cart found here: https://www.lexaloffle.com/bbs/?tid=2294

P#35981 2017-01-18 02:19 ( Edited 2017-01-18 07:25)

I'm looking forward to trying out luamin over the weekend. I must be one of the rarer cases that have hit the character limit before the token limit. I am parsing long strings in order to reduce token count which is where most of it would be going.

P#37319 2017-02-09 18:31 ( Edited 2017-02-09 23:31)

Hi Dan, I understand if you aren't interested in supporting this. But just on the off chance;

I have just run p8tool luamin on my pico quest card #37305# (found here: https://www.lexaloffle.com/bbs/?tid=28574) and got a syntax error when running (on startup):

syntax error line 99
....
'then' expected near 'cg'

P#37324 2017-02-10 04:45 ( Edited 2017-02-10 09:45)

radcore: Good find! I filed this as a bug: https://github.com/dansanderson/picotool/issues/11

I'm going to blame Pico-8 for this one because it appears to be a shortcoming in its short-form "if" statement that luamin doesn't take into account. luamin will collapse space between a non-word non-space character and a word character. This is valid for Lua, but breaks Pico-8's short-if, which is implemented as a preprocessor replacement.

x=true
y=(0)
if (x) y=1
print(y)

minifies to

x=true y=(0)if (x) y=1
print(y)

Pico-8 doesn't recognize the "if" as a short-if and fails to insert the "then" and "end" keywords before parsing the Lua. (picotool's own parser doesn't see the problem because I implemented short-if as a formal part of the grammar before I realized that Pico-8 was using a preprocessor replacement.)

I see a similar issue for a space before a "local" keyword being collapsed and causing problems. I'm less sure of the cause within Pico-8 but the solution is similar: it needs leading spaces preserved.

You can work around the short-if issue by avoiding short-if (using if-then-end) in problem locations. Unfortunately the workaround for "local" would be to make sure the previous line ends with a word character (and not in a comment), which is not a good workaround. As a cheap hack, you could write a little script that replaces "local" with " local" after the minification.

If you want to experiment, I found the first problematic short-if on line 454 of your original cart. Let me know if you find other cases beyond short-if and local. Sorry for the inconvenience.

P#37331 2017-02-10 16:40 ( Edited 2017-02-10 21:40)

Awesome, thanks for getting back to me :) I will try replacing my short if's and let you know how I go. I'm hoping I ultimately can get by without the luamin tool. But it is a great backup plan.

Cheers!

P#37409 2017-02-12 21:51 ( Edited 2017-02-13 02:51)

A few recent changes to picotool:

BREAKING CHANGE: All Lua code is now handled in the API as bytestrings and not text strings. Pico-8 uses a custom text encoding equivalent to lower ASCII plus glyphs as high bytes. Instead of trying to manage conversion between these high bytes and Python text strings via an encoder, I just converted the whole thing to keep Lua code in bytestring form. This is a breaking change for the parts of the API that accept or return strings for/from the Lua code (token data, cart title, etc.). I probably busted stuff in the process so let me know if you see bugs.

With thanks to juanitogan, the minifier knows about more Pico-8 built-ins (and so doesn't munge their names). Also, a few bits were using an API only available in Python 3.5. I replaced those bits to use juanitogan's suggested Python 3.4-compatible equivalent, so Cygwin users can play. (Cygwin is currently still using Python 3.4 as its 3.x package.)

I added support for loading and saving .p8 files that have a label. This does not yet know how to convert labels between .p8 and .p8.png carts. (That'd be cool, especially since the same code could also support importing and exporting PNG images as spritesheet or label data.)

P#37691 2017-02-22 03:53 ( Edited 2017-02-22 08:53)

Hallo,
I just tried luamin with the latest Version. The generated _fmt.p8 gives me a Syntax error:

SYNTAX ERROR LINE 67
...
SYNTAX ERROR NEAR 'END'

Tested with this cartridge:

Also 'stop' was replaced with an undefined function..
I would be so happy if this would work

P#37696 2017-02-22 09:03 ( Edited 2017-02-22 14:03)

OK... probably I changed something while testing. Thanks alot

P#37704 2017-02-22 13:28 ( Edited 2017-02-22 18:28)

@Sascha217: Thanks for the report. I submitted a few fixes to get your cart to go through luamin, including adding "stop" to the keyword list. There's still a syntax error in the result, and I believe this is related to issue #11 regarding luamin being too aggressive with eliminating space between non-word chars and keywords. I can edit the result and insert spaces to resolve some syntax errors (though this needs to be done throughout). I'll look at this issue more this evening (PT).

P#37705 2017-02-22 13:30 ( Edited 2017-02-22 18:30)

One more thread bump to say that I've committed another batch of changes and small improvements. This includes support for a few popular but unofficial or accidental quirks in Pico-8's Lua syntax. if-do is now supported, as are C++-style // line comments. If you pulled since Feb 22, please pull again.

I'm pleased to say that with this last set of improvements, picotool successfully recognizes as valid all valid carts published to the BBS up to cart ID 28280. In this set there are four carts with errors that I've jailed for obscure reasons where Pico-8 accepts them but I don't see a good reason. I'll do a fresh crawl and test again on more recent carts. (This testing methodology only shows that picotool accepts these carts as valid. it does not prove that it produces correct parse trees. :) )

P#37764 2017-02-24 04:54 ( Edited 2017-02-24 09:54)

just tested the latest version... My minified cart has still systax errors

https://mega.nz/#!qN1RnLIC!DLN9Kq2-KI7W8EdR6IEENBY-ulp6HRZVxrvjwch424M

needed to change, amongst other things

bs[jp][jq]=jn[jo]jo+=1 

TO

bs[jp][jq]=jn[jo]
jo+=1 

also added a whitespace before each 'end' and 'local'. Probably that are way to much whitespaces. But anyway, this reduced the compressed code size from 15 to 10kb.. Thank You

P#37767 2017-02-24 05:29 ( Edited 2017-02-24 11:10)

Sascha217: I submitted a fix, please try again. I can minify your cart and it runs successfully. mario005.p8 goes from 41096 uncompressed to 233353 uncompressed, and from 15743 compressed to 10389 compressed.

It's a quick fix and may or may not be a complete solution to the problem. (In theory it might actually add chars in trivial extreme cases, but I'm not worried.)

P#37794 2017-02-24 19:39 ( Edited 2017-02-25 00:39)

Pretty significant compressed chars savings that I desperately need. Nice work. I did run across a bug.

Example:

print("Sum: "..A+1 .."!")

A whitespace is required after the 1 so that the following period doesn't get misconstrued for a decimal point. Right now, the script creates a "malformed number" error when the program is run.

P#37795 2017-02-24 19:48 ( Edited 2017-02-25 00:48)

FunFetched: Fixed, please try again (without the space). I broke that case to support "4." as a numeric literal, but I can support both with a lookahead assertion. :)

P#37799 2017-02-25 02:19 ( Edited 2017-02-25 07:19)

Thanks a lot. Will try this version

P#37840 2017-02-26 02:51 ( Edited 2017-02-26 07:51)

Again.. thank you very much. Works now!

Following warning is outputted:
warning: token count 27725 exceeds the Pico-8 limit of 32768

Works anyway... The token count was probably increased to 65536

P#38000 2017-03-02 16:18 ( Edited 2017-03-02 21:19)

Sascha217: Can you send me the cart that produced that warning? 27725 < 32768. :) Thanks!

P#38003 2017-03-02 22:55 ( Edited 2017-03-03 03:55)

Here's the cart, more or less... did some refactoring last night
https://dropfile.to/Ew3tGhW

P#38006 2017-03-03 08:32 ( Edited 2017-03-03 13:32)

Thanks. The issue was just in the error message. The actual token limit is 8192, so 27797 definitely exceeds that. Also I can't save your cart as a .p8.png because the compressed code size is too large. You may need to consider a multi-cart solution.

P#38009 2017-03-03 10:46 ( Edited 2017-03-03 15:46)

strange.. it's OK here

tokens: 7763/8192
program chars: 27911/65536
compressed: 12656/15360

P#38014 2017-03-03 13:22 ( Edited 2017-03-03 22:16)

Sorry, it was still reporting chars as tokens as of my previous message. You don't have 27797 tokens. :)

picotool does still appear to have a significant disparity from Pico-8 in token counting. Other than doing rigorous black box testing (with a lot of manual effort) through various parts of Lua syntax, there's not much I can do to nail down the last few differences. I'd like picotool to be accurate so it can report when it's building a cart that won't fit, but maybe that's an impractical feature for a weekend hobby project.

p8tool stats mario010.p8 reports 8560 tokens, 47866 chars, 18710 compressed chars. Pico-8 reports 7719 tokens, 47865 chars, 18720 compressed chars. So picotool erroneously overcounts and warns that it might be over the limit. (Then on top of that I had a broken message, which I'm about to fix.) Shrug emoji.

P#38023 2017-03-03 21:11 ( Edited 2017-03-04 02:11)

Found something else. menuitem() is a function in the PICO-8 API, but it gets clobbered by the script. I went ahead and fixed it on my end by adding it to PICO8_BUILTINS in lua.py.

P#38498 2017-03-21 04:08 ( Edited 2017-03-21 08:08)

about luamin,
I get a lot of these (the dreaded short-if):

if(a) then
 if(b) c=1
end

gives this

if(a) then if(b) c=1
end

which doesn't work in pico ('then' expected). I replaced all 'if' by '\nif' afterwards as a workaround.
now I have function "gu" set to nil or something... (*)

luamin does both variable renaming and separator optimizing, a way to disable the one or the other would be nice!

(*)edit: needed __index in PICO8_BUILTINS, works a treat!
thanks a lot!

P#43052 2017-08-04 11:49 ( Edited 2017-08-04 16:31)

Added these to the issue tracker. Thanks!

https://github.com/dansanderson/picotool/issues

P#43126 2017-08-08 17:22 ( Edited 2017-08-08 21:22)

This is a cool tool. I do have a minor issue to report: it was failing to load my game because I was using an alternate form of the print command that's smaller with fewer tokens.

PRINT("HELLO",1,1)

becomes

? "HELLO",1,1

Picotool was hanging at the question mark.

I don't believe this is a bug or deprecated form or anything like that, it's been mentioned in Pico-8 changelogs before. In fact you could probably incorporate it into the minifier to replace PRINT.

P#48939 2018-02-03 20:46 ( Edited 2018-02-04 01:46)

(Mentioned in Discord but posting the link here for posterity:) Bug tracking link for short-print support: https://github.com/dansanderson/picotool/issues/39

Thanks for the report!

P#48945 2018-02-04 00:57 ( Edited 2018-02-04 05:57)

Love this tool! But can't seem to get luamin to work on my cart. Keep getting syntax errors about nil values. I assume it's messes with my table references? Any idea what might cause that?

P#49142 2018-02-10 13:33 ( Edited 2018-02-10 18:33)

Can you post or send me a sample? Feel free to just post it as an issue in Github. https://github.com/dansanderson/picotool/issues

P#49144 2018-02-10 17:56 ( Edited 2018-02-10 22:56)

Will do, thanks! And thanks again for developing and supporting such a useful set of utils.

P#49151 2018-02-11 02:33 ( Edited 2018-02-11 07:33)

Nice tool! I want to use it to modularize my project, splitting code into different files and separating code from data. However, the PICO-8 editor directly puts sprite and music data into the game file (p8 or p8.png). I need a method to extract those data chunks and export them into separate files. Then, I can call "p8tool build" with the options to import the data files back into the final game file.

This would allow me to version control each data file alongside the code, and to exclude the built game file from version control. Currently, I would need to track the main game file because it's the only file containing my data, which means I'm tracking the code twice: in the source lua files and in the game file.

Without a data export script, I would need to copy-paste the data chunks to data files manually before each commit.

From what I can see, exporting data is feasible with the current API. list_lua uses g.lua.to_lines() to access the lua chunk, and we can similarly access other chunks.

Ex: for l in g.gfx.to_lines(): print(l)

We could output these lines in the correct format in some game.gfx file. Same for other data chunks.

For now I'll just make a simple data export script for my own game, but since you offer a tool to build a game from both code and data, I think such a feature would be a nice addition to the core tool package.

P#49391 2018-02-18 13:39 ( Edited 2018-02-18 18:39)

@huulong: The "p8tool build" command already does exactly what you describe. You would keep your graphics data in a Pico-8 cart file of its own, then pull it in at build time:

p8tool build mygame_final.p8.png --gfx mygame_gfx.p8.png --lua mygame.lua

If "mygame_final.p8.png" does not exist, it'll create it, using empty sections for any section you don't specify. If it does exist, it'll preserve any section you don't specify.

It sounds like you want to track source files in version control, and not track the final built output. In that case, you could either 1) specify all options to p8tool build to create a fresh cart during the build, or 2) keep everything but Lua in a source cart, copy that cart to the build location, then run p8tool build to add the Lua to that cart.

Yes you can write your own tool using the library if you'd like. You can look at the source for p8tool build for ideas on how to use the API. But it sounds like your use case is covered by the existing tool.

P#49392 2018-02-18 16:39 ( Edited 2018-02-18 21:39)

Hm, I hadn't considered working on a cartridge made purely of data. That would allow me to work in the P8 editor, save the data cartridge then build the final game, without even the need to extract data. I will try this today, thanks!

P#49655 2018-02-25 06:18 ( Edited 2018-02-25 11:18)

And it works! I just started my project, you can see the project structure with sublime text build commands here: https://github.com/hsandt/what-happened/tree/644c4a21a777f187da87feb4b7b7c9b1d736a5ed

I can open the data to edit it and save it, build and run the game, or just build it if I prefer reloading the game with Ctrl+R to avoid a full restart.

P#49658 2018-02-25 08:34 ( Edited 2018-02-25 13:34)

I have another problem now. I'm trying to unit test my modules with pico-test (https://github.com/jozanza/pico-test). As you explained in the README, we should be able to create lua scripts containing their own tests. Except it doesn't work if you use the "local module = {} / return module" pattern, because building a cartridge directly from the script will preserve the "return" which causes a parsing error (you're not supposed to do that in a normal game script).

So either you need to use one of the 2 other export methods (global function or global module), or you need to build from a main testmodule.lua script which will require the said module. In this case, you can either keep the test on the module side and just call them, or move the tests to the testmodule script. But this method requires an extra test runner script for each module.

What would do recommend, and do you have other workarounds?

P#49665 2018-02-25 11:10 ( Edited 2018-02-25 16:10)
Page:

[Please log in to post a comment]

Follow Lexaloffle:          
Generated 2024-03-19 08:52:41 | 0.039s | Q:75