I'm pleased to announce picotool, a set of tools and Python libraries for manipulating Pico-8 game files. picotool can read .p8.png and .p8 files, and can write .p8 files.
To get picotool and learn how to use it, visit the picotool Github repo.
The p8tool command is a multi-tool with multiple functions. You can run it like this:
% p8tool stats helloworld.p8.png hello world (helloworld.p8.png) by zep version: 0 lines: 48 chars: 419 tokens: 134
picotool currently includes the following tools:
- build: builds a cart from pieces of other carts
- stats: a statistics reporting tool
- listlua: prints the Lua region of a cart
- writep8: converts a .p8.png cart to a .p8 cart
- luafmt: writes a new .p8 cart with Lua rewritten with regular formatting, to make it easier to read
- luamin: writes a new .p8 cart with Lua rewritten to use a minimal number of characters, to make it difficult to read
- luafind: searches for lines of Lua code containing a string or matching a pattern
- listtokens: prints a visualization of Lua tokens
- printast: prints a visualization of the Lua parser abstract syntax tree (AST)
The Python library includes a hand-written Lua parser with support for Pico-8 features, with API access to the token stream and abstract syntax tree. Additional modules provide access to the graphics, map, sound, and music data. All game data can be transformed or created with Python code and written to a .p8 or .p8.png cart file.
I've included the Lua minifier (luamin) as an example tool, though I don't actually think it's a good idea to use it. Statistically, you'll run out of tokens before you run out of characters. The Pico-8 community benefits more from published carts with code that is easy to read and well commented. All luamin does is make the code difficult to read, without much benefit.
Here is a chart of the character count, minified character count, and token count of 496 carts that have been posted to the BBS, relative to the Pico-8 maximums:
As shown, even un-minified code tends to stay below the token count, percentage-wise. Minifying reduces the (uncompressed) character count to 65% of the original size on average.
The inspirational use case for this library is actually luafmt, which re-formats the Lua code of a cart to be easier to read. The current version of this tool is simple and only adjusts indentation. There is much more luafmt can do by analyzing the AST, but this is not yet implemented.
As of today, this is very much a v0.1 early release. There are known issues regarding parsing and token counting, and the library APIs are incomplete and messy. This project has already gotten away from me a bit, so I'm not sure how far I'll take it. But if you do want to build against it, patience with non-backwards compatible changes will be appreciated. :)
Let me know what you think! Thanks!
Update 2015-10-29: Added luafind tool. Refactored AST walking code to make it easier to build tools that read or transform the parser tree.
Update 2016-10-17: Added build tool. Added support for updating and creating .p8.png cartridge files.
There may be a few token optimizations that a tool could do automatically, but they'd be pretty smart compiler-style optimizations that actually rewrite the semantic structure of the code to a shorter equivalent. I'm thinking about building a tool that does dead code elimination, for example. (No promises, I might not have time, but it's the kind of thing you can do with access to the AST.)
I'm pleased to announce that picotool now supports writing .p8.png files, including code compression.
I've also started a new build tool. Its features are modest for now: it can replace sections of a cart (Lua, spritesheet, sfx, etc.) with sections from other carts, and it can also read Lua code from a .lua file and add it to a cart. It can also create carts from scratch and erase sections.
I have plans for more compelling features for the build tool (require() support, dead code elimination). My summer sabbatical is coming to an end so I might not get to it right away, but I thought I'd push what I have so far.
The new .p8.png code includes the code compression routine, the dual of the decompression routine that we've had for a while. Thanks to zep for providing the last few missing pieces!
[Update: fixed] Oof, just noticed picotool's compressor is compatible but not efficient, and is not producing the same result as Pico-8's compressor. It's bad enough that I don't think this is quite ready for the build workflow of a real (large) game. I'll update again once I've fixed it, but I'm mentioning it in case I don't get around to it right away. (My vacation is over! :( )
Tracking bug: https://github.com/dansanderson/picotool/issues/7
I am trying to use your picotool to minify my game. Currently I have:
65302/65536 char count and 8136/8192 token count.
When I am trying to use luamin on my cart I get:
C:\Python34>python p8tool luamin saveyourself.p8 saveyourself.p8 -> saveyourself_fmt.p8 warning: token count 29022 exceeds the Pico-8 limit of 32768Traceback (most rece nt call last): File "p8tool", line 8, in <module> sys.exit(tool.main(sys.argv[1:])) File "C:\Python34\pico8\tool.py", line 512, in main return args.func(args) File "C:\Python34\pico8\tool.py", line 349, in do_luamin return process_game_files(args.filename, luamin, args=args) File "C:\Python34\pico8\tool.py", line 210, in process_game_files procfunc(g, out_fname, args=args) File "C:\Python34\pico8\tool.py", line 242, in luamin lua_writer_cls=lua.LuaMinifyTokenWriter) File "C:\Python34\pico8\game\game.py", line 746, in to_file self.to_p8_file(outfh, *args, **kwargs) File "C:\Python34\pico8\game\game.py", line 659, in to_p8_file for l in self.gfx.to_lines(): File "C:\Python34\pico8\gfx\gfx.py", line 95, in to_lines yield bytes(newdata).hex() + '\n' AttributeError: 'bytes' object has no attribute 'hex'
I have ton of text packed into loooong strings if that helps.
Please help me, as I can't even export my game to html at this point.
Thanks in advance!
@dagondev: Can you share the cart with me so I can troubleshoot?
It's likely that the cart just doesn't compress under the limit and there's not much to be done, but I'd like to clean up the error messages at least. If we're lucky, minification will get it just under the wire, but no guarantees. :)
Hey, I wrote followup at issue page in github: https://github.com/dansanderson/picotool/issues/9
Here is the code: https://bitbucket.org/dagondev/save-yourself/src
I ended up minifing my code by hand, so no pressure now... But if you get this working, that would mean I could fix bugs on non minified version - so I would be grateful regardless. :)
Replied on the bug but will summarize here for reference: I need to add support for glyphs and labels before I can properly parse your cart. These are likely unrelated to the symptom you're seeing but I can't repro otherwise, and they're notable shortcomings regardless. I'll keep the bug open and revisit once I've had a chance to make these changes. Thanks for the report!
:: Writep8 issues
I'm having a strange problem; I'm attempting to use writep8 on Mac OS Sierra/Python 3.5 to convert p8.png carts to .p8 carts (to be edited in Sublime Text). However, every time I've attempted to do so, p8tool reports:
../carts/cartname.p8.png -> ../carts/cartname_fmt.p8.png
and sure enough, when I check the resulting output file, it's still a *.p8.png cart, and attempting to read it with cat or Sublime only yields byte data. Any ideas what could be wrong?
EDIT: The most recent cart this failed on was the TinyTV Jam base cart, found here: http://www.lexaloffle.com/bbs/?tid=28480
It also failed on the Emily Quest cart found here: http://www.lexaloffle.com/bbs/?tid=2294
Hi Dan, I understand if you aren't interested in supporting this. But just on the off chance;
I have just run p8tool luamin on my pico quest card z37305# (found here: http://www.lexaloffle.com/bbs/?tid=28574) and got a syntax error when running (on startup):
syntax error line 99
'then' expected near 'cg'
radcore: Good find! I filed this as a bug: https://github.com/dansanderson/picotool/issues/11
I'm going to blame Pico-8 for this one because it appears to be a shortcoming in its short-form "if" statement that luamin doesn't take into account. luamin will collapse space between a non-word non-space character and a word character. This is valid for Lua, but breaks Pico-8's short-if, which is implemented as a preprocessor replacement.
x=true y=(0) if (x) y=1 print(y)
x=true y=(0)if (x) y=1 print(y)
Pico-8 doesn't recognize the "if" as a short-if and fails to insert the "then" and "end" keywords before parsing the Lua. (picotool's own parser doesn't see the problem because I implemented short-if as a formal part of the grammar before I realized that Pico-8 was using a preprocessor replacement.)
I see a similar issue for a space before a "local" keyword being collapsed and causing problems. I'm less sure of the cause within Pico-8 but the solution is similar: it needs leading spaces preserved.
You can work around the short-if issue by avoiding short-if (using if-then-end) in problem locations. Unfortunately the workaround for "local" would be to make sure the previous line ends with a word character (and not in a comment), which is not a good workaround. As a cheap hack, you could write a little script that replaces "local" with " local" after the minification.
If you want to experiment, I found the first problematic short-if on line 454 of your original cart. Let me know if you find other cases beyond short-if and local. Sorry for the inconvenience.
:: picotool update, February 21, 2017
A few recent changes to picotool:
BREAKING CHANGE: All Lua code is now handled in the API as bytestrings and not text strings. Pico-8 uses a custom text encoding equivalent to lower ASCII plus glyphs as high bytes. Instead of trying to manage conversion between these high bytes and Python text strings via an encoder, I just converted the whole thing to keep Lua code in bytestring form. This is a breaking change for the parts of the API that accept or return strings for/from the Lua code (token data, cart title, etc.). I probably busted stuff in the process so let me know if you see bugs.
With thanks to juanitogan, the minifier knows about more Pico-8 built-ins (and so doesn't munge their names). Also, a few bits were using an API only available in Python 3.5. I replaced those bits to use juanitogan's suggested Python 3.4-compatible equivalent, so Cygwin users can play. (Cygwin is currently still using Python 3.4 as its 3.x package.)
I added support for loading and saving .p8 files that have a label. This does not yet know how to convert labels between .p8 and .p8.png carts. (That'd be cool, especially since the same code could also support importing and exporting PNG images as spritesheet or label data.)
I just tried luamin with the latest Version. The generated _fmt.p8 gives me a Syntax error:
SYNTAX ERROR LINE 67 ... SYNTAX ERROR NEAR 'END'
Tested with this cartridge:
Also 'stop' was replaced with an undefined function..
I would be so happy if this would work
@Sascha217: Thanks for the report. I submitted a few fixes to get your cart to go through luamin, including adding "stop" to the keyword list. There's still a syntax error in the result, and I believe this is related to issue #11 regarding luamin being too aggressive with eliminating space between non-word chars and keywords. I can edit the result and insert spaces to resolve some syntax errors (though this needs to be done throughout). I'll look at this issue more this evening (PT).
One more thread bump to say that I've committed another batch of changes and small improvements. This includes support for a few popular but unofficial or accidental quirks in Pico-8's Lua syntax. if-do is now supported, as are C++-style // line comments. If you pulled since Feb 22, please pull again.
I'm pleased to say that with this last set of improvements, picotool successfully recognizes as valid all valid carts published to the BBS up to cart ID 28280. In this set there are four carts with errors that I've jailed for obscure reasons where Pico-8 accepts them but I don't see a good reason. I'll do a fresh crawl and test again on more recent carts. (This testing methodology only shows that picotool accepts these carts as valid. it does not prove that it produces correct parse trees. :) )
just tested the latest version... My minified cart has still systax errors
needed to change, amongst other things
also added a whitespace before each 'end' and 'local'. Probably that are way to much whitespaces. But anyway, this reduced the compressed code size from 15 to 10kb.. Thank You
Sascha217: I submitted a fix, please try again. I can minify your cart and it runs successfully. mario005.p8 goes from 41096 uncompressed to 233353 uncompressed, and from 15743 compressed to 10389 compressed.
It's a quick fix and may or may not be a complete solution to the problem. (In theory it might actually add chars in trivial extreme cases, but I'm not worried.)
Pretty significant compressed chars savings that I desperately need. Nice work. I did run across a bug.
print("Sum: "..A+1 .."!")
A whitespace is required after the 1 so that the following period doesn't get misconstrued for a decimal point. Right now, the script creates a "malformed number" error when the program is run.
Sorry, it was still reporting chars as tokens as of my previous message. You don't have 27797 tokens. :)
picotool does still appear to have a significant disparity from Pico-8 in token counting. Other than doing rigorous black box testing (with a lot of manual effort) through various parts of Lua syntax, there's not much I can do to nail down the last few differences. I'd like picotool to be accurate so it can report when it's building a cart that won't fit, but maybe that's an impractical feature for a weekend hobby project.
p8tool stats mario010.p8 reports 8560 tokens, 47866 chars, 18710 compressed chars. Pico-8 reports 7719 tokens, 47865 chars, 18720 compressed chars. So picotool erroneously overcounts and warns that it might be over the limit. (Then on top of that I had a broken message, which I'm about to fix.) Shrug emoji.
Log in to post a comment