Log In  

Finally hit the milestone in Picoder development where I'm able to decode p8.png files in pure JavaScript. I couldn't have done it without the help of @gamax92 and @dddaaannn, as well as the source code of their respective projects PICOLOVE and PicoTool. It's been a long, bumpy ride, but here it is: http://clowerweb.com/picoder-test/. This will be used to import .png files into Picoder, and the decoding algorithms reversed in order to compile code (edited in Picoder's online IDE) to be playable in the web player for testing.

So huge shoutout and thanks to the above named! Hopefully the IDE itself will be available for public testing soon. Please let me know if you find any issues with the decoder - this is my chance to let you all test it and help me find bugs. Thanks!

Edit: source, if anyone wants it

P#30345 2016-10-08 20:24 ( Edited 2016-10-11 01:04)

Not to beat a wet hen (or some other similar analogy), wouldn't ZEP, the owner have helped you decrypt ?

I thought he was working with you guys all along on this port.

Trying out your program, it does work. Looking forward to the Online IDE.

P#30347 2016-10-08 20:48 ( Edited 2016-10-09 00:48)

At least from my side, I didn't ask zep to help. Not sure if he would have or not. I can't speak for the others as to whether he helped them or not, but there's definitely some stuff that, if they figured it out without his help, I truly have no idea how. It's definitely possible but you'd have to be a wizard and well beyond my skill level.

P#30349 2016-10-08 20:56 ( Edited 2016-10-09 00:56)

Well wow ... I hope ZEP approves then. I know I never could get that sneaky snaky Python GUI to work. One project I had in mind was to build my own PICO GUI in BlitzMAX once I slowed down on other stuff.

Very intuitive, it would watch as you type and, for instance, if you use an IF without parentheses, it would automagically put a THEN on the end. Same with FOR(),() , it would put a DO if you forget.

Save your work to an autosave.* every 5-minutes, Folding functions, auto code arrangements, lots of things in it actually, including a SFX/MUSIC port where each sound shows up as a colored jewel and you get to to work with more than one source at a time and transfer jewels from one code to the next - mix and match.

PHEW ! Oughta keep me busy.

Since you've done a little deciphering, perhaps you could help me ?

A few technical questions:

What is 0x3000 GFX_PROPS, what purpose does it serve ?

What is 0x5F00 DRAW State ?

Starting with 0x5F40 to 0x6000, are there any PEEKS in there that read a true QWERTY keyboard, either local run or Online ?

0x4300 is User Data, is this Source code, and if not, what lies within it to 0x5E00 ?

I had to write my own decimal to hexadecimal routine. PICO does not already contain this ?

No ASC() or CHR() type of internal routine ?

Now the big question.

I'm working directly with memory 0x3200, but no length is given on it. Do you know its length, and you do you know the delimiter for each of the 64-sounds within and/or the construction of it - byte-per-byte ?

The screen is listed as 0x6000. To 8k that would be to 0x8000 (8192-bytes). Does anything lie beyond this horizon ? Dragons, perhaps ? :)

P#30351 2016-10-08 21:13 ( Edited 2016-10-09 01:16)

I'm not really sure exactly how the player itself works, as I haven't done any hacking on it (yet). All I've done here is decoded png images to p8 cart data. If you're interested in looking at how the cart data is stored/decoded, here is my source code: http://pastebin.com/VwgkU5kz (you will need to feed the getData function with a Uint8ClampedArray of bytes in RGBA order. I'm using the PNGToy library in there to get that RGBA data, but you can use whatever you want, such as rendering the image to canvas and using context.getData(), which would require some workarounds due to the transparent corners of the PNG and HTML5 canvas multiplying the RGB by the alpha. Another way would be to put the image into a WebGL texture buffer and extract the RGBA values from that - those aren't pre-multiplied, so no workarounds necessary.

Beyond that, for the player itself, you might try using something like Cheat Engine to look at the memory in the player and play around with the values to see what they do. 010 Editor might yield some clues about various things as well, and I think you can decompile it to Assembly and/or C.

P#30354 2016-10-08 21:27 ( Edited 2016-10-09 01:27)

You know what's funny. I was building something like this a few years ago. Mine wasn't so pretty though. I thought it was clever of me to build a PNG (no hidden data) where the data of my game maker was stored directly as raw pixels.

You had room for 16-vertical pixels above and essentially 500-pixels across for your 'splash' screen with up to 486 pixels below for raw data.

This ... hidden pixel stuff for a PNG that PICO and ZEP use is mighty intriguing to me. I think I asked about this in my first post to BBS. I'm now understand (you're saying), it's no easy feat to get the source out.

And yes, you can store hidden pixels in the ALPHA array, something I knew about, but many graphic servers do not LIKE so if you tried to post this .PNG in other picture websites, they would swoop down and RIP out the 4th bit of data, transparency.

The site I write my books on does this, for instance, so I was limited to using only R G B and abandoned the 4th hidden ALPHA factor.

I still have this game maker idea, even more so seeing someone so successful at it (tips hat to ZEP), yet, unlike his language, the one I had planned would let you use a resolution of 800x600.

The libraries I've written are quite intelligent. If you hit "+" from the NUMBER KEYPAD, it would ZOOM the vertical to meet the edge of your windows taskbar. Hit it AGAIN and it zooms the horizontal to a 1.5 proportion factor of the standard 4x3. Hit it again, and you get a center 800x600 screen, no dithering to enlarge. Functioning entirely within a user customizable 8x8 font.

I thought it was a good idea at the time. I've given myself a few days of October to work and build and learn more about PICO and its properties and proprietaries. (meaning data formats and storage).

P#30355 2016-10-08 21:42 ( Edited 2016-10-09 01:45)

In PICO's case, the data isn't just in the alpha channels. It's the last 2 bits of each of the R, G, B and A bytes. So you grab the last 2 bits of R, last 2 bits of G, etc., shift R, G and A left, and then combine each of those bits (2+2+2+2=8) for your 1 byte of cart data. That's not all you have to do though. Some of them are acceptable as-is, depending on which section you're in. GFX has to have endianness changed, music and SFX are a bit of a pain in the ass to put together, and then map and GFF are both fine as-is. For the Lua code, it's good as-is UNLESS it's compressed (which it does after a very small number of characters, I haven't played around to see how many yet, but it's probably something like 50; I know it's a very very small number).

Decompressing the compressed code is something you could almost write a small book about, so I won't get into it too much, but it's a real PITA. Basically a TL;DR on it would be that sometimes you can use the bytes as-is, sometimes you're doing dictionary replacements on the bytes rather than just converting them directly to ASCII, and other times you're grabbing a variable length of bytes from a specific offset, putting them into a special buffer, and then appending them to your output. Other times you have NULL bytes which you just skip over. Which of these operations you need to do all depends on whether the byte is 0, less than 60, or greater than 59.

P#30357 2016-10-08 22:07 ( Edited 2016-10-09 02:07)

Wow ! That's genius ! Darnit, why didn't I think of that ?

Well, yaah, I could've snipped off a few high bits from a user's .PNG, get those colors down a shade, and save sneaky data therein. I still can do that now that you mentioned the secret ingredient to the RECIPE of PICO's PNG. Thank you ! :)

I really did do some deep investigating to see how to 'hide' data in a .PNG and was coming up with a dry well. I was sure he was storing naught but raw data somewheres in there.

I'm not so interested in decrypting the PICO .PNG though.

That's proprietary information from genius ZEP. However, if someone just has the .P8 (meaning they purchased PICO-8), then I could write a GUI to read and handle that separate .P8 .TXT file. And you can only extract that if you actually have bought the OS.

And ... there is still the interesting idea of snarking out some bits of data from a picture or splash page of a cart for my own future game maker. I wouldn't be all that tricky as he was.

3-bits from red, 2-bits from green, 3-bits from blue .... to get that ever-loving BYTE out per pixel.

Wow ! Reading above after the TLDR. You really did your homework ! I've had jobs in the past of deciphering code for businesses (mostly completed data forms) but that's a real brainweave you have there, Scathe. :)

P#30361 2016-10-08 22:47 ( Edited 2016-10-09 02:52)

Gah. Took me like 15 minutes to figure out how to stop win10 from declaring it a scary file and making it disappear.

edit: trying to save the js from the pastebin. Considering the code just returns a value, I assume it was something dumb like a match on the dict variable or something that set off win defender.

P#30390 2016-10-09 01:35 ( Edited 2016-10-09 05:51)

@tyroney Hm? What file?

P#30391 2016-10-09 01:43 ( Edited 2016-10-09 05:43)

tyroney--

That sort of thing is why I haven't upgraded to win10. Can it be disabled entirely or did you have to do some kind of workaround for the specific file?

P#30453 2016-10-09 15:46 ( Edited 2016-10-09 19:46)

@tyroney Ah, gotcha. Pastebin actually made me do a captcha to save it as well, saying it looked like malware. You're probably right about it being because of replacing bytes with items in a dictionary array.

P#30471 2016-10-09 17:41 ( Edited 2016-10-09 21:41)

It can be disabled, but since it seems to work and not ruin anything I just left it be. The quickest way to get the file back was to go into Win Defender, hit the history tab, on quarantined or just all, hit the button at the bottom for more info so that it actually shows you the list of SCARY FILES then you can highlight the item and allow it.

For the record, it was falsely flagged as Win32/Spursint.A!c!

ps - with an oem (no crap) clean install, (new drive, linux / xp64 on other partitions) I've found win10 to be just dandy so far.

(aside from one particular week early on when it kept trying to "upgrade" my nvidia driver to an older version which caused hard locks, but that got straightened out and I grabbed the install straight from nvidia and it's been great since)

P#30510 2016-10-10 00:02 ( Edited 2016-10-10 04:07)

Just another reason for me to avoid Windows 10. Yay Windows 8 and ... right, with Classic Shell it's a good OS again !

P#30511 2016-10-10 00:23 ( Edited 2016-10-10 04:23)

Here's an explanation of the code decompression algorithm from asterick from last year: https://www.lexaloffle.com/bbs/?tid=2400 I relied on this heavily for picotool's implementation.

Answering dw817's other questions:

  • 0x3000-0x30ff ("gfx_props" in the documentation, "gff" in the .p8 file) are sprite flags, eight flag bits for each of 256 sprites. They are intended as user-assignable metadata for sprites. You can set and get flags with the fset() and fget() functions, and you can set their initial state (stored in the cart) in the sprite editor. They're also used by map()'s layer feature to selectively draw sprites from the map.

As an example, Jelpi's platforming collision code uses sprite flags to declare specific sprites as "solid". Try clearing the second (orange) flag on the pink block in the sprite editor and see what happens.

  • 0x5f00-0x5f3f is the runtime draw state, which includes the current draw color (see color()), camera position (see camera()), print cursor position (cursor()), clipping region (clip()), and palette modifications (pal(), palt()). The memory layout is undocumented but it's easy to reverse engineer it by calling one of these functions and comparing the memory region's before and after. Changing individual elements is best done by the functions. You might read and write from the memory region if you want to save and restore the entire draw state at once. [Update: I added draw state memory layout to the Memory article.]

  • There is no built-in way to read from the entire keyboard. dppc built this clever hack reads the keyboard in browser JavaScript then pipes keycodes into the web player via JavaScript GPIO, but naturally that only works in the web player: https://www.lexaloffle.com/bbs/?tid=3556

  • 0x4300-0x5dff is reserved for general use by programs. It is not used by the runtime environment itself. This region is not initialized when the cart is loaded, though the program itself can use reload() to initialize this region from a cart file.

  • Pico-8 does not provide a built-in way to generate a string of hexadecimal characters for a given integer. It's not difficult to write such a routine, but you are correct that one is not provided.

  • Pico-8 does not provide a built-in way to get a one-character string for a character code number value, nor the other way around. A common solution is to define a look-up string with all of the characters in order and use sub(). zep provides one such implementation here: https://www.lexaloffle.com/bbs/?pid=24454#p24454

  • Memory layouts for the sfx region and others are described here: http://pico-8.wikia.com/wiki/Memory

  • There is nothing accessible from userspace beyond address 0x7fff. Code storage ROM and the runtime RAM for the Lua interpreter are present but not addressable or accessible from code directly.
P#30557 2016-10-10 13:47 ( Edited 2016-10-10 19:27)

dddaaaMMnnn !!

This is quite a bit for me to take in ! Gimme a few days for a proper reply here. Believe it or not I do have more questions.

Thanks so much for this info ! (Copying to notes).

P#30560 2016-10-10 15:16 ( Edited 2016-10-10 19:16)

I updated http://pico-8.wikia.com/wiki/Memory with details about the draw state memory layout, based on a few minutes of poking around. Corrections welcome.

P#30562 2016-10-10 15:25 ( Edited 2016-10-10 19:25)

Bookmarked. Question, do you know the specific memory location I can read for the PRINT cursor X & Y ? Or might that vary on differing hardware ?

P#30564 2016-10-10 15:46 ( Edited 2016-10-10 19:46)

The print cursor is explained on that page in the section on draw state: http://pico-8.wikia.com/wiki/Memory#Draw_state 0x5f26=x, 0x5f27=y.

P#30568 2016-10-10 16:24 ( Edited 2016-10-10 20:24)

Ah ! You did have it listed. This will save me a little trouble as I was keeping track of a virtual cursor position based on PRITTY.

-- "pretty pritty print mess"
-- tricky arguments !
-- prit(t) text only
-- prit(c,t) color then text
-- prit(x,y,t) coords then text
-- prit(x,y,c,t) all four
function prit(v1,v2,v3,v4)
P#30571 2016-10-10 16:46 ( Edited 2016-10-10 20:57)

I went ahead and wrote up the file formats, if this helps anybody:

Corrections welcome as always.

P#30576 2016-10-10 17:12 ( Edited 2016-10-10 21:12)

I'm reading the .p8.png file format. You mentioned ZEP is using a compressor if the data is past a certain size. Could you go in a bit more detail about it ?

I have my own data compressor (above), which works pretty good for what I need. I'm curious to see if he followed suit.

P#30580 2016-10-10 17:20 ( Edited 2016-10-10 21:20)

I'm not exactly sure what the threshold is for the size of code before it is compressed, but I can tell you that it's VERY small, and would be easy to figure out. When I was making the PNG decoder, I noticed that carts with something simple like "function _draw() cls() print("hi") end" were uncompressed, so I was able to output them by using String.fromCharCode() on the raw bytes and see the code properly. Since I didn't have the decompression done yet, trying even with test carts as simple as this:

function _init()
  local str="hello world!"
end

function _update()

end

function _draw()
  cls()
  print(str)
end

... resulted in compressed code. So the threshold is probably not much more than something like 50-100 characters. I remember looking at it and thinking "Why even bother having any code uncompressed when the threshold is THIS small?! Why didn't he just compress all of it?" But yeah, I don't know the exact number. I could probably sit down and figure it out in 5 or 10 minutes though if you really want to know.

P#30590 2016-10-10 19:08 ( Edited 2016-10-10 23:12)

My guess is that the condition is, "Is the compressed code smaller than the uncompressed code?" Very small code regions might not compress much at all, and slightly larger ones would. With this compression method, certain pathological cases will be smaller than their compressed counterparts. It may not be strictly necessary in practice, but in theory it allows us to squeeze in a few more bytes.

Prior art on Pico-8 compression:

P#30592 2016-10-10 19:41 ( Edited 2016-10-10 23:42)

You know ? The compressor I wrote does a comparison, and it only compresses blocks where the compressor signature is smaller - otherwise it leaves it alone. But it does this thorough check for the entire data field.

So the one I put together will ALWAYS either be exactly the same size or smaller. It's not that hard to do.

So back to ZEP's compressor. I guess if the data is REALLY small, like mine can't compress "Hello World" but CAN compress "Hellllllo world." to be one-byte smaller smaller. (with 6 L's).

P#30595 2016-10-10 21:04 ( Edited 2016-10-11 01:04)

[Please log in to post a comment]

Follow Lexaloffle:          
Generated 2024-03-28 16:05:47 | 0.025s | Q:40