Sounds like you're thinking that I'm accusing you of being mistaken in your findings, that's definitely not the case. It's just that in your initial post you said the error appeared to stem from a lack of image complexity in a compressed, mostly single-color splash screen, and that adding a bit of visual complexity seemed to solve the issue. As I'm intending to use PX9 for compressing and storing relatively dense spritesheets, it doesn't seem like the issue should crop up. I'll try and adapt the new version for string capability just in case, though.
@JadeLombax, sorry no not at all. The new post is entirely new train of thought. I realize it was worded poorly. Disregarding entropy issues, I am seeing an effect where a properly compressed image seems to improperly decompress using the same string in the same cart just a few lines later, but if I save the string to a file and copy it into another cart, it works fine. I am asking a new question, if this weird behavior makes any sense to you, or despite thinking I was being thorough, I have some mistake in my test that I'm not seeing. Again I'm not challenging anything, just that you implemented/use the str version, and I'm seeing this weird behavior with the string version, so I reached out for your opinion/crosscheck.
Okay, I think I might know what the problem is. The string encoder outputs full 8-bit strings, which include 'puny font mode' characters, and unfortunately you have to press ctrl+p to enter puny font mode' before pasting them or they'll incorrectly show up as normal letters. Have you been doing that?
@JadeLombax I just wasn't thinking straight. PX9_COMP returns a string with escaped characters, which have to be pasted into the code as a literal in order for them to be processed correctly. I was taking the string result from PX9_COMP, as a live variable in running code, and directly shoving it back into PX9_DECOMP, and was confused by the garbled output.
<<< entire page of crap deleted that doesn't matter anymore >>>
Here is a demo cart fwiw:
Sorry for going down a rabbit-hole for a useless use-case side effect and adding pages to the BBS thread.
Well, it's not useless if it helps someone else avoid a similar problem in the future.🙂
Here now, this is not good. I tried Px9 to compress some test pictures to see how it would do and BOY did it fail on one of them !
Also these are findings AFTER I wrote my own compressor, mind you, not before. I just wanted to try out @zep's brainchild and compare it with the one I wrote.
The import is standard, 128x128 single color, font I doodled up along with ZEP's own characters >128 and <32.
I can post mine which does work but I really don't wanna do that cause this is ZEP's post. I'd be more interested in seeing it work for Px9.
. . .
Update 12-19-22. It works in a version someone ELSE posted. So I guess I have to say, do not use the cart that is listed directly at the top as it has not been updated yet.
Further I have not found a version that saves to string AND still correctly decompresses my test image, so if there isn't one, that means I have written the only one.
And I would like that very much not to be the case as I want @zep to be able to own this.
...
Also, @freds72 replied to this comment, not in this thread, but this ONE:
https://www.lexaloffle.com/bbs/?tid=50713
Why did he ? I don't know. Yes that is confusing.
Having an issue where a 128x64 image with only 3(4 if you count transparent black) colors compresses to the same size as a full-color 128x128 image. if it's all going to be the same size, could I put 2 128x64 images on one thing, compress it, then extract half the image somehow?
If I wanted to export my spritesheet data to the map, so that I can read from map for extra sprite data when I boot up the game, how would I do that? Is there a way to copy or export the map data affected by the compression so I can put it in my game? PX9 doesn't do this by default, correct?
In other words, I'm unsure how PX9 works. I would like it to compress my spritesheet data to map data, but I'm unsure how to retrieve that map data so I can put it in my game.
Thanks for this great tool. I used it to store a title screen image in my game: https://www.lexaloffle.com/bbs/?tid=143267
Question on the compressed data - does it always start and/or terminate with a specific sequence?
I would like to use px9 to store lots of small levels, so I will be storing them in sequence in cart memory. I can just keep track somewhere else in memory of all the start locations... But it would be cool if I could dynamically look at a long sequence of compressed data and detect how many compressed maps are in there. I could do that if each sequence starts or ends with a predictable byte.
How can I detect the start or end of data that has been compressed by px9? Is it possible?
that would mean scanning the whole sequence?
you'd better compress the maps, record length for each compressed bytes and store that table.
@freds72 yes, I am proposing scanning the whole sequence. My thought is I would do this once during _init and then create the table you propose.
I'm interested in this for when I am swapping the compressed data between carts. If I can scan the whole sequence, then I can just worry about copying the compressed data from cart to cart. However, if I cannot scan it and produce the table you suggest just from the data itself, then I also need to save the lengths, as you suggest.
It would just make the data-swapping from cart to cart easier if I can reconstitute the table from the data itself, rather than also having to save and transfer the length of each sequence.
Does this idea make sense? Is it possible?
Edit I think another way of rephrasing my question is ... Can someone help me understand the "header" portion of the px9_compress function? The info I'm seeking is probably parsable from that header, but the code is 1) pretty technical and 2) code-golfed quite a bit.
bump: belated update to v10 with @pancelor's bug fix for low entropy source images + 215 token decoder (!), and added some tests in tab 3.
I also updated the head post's algorithm description with a visualisation (load #px9v -- press x to step through).
@paloblancogames (a bit late to be useful, but!) Unfortunately there isn't a marker at the start/end of the px9 output. It writes the whole thing in bitstream form, so there aren't necessarily any byte-wise values. The header is 4 values written as variable-width numbers (width-1, height-1, colbits-1, and num_cols-1), followed by the list of colours used in the source image, each encoded with colbits bits).
It might be possible to add your own 4-byte marker, but there isn't any guarantees that that value is not used in the encoded data (there are no impossible sequences, I think). As mentioned above, best is to store a 2-byte length at the start of each map followed by length:x0000 at the end, and then jump through those to count them.
I kinda hate to generate a v11 over this, but the decompressor has an unnecessary set of parens that could save a token:
i was discussing this in the PicoMap thread a few days ago, but the root issue seems to stem from PX9 itself:
compressing a very busy, tightly packed region of map memory by using MGET() in PX9_COMP() results in this strange, inefficient data structure of mostly color 0xF.
this means it becomes somewhat significantly more effective to just move the map data to sprite memory and compress it as GFX instead.
can someone smarter or more familiar with the algorithm than me explain what causes this, and if anything can be done to fix it?
@_jspecter o/
- for many images, lots of pixels can be predicted from nearby pixels (e.g. a large patch of blue probably has more blue afterwards, or alternating white and gray probably keeps alternating)
- px9 keeps a prediction list, which starts as
{0,1,2,3,4,5,...,15}(the length of the list depends on your data. for map data, it likely is{0,1,2...,255})- (technically, px9 has multiple prediction lists, but I'm glossing over that here)
- as the compressor reads the data, it updates the prediction list by moving colors to the front when seen. e.g. after seeing a red pixel the list becomes
{8, 0,1,2,3,4,5,6,7, 9,10,11,12,13,14,15} - instead of storing your data directly, PX9 stores the prediction index in a unary-ish format (see "putnum" in the code)
- all the choices up to this point this are helpful, because px9 can store low prediction indexes in a tiny amount of space. but high indexes (unpredictable data) take a large amount of space. this generally works out in your favor for 16-color images
so now an example:
- let's say the first "pixel" (map tile) in your "image" (map) is tile 3. then PX9 encodes 3 (the prediction index of 3 inside the list
{0,1,2,...,255}) in its unary-based format:1 10. (why this odd format? b/c it works well normally) - the next tile is 198, so PX9 encodes 198 (the index inside
{2,0,1,3,4,5,...,255}) as...1 11 111 1111 11111 111111 1011001. yikes!
so, when you have highly unpredictable map data with values from 0 to 255, the output ends up being full of the unary 1 11 111 1111 etc ramp-up, which shows up as a lot of color 15 when the bits are shown as colors in your spritesheet
the reason it doesn't show up as badly with pget is because your data is between 0 and 15, instead of 0 and 255, and the unary format doesn't get too out of control in the 0 to 15 range. (index 15 is stored as 1 11 111 1100, just 2 bits larger than a byte)
what to do about it? not much. change your data to be more predictable in the specific ways px9 can exploit, or find/write a different compressor (rle? lzw?) that can deal with your data in a smarter way.
(hm actually, you can get a bit better compression by using lower tile numbers. the length of the prediction list is based on the highest-used tile number, so if you only use tiles 0-31 then it will compress better than if you use tiles 32-63)
interesting - thanks for the explanation!
my first impression is that reading a full byte instead of handling everything in nybbles like GFX is strange, but comparing it on a simplistic map it makes a massive difference.
for my project, i can't really work around it - since its map data is for PicoMap object definition, there's not much reason to stick together large blocks of alike tiles that would compress well.
i think i'll still end up just treating it as GFX data and copying it over when necessary - the most i could figure to do further is rearrange the data to compress better as GFX, though that comes with the caveat of more complicated code to read it.
I second the thanks for the explanation! I was glad to hear there's a way to get better compression on PicoMap's object definition data. I've switched to using this for my current project, and will probably incorporate it into the next editor update.
[Please log in to post a comment]
