Try as I might, I just don't have the brainpower to figure out how to compress data from decompressed 8-bits to compressed 8-bits. Now I can neatly do 8-bits to 6-bits, compressed, and back again, but not straight 8 to 8.

The problem I am running into is I want to use one of the bytes # (0-255) to represent a compression mark. I am scanning the entire frame of memory desired to be compressed for the one number in the data this is used least (in many cases, it appears zero times).

The problem is, that particular byte # might also be just data if there is one or more of these bytes occurring.

Now you would think you could simply go through the data and store a zero at the end of that signature byte to tell the computer that this is data, and not a compression mark, but then if you search for multiple instances of patterns for compression later, it is still possible to get that elusive byte as part of your data when it needs to be just a marker - thereby corrupting your compression.

Suggestions ?

Checksums might be your answer here.

The type of compression you're seeking involves potentially and in most cases actually losing some of the data along the way but you have to just sort of "make an educated guess" to recover it. But as long as you store like an md5 of what the data should be after decompression, (or was before compression, I suppose) you can compare to see if your decompression was a success, if not, try again. This allows for you to not need perfect information.

Now don't quote me on any of this, I ain't a wizard nor a compression guru, but it just seems to make the most sense to me logically.

I considered MD5, but it's - not going to be fast enough for what I need - nor can I really wrap my head around it easily. What I want to do is write a really small coded and super fast compressor that does a great job, like 75% compression or higher for most logo screens or can also be used to compress that 256-byte persistent data.

That's what I'm working on - and as long as I stick to 6-bit compression, it's working quite well. Any higher bits like 7 or 8 then the compression times goes way higher and the quality of the compression goes way worse.

And ultimately I need the data to be portable and transportable, so 6-bit is working well.

For those curious:

https://en.wikipedia.org/wiki/MD5

Well yeah, that's the trade off generally. For good compression you usually lose bits, especially for speed. I too wish I could have high compression ratios at fast speeds. I feel like that might just be magical christmas land, especially in a small amount of characters.

What comes after the mark? Some other byte or bytes, I assume?

You can free up one of the values for the byte after the marker and use it as a literal byte, E.g assuming the marker byte is 0xbe and the freed up value is 0xff:
0xbe <anything but 0xff> <more stuff maybe> = marker
0xbe 0xff - literal 0xbe

That's pretty much what jpeg does.

Complete ! I'm satisfied.

The 6-bit compression string is smaller than the 8-bit decompressed binary image I tested. That's a whole lot better than the initial 5-bit compressor I wrote years ago, and it has a complexity setting so you can compress the data even further, up to a depth of 255 if you like.

2 which is quite speedy is more than sufficient for Pico's 16-color images.

I'll make a tutorial and usage of it on the morrow.