I am pleased to announce Questionable PICO-8 Audio (QPA), a compressed audio format for PICO-8.
(Compatibility note: if you encoded any QPA files before 2023-09-07, please see the Changes section below.)
DON'T YOU LECTURE ME WITH YOUR 30K PICO-8 CART
Oh, but I will!
Your song is synced with the clipboard - you can copy your songs out to text files (or this thread!) and paste song strings back in to the cart. You can right-click samples to preview them, and you can right-click the play button to skip the intro sample. Numeric modifiers come before the thing they modify. Hopefully you can figure out the rest!
Copy/paste is a little annoying on the web player, and PCM audio can also be unreliable on the web, so consider trying this cart on desktop PICO-8 if you have it.
You can also check out this song by me:
7⬆️✽0⬆️✽✽✽⬇️✽⬆️✽3⬇️▶▤웃⬆️7◀0⬆️ˇˇ♪♪♪□∧7⬆️∧7⬇️7⬇️∧ |
Or this much better song by @packbat:
➡️▶▒7◀▶…6◀░▶▒7◀▶…6◀★▶▒7◀▶…6◀█▶😐7◀▶⁘3◀⬆️▶⁘3◀⬇️▶▒7◀▶…6◀░▶▒7◀▶…6◀★▶▒7◀▶…6◀⬆️★⬇️⬇️⬇️⬇️▶★7◀▶▥▥⬆️2◀∧ |
(The reference, if you're not familiar)
Why QPA?
Because you want to put fun sound samples in your carts, but don't want to use too many tokens or too much space!
QPA provides three main benefits:
- Tiny decoder. A fully-featured decoder that reads from strings, does quality-level detection, and rejects invalid files is 228 tokens. A minimal decoder is just 175 tokens.
- Reasonable quality at very low data rates. QPA encodes mostly-intelligible speech at just 1.14 bits/sample (788 bytes/second) - or completely-intelligible speech, as well as usable instrument samples, at twice that rate.
- Faster-than-realtime decoding speed. Precise speed depends on the decoder implementation, but around 4x realtime is a reasonable expectation for a token-efficient decoder.
As for how this works in practice: the demo cart PNG is just under 32k. It contains about 18k of samples (split 17k in sprite/map/etc. space, 1k in code) and 2k of other compressed code; the rest of the space is taken up by the label and other overhead. That 18k of sample data stores almost 14 seconds of audio. The samples are stored at a variety of quality levels, but more audio could have been included at lower quality, or higher-quality audio could have been used in exchange for more space. There also could have been a lot more audio if I hadn't tried to keep the cart around 30k - there's another 12k of compressed code space that could have been filled with QPA data strings!
What Is QPA?
QPA is an adaptation of the Quite OK Audio Format (QOA) to PICO-8's constraints. As such, it is a lossy, constant-bitrate format, in which the decoder attempts to predict future sample values one at a time, and compressed files store approximate differences between these predictions and the real sample values. To better understand how this scheme works, I suggest reading this introduction to QOA. QPA changes some details, but the basic structure remains the same.
So what changed to put the PICO in QPA?
- Multiple quality levels. QPA offers a choice of compression ratios so you can optimize your valuable cart space. The highest compression ratio (7x) is noisy, but is still suitable for some speech, percussion, and sound effects. The lowest two compression ratios (2.25x and 1.75x) are approximately transparent for most sounds.
- Minimal metadata. The only pieces of metadata in a QPA file are a magic number and a sample count. QPA includes no sample rate information, and does not support multi-channel audio (PICO-8 is mono only) or seeking to arbitrary points in a file (PICO-8 carts are small). This makes the decoder simpler.
- 8-bit audio only. QPA assumes an 8-bit output resolution, and is tuned to avoid single-bit output noise.
- PICO-8 numbers. Almost all arithmetic works on PICO-8's native q16.16 number representation, slices are now 32 bits to match PICO-8's word size, and the format is little-endian instead of big-endian.
- Smarter encoder. At PICO-8's low sample rate, it is critical to avoid high-frequency noise, so the encoder includes some light perceptual tuning to push noise into lower frequencies. It also expands QOA's use of exhaustive search for parameter selection, making it easier to incorporate new encoding strategies in the future.
Resources
To use QPA in your own carts, please check out the following resources:
- @bikibird's Defy encoder and Defy cart. You can use these to encode and play back your own audio files.
- The QPA utility cart. This cart makes it easy to convert QPA files to strings, and contains low-token decoding functions to copy into your own carts.
- The QPA repository on Github. Here you can find more documentation, a format spec, and a CLI encoder/decoder for QPA files. (CLI requires Node and NPM.)
Changes
On 2023-09-06 and 2023-09-07, new versions of all tools, code, and encoders were released to fix problems identified by Packbat. These new versions broke compatibility with earlier versions - if you switch to the new code, please keep in mind that you will have to re-encode your files. Hopefully this is a one-time only sort of change.
Future Work
I'd like to focus on getting better quality at approximately 1 bit/sample. It's possible that VBR or variable sample rate enhancements to QPA could get there, or it might be necessary to explore frequency-domain techniques ... although that might bode poorly for keeping decoder token count low.
But first - I think it's time to use QPA to build a tracker. No promises on release date!
Acknowledgements
- Dominic Szablewski for the QOA audio format
- @packbat for discussions and a whole lot of listening tests
- @bikibird for Defy and being willing to integrate QPA
- ferris for providing design feedback and pointers for future work
- MissMouse for the demo cart suggestion
- @NerdyTeachers and @Heracleum for help with the cookie/coffee test
- The entire #music-sfx channel on the Discord for discussions and for putting up with a whole lot of QPA chatter
The QPA format has been such an awesome project to follow! Really hoping people can make some awesome things with it!
Unrelatedly, a slow tempo pastestring! Figured out that you can get a pretty good echo with volume controls, too.
5⬅️▶☉3-◆◆◆3+◀⁙5-□2+◆◆3+▶▤2-█2+◀▶☉3-◆◆◆3+◀4⁘7⬆️⁘⁘7⬇️⁘▶⌂3-⌂3+◀▶☉3-◆◆◆3+◀⁙5-□2+◆◆3+▶▤2-█2+◀▶☉3-◆◆◆3+◀▶●2-●2+3◀ |
I need to stop playing with this so I can actually do needful things like buy groceries! (It's very fun tho!)
2➡️▶□-□□□+3◀3⬆️▶□-□□□+◀⬇️▶□-□□□+◀2⬇️▶□-□□□+3◀⁘◆◆⁘5⬆️⁘5⬇️○♥♥▶▤4-▤4+3◀▶□-□□□+3◀3⬆️▶□-□□□+◀⬇️▶□-□□□+◀▶⌂2-⌂2+◀7⬆️▶⌂2-⌂2+◀7⬇️▶⌂♪◀▶✽3◀3⬇️▶★7⬇️★7⬆️3◀░⬆️░⬆️░⬆️░⬆️○⬇️-○⬇️-○⬇️-○ |
Oh, @luchak, bug report/question: does the QPA 1.1-bit encoder or decoder have a cap on number of samples? For fun, I dropped a song from my music library ("Constellation" by Powderpaint) into the Defy web encoder, and the 2.3-bit was able to play back the full song in the Defy cart, but the 1.1-bit version cut off about 14 seconds in (an instant after the vocals start).
@bikibird Thank you!
@packbat Oh no ... I just bought the album, and yeah, I can reproduce this. Looks like more of the instability issues that I thought I had managed to tune out. I guess just "loud" isn't the only thing I needed to look out for in source material. I'll have to see what I can do about this; just increasing shifts even further (a) would be a breaking change, (b) would worsen prediction for most material, and (c) might not even fix the issue.... But I also don't want to just leave things broken. Thank you for pointing this out!
Good suggestion! It does extraordinarily well at handling that case, even for very slow, high-amplitude sweeps. But a nearly-fixed single frequency is captured very well by the predictor.
I think my new strategy is, instead of effectively de-tuning the system for stability, to just get the encoder to avoid unstable results. This means I have to let it look ahead a lot further, but I have some thoughts on how to do that.
Or, come to think of it, maybe lookahead isn't necessary either? I can explicitly penalize weight magnitude in the encoder's objective function. Regularization via weight decay didn't work very well, but perhaps a "please don't pick values that indicate future instability" penalty function will work better. Not sure why I didn't think of this before.
edit: this regularization idea seems to work so far, and is fortunately an encoder-only fix. Needs more testing though.
edit 2: Github repo updated. Encoder/decoder compatibility broken unfortunately - newer files from the fixed encoder will not work with old decoders.
Probably a bug: Put one loop inside another and the outer loop will never exit, even if you set a loop timer with a modifier.
EDIT: Might have to do with how the loop point is stored. Only one is there at a time, so when you exit the inner loop, the next “goto” command will go the last loop point before the cursor, entering the inner loop and resetting the loop timer. This is a simple fix. Hopefully.
probably based on https://thirtydollar.website/
●●●●●●●●⬇️●●●●●●●●⬇️●●●●●●●●⬇️●●●●●⬆️●⬆️●⬆️●●●●●●●●●⬇️●●●●●●●●⬇️●●●●●●●●⬇️●●●●●⬆️●⬆️●⬆️●●●●●●●●●⬇️●●●●●●●●⬇️●●●●●●●●⬇️●●●●●⬆️●⬆️●⬆️●●●●●●●●●⬇️●●●●●●●●⬇️●●●●●●●●⬇️●●●●●⬆️●⬆️●⬆️●●●●●●●●●⬇️●●●●●●●●⬇️●●●●●●●●⬇️●●●●●⬆️●⬆️●⬆️●●●●●●●●●⬇️●●●●●●●●⬇️●●●●●●●●⬇️●●●●●⬆️●⬆️●⬆️●●●●●●●●●⬇️●●●●●●●●⬇️●●●●●●●●⬇️●●●●●⬆️●⬆️●⬆️●●●●●●●●●⬇️●●●●●●●●⬇️●●●●●●●●⬇️●●●●●⬆️●⬆️●⬆️●
[Please log in to post a comment]
