Log In  

I've noticed some inconsistent behavior when using FLR and when doing floating point math in general:

In particular:

  • (9/10)x10 returns 9, but FLR((9/10)*10) returns 8.
  • (8/10)*10 returns 7.9999. I suspect this bug is related to the FLR bug, as FLR(7.9999) correctly returns 7.

I suspect the problem is not related to string conversion / PRINT, as I was seeing these issues in other contexts as well.

Apologies if this has been reported before, I searched and didn't find it, but I'm new to Pico-8 so might not know where to look...

Thanks,
Eric

P#63736 2019-04-20 18:36

1

The issue you're having is because the numbers on PICO-8 are internally represented with signed 16.16 fixed-point format.

When you divide 9 by 10, you get a value that's roughly 0.9, but which is actually represented internally as the integer 58982 with an implicit divisor of 65536. Thus you actually have 58982/65536 = 0.899993896484375. When you multiply that value by 10 you obviously get 8.99993896484375, and print() is willing to round that up to just "9" since it's limited to four decimal places, but if you flr() it first, you'll get 8, since it wasn't quite 9.

The inconsistency you saw with different fractions will be down to exactly how inaccurate the fraction is when it's represented in the fixed-point format. Some will round down to x.9999 instead of rounding up to x(.0000).

If you want to know the exact value, you can print the hex version of the fixed-point number by manually converting it to a string with tostr(), adding the 'true' flag that says to use hex format:

> print(tostr(9/10, true))
0x0000.e666

> print(tostr((9/10)*10, true))
0x0008.fffc
P#63760 2019-04-21 12:54 ( Edited 2019-04-23 01:14)

I hate to revive a year-old thread, but I've just found a related issue where flr((1/x)*x)) results in 0 when x is odd, (because for example, 1/3*3 results in 0x0000.ffff) which is somewhat confounding. Hopefully it helps anyone else who also thought they were going crazy!

P#76412 2020-05-12 02:52

[Please log in to post a comment]

Follow Lexaloffle:          
Generated 2024-03-28 11:24:30 | 0.006s | Q:14