anyone having an idea how to read fixed point numbers with peek? I managed to get the integer part read and I think I understand as well how the decimal part is designed, but as the decimal part uses the unsigned range of 16 bits, I don't get it read correctly. Currently I have:
function rnum() local dlo=peekb() local dhi=peekb() local ilo=peekb() local ihi=peekb() local nu=bor(bor(shl(ihi,8),ilo),shr(bor(shl(dhi,8),dlo),16)) return nu end
Where peekb simply reads a byte from memory with peek and increments a pointer b.
local ilo=peekb() local ihi=peekb()
These bring me the integer part nicely, but I'm having no success in getting the decimals right. Any help or hint is very appreciated :)
After a bit of messing around I came up with this:
function rnum() local ihi=peekb() local ilo=peekb() local dhi=peekb() local dlo=peekb() local nu=bor(bor(shl(ihi,8),ilo),shr(bor(dhi, shr(dlo,8)),8)) return nu end
By shifting the decimal part into the leftmost 8 bits you're affecting the sign bit, which is then carried across in all future arithmetic operations, so you can end up with sign-inverted or 'mirrored' values (due to two's complement rules).
NB- I had to swap the byte order around to get this to work with the data I tested against, you may need to swap it back, depending on the format you're using.
edit: oh, didn't spot the date on the thread. Well, hope you figured it out in the end!
[Please log in to post a comment]