Log In  

Hi there,
anyone having an idea how to read fixed point numbers with peek? I managed to get the integer part read and I think I understand as well how the decimal part is designed, but as the decimal part uses the unsigned range of 16 bits, I don't get it read correctly. Currently I have:

function rnum()
 local dlo=peekb()
 local dhi=peekb()
 local ilo=peekb()
 local ihi=peekb()
 local nu=bor(bor(shl(ihi,8),ilo),shr(bor(shl(dhi,8),dlo),16))
 return nu
end

Where peekb simply reads a byte from memory with peek and increments a pointer b.

 local ilo=peekb()
 local ihi=peekb()

These bring me the integer part nicely, but I'm having no success in getting the decimals right. Any help or hint is very appreciated :)

P#19227 2016-03-14 21:09 ( Edited 2016-10-01 19:54)

After a bit of messing around I came up with this:

function rnum()
local ihi=peekb()
local ilo=peekb()
local dhi=peekb()
local dlo=peekb()
local nu=bor(bor(shl(ihi,8),ilo),shr(bor(dhi, shr(dlo,8)),8))
return nu
end

By shifting the decimal part into the leftmost 8 bits you're affecting the sign bit, which is then carried across in all future arithmetic operations, so you can end up with sign-inverted or 'mirrored' values (due to two's complement rules).

NB- I had to swap the byte order around to get this to work with the data I tested against, you may need to swap it back, depending on the format you're using.

edit: oh, didn't spot the date on the thread. Well, hope you figured it out in the end!

P#29807 2016-10-01 15:37 ( Edited 2016-10-01 20:35)

If you can, I suggest using cartdata storage, the code will be extremely short:

function getnum(address)
    memcpy(0x5e00, address, 4)
    return dget(0) -- make sure cartdata was called once!
end
P#29808 2016-10-01 15:43 ( Edited 2016-10-01 19:43)

Oh nice, I hadn't considered that shortcut, thanks.

P#29809 2016-10-01 15:54 ( Edited 2016-10-01 19:54)

[Please log in to post a comment]

Follow Lexaloffle:        
Generated 2020-04-10 03:49 | 0.010s | 4194k | Q:15