Log In  

The string to number conversion code is sometimes setting the least significant bit erroneously when it converts negative numbers.

For example: "-1"*1 should and could exactly equal -1, but it does not, it equals (-1 + SHR(1,16)). This is vexing to the kind of programmers like me who like to use -1 as a special value...

BONUS WEIRDNESS: It doesn't always set it on negative numbers. In fact it seems like the pattern is if the number could be exactly represented, the LSB is set, whereas if the representation is already inexact the LSB is the same as the non-converted version.

So "-0.5"1 == -0.5+SHR(1,16) (not desired), but "-0.2"1==-0.2

(This function was useful looking at this...)

function bits(n)
 local a,s
 a=shr(1,16)
 s=""
 for i=0,31 do
  if band(n,shl(a,i))==0 then
   s="0"..s
  else
   s="1"..s
  end
 end
 print(s)
end
P#12794 2015-08-19 03:35 ( Edited 2017-04-02 07:24)

This might be related, it seems the range -1 to 0 is rounding improperly.

P#12813 2015-08-19 23:51 ( Edited 2015-08-20 03:51)

Looks like I've also been bitten by this "bug" - sadly, took me AGES to realise where the problem was, as I assumed (first mistake!) that all string to number conversion just "works".

Simple replication of issue:

txt1="1"
txt2="-1"
num1=1
num2=-1
con1=txt1+0
con2=txt2+0

if num1==con1 then
  print("num1 match")
else
  print("num1 diff!")
end

if num2==con2 then
  print("num2 match")
else
  print("num2 diff!")
end

Output is:

num1 match
num2 diff!

Now I'm sure I can workaround it, but it will waste tokens doing so - which is my bigger concern :o/

Hope this helps give ppl a heads-up! :o)

P#38999 2017-04-02 03:24 ( Edited 2017-04-02 07:24)

[Please log in to post a comment]

Follow Lexaloffle:          
Generated 2024-04-18 18:49:35 | 0.006s | Q:11