Log In  


Not really sure if this was mentioned anywhere, but I thought it might be useful for anyone who wants to make use of vector math, specifically calculating the length of a 2D vector, while minimizing the risk of overflow.

Because the sqrt(32767) ~= 181, it is easy to overflow when using the textbook equation for calculating vector length:
len = sqrt(x*x + y*y)

To derisk overflowing when squaring the terms, you can first scale down the vector's x and y components:
m = max(x,y)
x = x / m
y = y / m

then scale the square root of their sums:
len = sqrt(x*x + y*y) * m

Lua code (simplified since one term will always == 1):

function length(v)
  local d = max(abs(v[1]),abs(v[2]))
  local n = min(abs(v[1]),abs(v[2])) / d
  return sqrt(n*n + 1) * d
P#38407 2017-03-19 11:39


This is great, I just ran into overflow for the first time yesterday when I tried to calculate the distance between two objects on the opposite sides of the map. Since I really just wanted to know if they were "near" each other, not the precise distance, I ended up using a fake measure of "distance" that's just abs(a.x-b.x)+abs(a.y-b.y) but this is a much better general solution. Thanks!

P#38684 2017-03-24 11:32

Log in to post a comment


New User | Account Help
:: New User
About | Contact | Updates | Terms of Use
Follow Lexaloffle:        
Generated 2017-04-27 03:10 | 0.179s | 1572k | Q:10