Not really sure if this was mentioned anywhere, but I thought it might be useful for anyone who wants to make use of vector math, specifically calculating the length of a 2D vector, while minimizing the risk of overflow.
Because the sqrt(32767) ~= 181, it is easy to overflow when using the textbook equation for calculating vector length:
|len = sqrt(x*x + y*y)|
To derisk overflowing when squaring the terms, you can first scale down the vector's x and y components:
|m = max(x,y)|
x = x / m
y = y / m
then scale the square root of their sums:
|len = sqrt(x*x + y*y) * m|
Lua code (simplified since one term will always == 1):
function length(v) local d = max(abs(v),abs(v)) local n = min(abs(v),abs(v)) / d return sqrt(n*n + 1) * d end
This is great, I just ran into overflow for the first time yesterday when I tried to calculate the distance between two objects on the opposite sides of the map. Since I really just wanted to know if they were "near" each other, not the precise distance, I ended up using a fake measure of "distance" that's just abs(a.x-b.x)+abs(a.y-b.y) but this is a much better general solution. Thanks!
Log in to post a comment