Not really sure if this was mentioned anywhere, but I thought it might be useful for anyone who wants to make use of vector math, specifically calculating the length of a 2D vector, while minimizing the risk of overflow.

Because the sqrt(32767) ~= 181, it is easy to overflow when using the textbook equation for calculating vector length:

len = sqrt(x*x + y*y) |

To derisk overflowing when squaring the terms, you can first scale down the vector's x and y components:

m = max(x,y) x = x / m y = y / m |

then scale the square root of their sums:

len = sqrt(x*x + y*y) * m |

Lua code (simplified since one term will always == 1):

function length(v) local d = max(abs(v[1]),abs(v[2])) local n = min(abs(v[1]),abs(v[2])) / d return sqrt(n*n + 1) * d end |

### ::

This is great, I just ran into overflow for the first time yesterday when I tried to calculate the distance between two objects on the opposite sides of the map. Since I really just wanted to know if they were "near" each other, not the precise distance, I ended up using a fake measure of "distance" that's just abs(a.x-b.x)+abs(a.y-b.y) but this is a much better general solution. Thanks!

## Log in to post a comment |