Displaying floating point values

Question

My app deals with mathematics and geometry, and it will often have to display floating point values. As you may know, floats often have gunk in their least significant digits after a few calculations so it is typically a bad idea to show them in all their glory (4.5000000000000000000003 or 3.294727893640192837562934), because your interface will end up being unreadable and intimidating. Simply rounding them to an arbitrary number of decimal places is better, but not ideal as it tends to sometimes remove information users do care about.

Does anyone know of a good formatting algorithm that displays floating point numbers in sufficient accuracy and sufficient readability for nearly all cases? I'm interested in figuring out when to round to how many digits, when to use scientific notation etc. etc.

Answer 1

Every rounding operation will decrease precision. When you want to round arbitrary number it is better to have constant allowable precision lost than the constant number of digits after floating point.

For example if you'll round numbers 0.0100005 and 0.014 to 2 digits after the point, the result will be 0.01 in either case, but in the 1st case the number will loose less then 1% of its precision, and for the 2nd number the precision lost is about 30%.

Without going much into details of data type sizes and concrete implementation (as this is more about programming than about UX), the rounding algorithm may sound like:

1. Determine the most extreme rounding possible for a given number. (For 0.14 it will be 0.1, for 5,143,432 it will be 6 millions)
2. Calculate lost of precision. (For 0.14 and 0.1 it will be: abs(0.14 - 0.1)/0.14 ≈ 0.29 or 29%)
3. Loose rounding step by step unless you'll achieve the allowed precision lost. (For 0.14 and 1% allowed lost this algorithm will find that no rounding is necessary, for 0.140005 it will find that 0.14 will be OK, for 5,143,432 and 1% precision lost the 5,100,000 will be the answer which may be better formatted like 5,1M or 5.1E6 after the rounding depending on the context of your app)

You can determine the allowable precision loss for your context and make it constant for the whole app or allow users to change it (this much depends on user's expectations, so requires some testing I think). Together with tooltip advice from Kit this may be your win solution.