My app deals with mathematics and geometry, and it will often have to display floating point values. As you may know, floats often have gunk in their least significant digits after a few calculations so it is typically a bad idea to show them in all their glory (4.5000000000000000000003 or 3.294727893640192837562934), because your interface will end up being unreadable and intimidating. Simply rounding them to an arbitrary number of decimal places is better, but not ideal as it tends to sometimes remove information users do care about.

Does anyone know of a good formatting algorithm that displays floating point numbers in sufficient accuracy and sufficient readability for nearly all cases? I'm interested in figuring out when to round to how many digits, when to use scientific notation etc. etc.

  • 5
    You have to know from your users/use case how many decimals to show. What's the app? What's it used for? Scientific apps may well need to display all or display up to a certain precision, non-technical apps may not need to show a decimal at all for some situations. – Ben Brocka Jun 28 '12 at 18:06
  • I don't think a single number of digits is good enough. Large numbers probably need fewer digits. Scientific notation may well need a fixed number of digits, with all the trailing zeroes included. Maybe numbers close to integers need to be rounded more severely than numbers that are probably not close to a meaningful value. I've got tens of thousands of users, you want me to ask them all and then pick the average answer? That's not going to lead to something anyone will be happy with. – David Rutten Jun 28 '12 at 18:46
  • 1
    Sounds like the ideal number may be specific to your application or even specific to your users; could you let users configure how many decimal points to display? – Jessica Jun 28 '12 at 18:50
  • 1
    Have you considered "5.124…" with the full precision in a tooltip? I generally hate hiding information I will want to copy/paste in a tooltip, but as a general solution it's kind of what they're for. – Kit Grose Jun 30 '12 at 13:59
  • 1
    Kit, interesting idea. Unfortunately the values are often already displayed in tooltips. Still, in those cases where they aren't, this will be a very good idea. – David Rutten Jul 2 '12 at 8:50

Every rounding operation will decrease precision. When you want to round arbitrary number it is better to have constant allowable precision lost than the constant number of digits after floating point.

For example if you'll round numbers 0.0100005 and 0.014 to 2 digits after the point, the result will be 0.01 in either case, but in the 1st case the number will loose less then 1% of its precision, and for the 2nd number the precision lost is about 30%.

Without going much into details of data type sizes and concrete implementation (as this is more about programming than about UX), the rounding algorithm may sound like:

  1. Determine the most extreme rounding possible for a given number. (For 0.14 it will be 0.1, for 5,143,432 it will be 6 millions)
  2. Calculate lost of precision. (For 0.14 and 0.1 it will be: abs(0.14 - 0.1)/0.14 ≈ 0.29 or 29%)
  3. Loose rounding step by step unless you'll achieve the allowed precision lost. (For 0.14 and 1% allowed lost this algorithm will find that no rounding is necessary, for 0.140005 it will find that 0.14 will be OK, for 5,143,432 and 1% precision lost the 5,100,000 will be the answer which may be better formatted like 5,1M or 5.1E6 after the rounding depending on the context of your app)

You can determine the allowable precision loss for your context and make it constant for the whole app or allow users to change it (this much depends on user's expectations, so requires some testing I think). Together with tooltip advice from Kit this may be your win solution.

  • 1
    Also watch out for binary conversion problems. You can not convert the number 10.45 to binary. So if you round it to 1 digit in decimal you would round it to 10.5 but because data is stored binary, your value will actually be 10.449999999... which will be rounded to 10.4 – Pieter B Jul 2 '12 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.