2.7.8 Modes Tutorial Exercise 1

Calc always stores its floating-point numbers in decimal, so even though one-third has an exact base-3 representation (‘3#0.1’), it is still stored as 0.3333333 (chopped off after 12 or however many decimal digits) inside the calculator’s memory. When this inexact number is converted back to base 3 for display, it may still be slightly inexact. When we multiply this number by 3, we get 0.999999, also an inexact value.

When Calc displays a number in base 3, it has to decide how many digits to show. If the current precision is 12 (decimal) digits, that corresponds to ‘12 / log10(3) = 25.15’ base-3 digits. Because 25.15 is not an exact integer, Calc shows only 25 digits, with the result that stored numbers carry a little bit of extra information that may not show up on the screen. When Joe entered ‘3#0.2’, the stored number 0.666666 happened to round to a pleasing value when it lost that last 0.15 of a digit, but it was still inexact in Calc’s memory. When he divided by 2, he still got the dreaded inexact value 0.333333. (Actually, he divided 0.666667 by 2 to get 0.333334, which is why he got something a little higher than 3#0.1 instead of a little lower.)

If Joe didn’t want to be bothered with all this, he could have typed M-24 d n to display with one less digit than the default. (If you give d n a negative argument, it uses default-minus-that, so M-- d n would be an easier way to get the same effect.) Those inexact results would still be lurking there, but they would now be rounded to nice, natural-looking values for display purposes. (Remember, ‘0.022222’ in base 3 is like ‘0.099999’ in base 10; rounding off one digit will round the number up to ‘0.1’.) Depending on the nature of your work, this hiding of the inexactness may be a benefit or a danger. With the d n command, Calc gives you the choice.

Incidentally, another consequence of all this is that if you type M-30 d n to display more digits than are “really there,” you’ll see garbage digits at the end of the number. (In decimal display mode, with decimally-stored numbers, these garbage digits are always zero so they vanish and you don’t notice them.) Because Calc rounds off that 0.15 digit, there is the danger that two numbers could be slightly different internally but still look the same. If you feel uneasy about this, set the d n precision to be a little higher than normal; you’ll get ugly garbage digits, but you’ll always be able to tell two distinct numbers apart.

An interesting side note is that most computers store their floating-point numbers in binary, and convert to decimal for display. Thus everyday programs have the same problem: Decimal 0.1 cannot be represented exactly in binary (try it: 0.1 d 2), so ‘0.1 * 10’ comes out as an inexact approximation to 1 on some machines (though they generally arrange to hide it from you by rounding off one digit as we did above). Because Calc works in decimal instead of binary, you can be sure that numbers that look exact are exact as long as you stay in decimal display mode.

It’s not hard to show that any number that can be represented exactly in binary, octal, or hexadecimal is also exact in decimal, so the kinds of problems we saw in this exercise are likely to be severe only when you use a relatively unusual radix like 3.