6.2 Precision

The p (calc-precision) command controls the precision to which floating-point calculations are carried. The precision must be at least 3 digits and may be arbitrarily high, within the limits of memory and time. This affects only floats: Integer and rational calculations are always carried out with as many digits as necessary.

The p key prompts for the current precision. If you wish you can instead give the precision as a numeric prefix argument.

Many internal calculations are carried to one or two digits higher precision than normal. Results are rounded down afterward to the current precision. Unless a special display mode has been selected, floats are always displayed with their full stored precision, i.e., what you see is what you get. Reducing the current precision does not round values already on the stack, but those values will be rounded down before being used in any calculation. The c 0 through c 9 commands (see Conversions) can be used to round an existing value to a new precision.

It is important to distinguish the concepts of precision and accuracy. In the normal usage of these words, the number 123.4567 has a precision of 7 digits but an accuracy of 4 digits. The precision is the total number of digits not counting leading or trailing zeros (regardless of the position of the decimal point). The accuracy is simply the number of digits after the decimal point (again not counting trailing zeros). In Calc you control the precision, not the accuracy of computations. If you were to set the accuracy instead, then calculations like ‘exp(100)’ would generate many more digits than you would typically need, while ‘exp(-100)’ would probably round to zero! In Calc, both these computations give you exactly 12 (or the requested number of) significant digits.

The only Calc features that deal with accuracy instead of precision are fixed-point display mode for floats (d f; see Float Formats), and the rounding functions like floor and round (see Integer Truncation). Also, c 0 through c 9 deal with both precision and accuracy depending on the magnitudes of the numbers involved.

If you need to work with a particular fixed accuracy (say, dollars and cents with two digits after the decimal point), one solution is to work with integers and an “implied” decimal point. For example, $8.99 divided by 6 would be entered 899 RET 6 /, yielding 149.833 (actually $1.49833 with our implied decimal point); pressing R would round this to 150 cents, i.e., $1.50.

See Floats, for still more on floating-point precision and related issues.