Go to the first, previous, next, last section, table of contents.

Arithmetic Tutorial Exercise 1

Dividing two integers that are larger than the current precision may give a floating-point result that is inaccurate even when rounded down to an integer. Consider 123456789 / 2 when the current precision is 6 digits. The true answer is 61728394.5, but with a precision of 6 this will be rounded to @c{$12345700.0/2.0 = 61728500.0$} 12345700. / 2. = 61728500.. The result, when converted to an integer, will be off by 106.

Here are two solutions: Raise the precision enough that the floating-point round-off error is strictly to the right of the decimal point. Or, convert to fraction mode so that 123456789 / 2 produces the exact fraction 123456789:2, which can be rounded down by the F command without ever switching to floating-point format.


Go to the first, previous, next, last section, table of contents.