Numerical Stability and Error

Numerical Stability and Error

The equality (0.1 + 0.2) == 0.3 seems completely obvious. Yet, due to how computers store floating-point numbers, this equality does not hold in many programming languages—evaluating to false.

On the other hand, why does (0.5 + 0.25) == 0.75 evaluate to true?

When performing floating-point arithmetic, errors inevitably arise to some degree. Below, we explore why these errors happen and what computational steps can be taken to mitigate them.

How Floating-Point Numbers Are Stored in Computers

Floating-point numbers are represented in computers as sequences of bits governed by the IEEE 754 standard. Single-precision floating-point numbers use 32 bits, while double-precision numbers use 64 bits. These bits are divided into a sign bit, a fraction (significand/mantissa), and an exponent.

• Sign bit (1 bit): 0 for positive, 1 for negative
• Fraction
• Exponent

Taking single precision as an example: with an 8-bit exponent, the representable range is 0 to 255. However, because negative exponents must also be represented, IEEE 754 specifies an exponent bias that must be added to obtain the value actually stored. For single precision, this bias is 127.

For example, the floating-point representation of -14.75 works as follows:

• Sign bit: Negative, so it is 1
• Exponent: First, convert 14.75 to binary, yielding 1110.11. Normalizing this gives 1.11011 * 2^3. The exponent is 3; adding the bias of 127 gives 130, which in binary is 10000010.
• Fraction: 11011, with the remaining bits padded with 0.

Thus, the floating-point representation of -14.75 is: 1 10000010 11011000000000000000000

From this, we can also understand where errors originate. When the exponent is negative, the resulting values fall strictly between 0 < x < 1. Therefore, any number that cannot be represented as a finite sum of powers of two (2^-x) can only be approximated rather than represented exactly. As mentioned earlier, 0.5 + 0.25 does not incur error because these numbers can be expressed as 2^-1 + 2^-2.

Why Store Numbers This Way?

Using scientific notation allows us to handle numbers across vastly different scales while maintaining a consistent level of precision. Numbers like 0.00000012345 and 1234567890000 can be written in scientific notation as 1.2345E-7 and 1.23456789E12. Computers typically store these values in floating-point format, which operates on the same principle as scientific notation, just in binary rather than decimal.

There are infinitely many real numbers, but computer memory is finite. Consequently, regardless of the storage format, errors are bound to occur. What we can do is navigate the trade-off between precision and the representable range of numbers.

Significant Digits

Significant digits serve as an indicator of precision. Numbers such as 0.001 or 0.0135 can be written as $1 \times 10^{-3}$ and $1.35 \times 10^{-2}$. In these cases, 0.001 has 1 significant digit, while 1.35 has 3 significant digits. The more significant digits there are, the higher the precision.

Wikipedia provides a simple set of rules for determining significant digits:

• All non-zero digits are significant.
• Zeros between non-zero digits are significant.
• Leading zeros are never significant.
• For numbers containing a decimal point, trailing zeros (zeros after the last non-zero digit) are significant.
• For numbers without a decimal point, trailing zeros may or may not be significant, depending on additional notation or error margins.

Cancellation of Significant Digits

When subtracting two floating-point numbers with nearly identical absolute values, because most of their leading digits match, the result leaves many leading zeros, leading to a loss of significant digits. This phenomenon is known as the cancellation of significant digits (or catastrophic cancellation).

For example, with (1.234567890 - 1.234567889), although the result is 0.000000001, under insufficient precision it may round down to 0.

This requires special attention in numerical computing. Take the half-angle formula as an example:

When the angle is very small, $\cos\theta$ is extremely close to 1. Subtracting it from 1 makes it very easy to lose significant digits. For instance, when the angle is 1 degree, using the formula (assuming 6 significant digits of precision):

If we look up the sine value directly:

Notice that the discrepancy between the two results is quite pronounced. Extreme care must be taken in numerical calculations. A few ways to prevent loss of significance include:

• Avoid subtracting two numbers with very close absolute values whenever possible.
• Use alternative formulas for computation (such as rearranging the trigonometric formula above).
  • The goal is ultimately to avoid subtracting nearly equal numbers.
• Increase computational precision.

Conclusion

Most experienced engineers are aware that floating-point arithmetic involves errors and know why 0.1 + 0.2 != 0.3. This article took a deeper look into fractional storage, IEEE 754, and the cancellation of significant digits.

Trigonometric identities like the half-angle and double-angle formulas were staple exam questions back in high school math—at the time, it just felt like plugging numbers into formulas.

In practice, however, calculations are carried out by computers, real-world problems rarely hand you neat angles like 30 or 60 degrees, and math teachers certainly never warned you about catastrophic cancellation in floating-point arithmetic.