Welcome Guest. Sign in or Signup

2 Answers

Flight Computer Error? Does anyone know why?

Asked by: 4910 views ,
General Aviation

I just got a CX-2 electronic flight computer.

I know that from 29.93 to 29.92 is 10 feet.

But I was playing around on my CX-2.  I entered 29.93 - 29.92 and got 0.01.  THEN, I multiplied that by 1000 and got 10.00023.  Odd.  I'm wondering why it didn't just display "10."

Consequently, when I just enter 0.01 x 1000 it DOES give me 10.

I didn't know if the 10.00023 was an error or if anyone had any idea why it might display that?  And yes, I tried it multiple times.

Ace Any FAA Written Test!
Actual FAA Questions / Free Lifetime Updates
The best explanations in the business
Fast, efficient study.
Pass Your Checkride With Confidence!
FAA Practical Test prep that reflects actual checkrides.
Any checkride: Airplane, Helicopter, Glider, etc.
Written and maintained by actual pilot examiners and master CFIs.
The World's Most Trusted eLogbook
Be Organized, Current, Professional, and Safe.
Highly customizable - for student pilots through pros.
Free Transition Service for users of other eLogs.
Our sincere thanks to pilots such as yourself who support AskACFI while helping themselves by using the awesome PC, Mac, iPhone/iPad, and Android aviation apps of our sponsors.

2 Answers



  1. Brian on Nov 06, 2010

    It is the degree of accuracy that computer accounts for. Nobody really cares about 23 ten thousandths of a foot (.07 millimeters), but someone will say the computer is better for it. 🙂

    The 0.01 you’re seeing in your first example is really 0.01000023 (9 digits). The computer just isn’t showing the remaining 6 digits, probably due to some limit in display, until you multiply and get 10.00023 (7 digits).

    +1 Votes Thumb up 1 Votes Thumb down 0 Votes



  2. Koehn on Nov 06, 2010

    Computers work in binary, even when doing math. There’s no accurate way to represent 1/100 in binary (just as there’s no accurate way to represent 1/3 in ten decimal digits), so the computer has to round 1/100 to a number that’s close, but not quite the same value. When you multiply the not-quite-accurate number, you multiply the error too, and get a not-quite-accurate result.
    See this for a thorough explanation of floating point math.

    +1 Votes Thumb up 1 Votes Thumb down 0 Votes


The following terms have been auto-detected the question above and any answers or discussion provided. Click on a term to see its definition from the Dauntless Aviation JargonBuster Glossary.

Answer Question

Our sincere thanks to all who contribute constructively to this forum in answering flight training questions. If you are a flight instructor or represent a flight school / FBO offering flight instruction, you are welcome to include links to your site and related contact information as it pertains to offering local flight instruction in a specific geographic area. Additionally, direct links to FAA and related official government sources of information are welcome. However we thank you for your understanding that links to other sites or text that may be construed as explicit or implicit advertising of other business, sites, or goods/services are not permitted even if such links nominally are relevant to the question asked.