Jump to content



Photo

Lua's Decimal Calculation


  • Please log in to reply
4 replies to this topic

#1 vanhoa

vanhoa

    Casio Overlord

  • Members
  • PipPipPipPipPipPipPip
  • 854 posts
  • Gender:Male
  • Location:Vietnam

  • Calculators:
    AFX 2.0, CP 300, CP 330, nSpire, TI 89, FX 5800

Posted 08 September 2006 - 05:07 AM

Why the result of an exp got wrong (nealy true) when use lua.

#2 Orwell

Orwell

    Casio Overlord

  • Members
  • PipPipPipPipPipPipPip
  • 777 posts
  • Gender:Male
  • Location:Paris - France

  • Calculators:
    Casio AFX 1.02 / Casio ClassPad 300

Posted 08 September 2006 - 08:50 AM

What makes you think this? Can you give an example?

#3 Kilburn

Kilburn

    Casio Technician

  • Members
  • PipPipPipPipPipPip
  • 491 posts
  • Gender:Male
  • Location:France
  • Interests:Blah

  • Calculators:
    FX-7500 G
    ClassPad 300

Posted 09 September 2006 - 05:12 PM

Maybe this:

print(789.000006)

Output: 789.00000599999998

:unsure:

#4 Orwell

Orwell

    Casio Overlord

  • Members
  • PipPipPipPipPipPipPip
  • 777 posts
  • Gender:Male
  • Location:Paris - France

  • Calculators:
    Casio AFX 1.02 / Casio ClassPad 300

Posted 09 September 2006 - 06:05 PM

floating point precision... Do we really need to explain that one more time? :rolleyes:

#5 PAP

PAP

    Casio Overlord

  • Members
  • PipPipPipPipPipPipPip
  • 681 posts
  • Gender:Male
  • Location:Somewhere in Europe.
  • Interests:Computer Algebra, Numerical Analysis.

  • Calculators:
    ClassPad 300 (plus an old Casio model, with only a few Kb ram).

Posted 09 September 2006 - 08:26 PM

floating point precision... Do we really need to explain that one more time? :rolleyes:

You are right, explaining the same thing again and again is annoying, but I will answer for the last time: This is not a bug. This is normal computer arithmetics which cannot be avoided, even on super computers. Using a computer means that you accept the fact that it has limited floating point precision.
In theory, it is possible to use arbitrary floating point precision, but this is not a good idea for the ClassPad, because it costs computation time... a lot of computation time.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users