# Lua's Decimal Calculation

4 replies to this topic

### #1 vanhoa

vanhoa

Casio Overlord

• Members
• 854 posts
• Gender:Male
• Location:Vietnam

• Calculators:
AFX 2.0, CP 300, CP 330, nSpire, TI 89, FX 5800

Posted 08 September 2006 - 05:07 AM

Why the result of an exp got wrong (nealy true) when use lua.

### #2 Orwell

Orwell

Casio Overlord

• Members
• 777 posts
• Gender:Male
• Location:Paris - France

• Calculators:
Casio AFX 1.02 / Casio ClassPad 300

Posted 08 September 2006 - 08:50 AM

What makes you think this? Can you give an example?

### #3 Kilburn

Kilburn

Casio Technician

• Members
• 491 posts
• Gender:Male
• Location:France
• Interests:Blah

• Calculators:
FX-7500 G
ClassPad 300

Posted 09 September 2006 - 05:12 PM

Maybe this:

```print(789.000006)

Output: 789.00000599999998```

### #4 Orwell

Orwell

Casio Overlord

• Members
• 777 posts
• Gender:Male
• Location:Paris - France

• Calculators:
Casio AFX 1.02 / Casio ClassPad 300

Posted 09 September 2006 - 06:05 PM

floating point precision... Do we really need to explain that one more time?

### #5 PAP

PAP

Casio Overlord

• Members
• 681 posts
• Gender:Male
• Location:Somewhere in Europe.
• Interests:Computer Algebra, Numerical Analysis.

• Calculators:
ClassPad 300 (plus an old Casio model, with only a few Kb ram).

Posted 09 September 2006 - 08:26 PM

floating point precision... Do we really need to explain that one more time?

You are right, explaining the same thing again and again is annoying, but I will answer for the last time: This is not a bug. This is normal computer arithmetics which cannot be avoided, even on super computers. Using a computer means that you accept the fact that it has limited floating point precision.
In theory, it is possible to use arbitrary floating point precision, but this is not a good idea for the ClassPad, because it costs computation time... a lot of computation time.

#### 0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users