<%=1/15%> retuns 6.66666666666667E-02

yes <%=1/15%> retuns 6.66666666666667E-02

anyone knows why or where i am wrong?

Re: <%=1/15%> retuns 6.66666666666667E-02

..nLL wrote:
> yes <%=1/15%> retuns 6.66666666666667E-02
>
> anyone knows why or where i am wrong?

Take a look at this series of blog articles about floating point
arithmetic. It starts here:
http://blogs.msdn.com/ericlippert/archive/2005/01/10/350108. aspx

You can find the rest of the series using the link to the chronological
list of posts on the right side of the page.

--
Microsoft MVP -- ASP/ASP.NET
Please reply to the newsgroup. The email account listed in my From
header is my spam trap, so I don't check it very often. You will get a
quicker response by posting to the newsgroup.

Re: <%=1/15%> retuns 6.66666666666667E-02

This is a bit complicated than i expected. is there any function that
will help me to return proper result (like 0.06) from 1/15?

Thank you


On Jun 25, 1:48 pm, "Bob Barrows [MVP]"
wrote:
> .nLL wrote:
> > yes <%=1/15%> retuns 6.66666666666667E-02
>
> > anyone knows why or where i am wrong?
>
> Take a look at this series of blog articles about floating point
> arithmetic. It starts here:http://blogs.msdn.com/ericlippert/archive/2005/01/10/35 0108.aspx
>
> You can find the rest of the series using the link to the chronological
> list of posts on the right side of the page.
>
> --
> Microsoft MVP -- ASP/ASP.NET
> Please reply to the newsgroup. The email account listed in my From
> header is my spam trap, so I don't check it very often. You will get a
> quicker response by posting to the newsgroup.

Re: <%=1/15%> retuns 6.66666666666667E-02

OK, a simple answer:
No.

The correctly rounded result is 0.07, so if you want the result to be
"0.06", you will need to write complicated code to round the result the
way you want to (why do you want to round down?).

There is no substitute to learning how to properly deal with f-p
arithmetic. This will not be the last time you encounter this.

For a simple answer, try the vbscript FormatNumber function.

..nLL wrote:
> This is a bit complicated than i expected. is there any function that
> will help me to return proper result (like 0.06) from 1/15?
>
> Thank you
>
>
> On Jun 25, 1:48 pm, "Bob Barrows [MVP]"
> wrote:
>> .nLL wrote:
>>> yes <%=1/15%> retuns 6.66666666666667E-02
>>
>>> anyone knows why or where i am wrong?
>>
>> Take a look at this series of blog articles about floating point
>> arithmetic. It starts
>> here:http://blogs.msdn.com/ericlippert/archive/2005/01/10/35 0108.aspx
>>
>> You can find the rest of the series using the link to the
>> chronological
>> list of posts on the right side of the page.
>>
>> --
>> Microsoft MVP -- ASP/ASP.NET
>> Please reply to the newsgroup. The email account listed in my From
>> header is my spam trap, so I don't check it very often. You will get
>> a
>> quicker response by posting to the newsgroup.

--
Microsoft MVP -- ASP/ASP.NET
Please reply to the newsgroup. The email account listed in my From
header is my spam trap, so I don't check it very often. You will get a
quicker response by posting to the newsgroup.

Re: <%=1/15%> retuns 6.66666666666667E-02

thank you formatnumber was the answer i was looking for


On Jun 25, 2:41 pm, "Bob Barrows [MVP]"
wrote:
> OK, a simple answer:
> No.
>
> The correctly rounded result is 0.07, so if you want the result to be
> "0.06", you will need to write complicated code to round the result the
> way you want to (why do you want to round down?).
>
> There is no substitute to learning how to properly deal with f-p
> arithmetic. This will not be the last time you encounter this.
>
> For a simple answer, try the vbscript FormatNumber function.
>
>
>
> .nLL wrote:
> > This is a bit complicated than i expected. is there any function that
> > will help me to return proper result (like 0.06) from 1/15?
>
> > Thank you
>
> > On Jun 25, 1:48 pm, "Bob Barrows [MVP]"
> > wrote:
> >> .nLL wrote:
> >>> yes <%=1/15%> retuns 6.66666666666667E-02
>
> >>> anyone knows why or where i am wrong?
>
> >> Take a look at this series of blog articles about floating point
> >> arithmetic. It starts
> >> here:http://blogs.msdn.com/ericlippert/archive/2005/01/10/35 0108.aspx
>
> >> You can find the rest of the series using the link to the
> >> chronological
> >> list of posts on the right side of the page.
>
> >> --
> >> Microsoft MVP -- ASP/ASP.NET
> >> Please reply to the newsgroup. The email account listed in my From
> >> header is my spam trap, so I don't check it very often. You will get
> >> a
> >> quicker response by posting to the newsgroup.
>
> --
> Microsoft MVP -- ASP/ASP.NET
> Please reply to the newsgroup. The email account listed in my From
> header is my spam trap, so I don't check it very often. You will get a
> quicker response by posting to the newsgroup.