Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [eclipselink-users] Off topic, but perhaps relevant: setting doubles, no operations, but precision loss?

On Sat, Aug 28, 2010 at 7:05 AM, Frank Schwarz <fs5@xxxxxxx> wrote:
> Not only operations are imprecise but also (most of) the numbers itself.
> There is no double which has the value of 1.3d as 0.3d has no terminating
> binary representation: 0.3d = 0*0.5 + 1*0.25 + 0*0.125 + 0*0.0625 ... =
> 0.01001100(1100)b.

Thanks.  As always, this problem came down to something that was
staring us in the face and we didn't see it: an Informix FLOAT is
really a SQL DOUBLE PRECISION.  So when we send the fraction down the
wire as a Java float (oops), we had already lost precision (compared
to the C application which sent a C double down the wire).  We should,
of course, have called setDouble() on the prepared statement instead.

Thanks for your informative response.

Best,
Laird


Back to the top