PHP Sockets: Reconstructing Floating Point Numbers

Hi All,

I'm writing a program that connects to a data stream containing a number of
different variable types. Ie 4 & 8 byte integers, ASCII characters and 8
byte floats.

I use fgetc to capture the data in characters then I reconstruct the
variables from these bytes.

For integers this is easy, just multiply the bytes by orders of 256 to get
the right offset values. The thing I can't figure out is how to reconstruct
a double precision floating point number from its 8 component bytes.

Is there an easy way to do this in PHP? I couldn't find anything useful
through
Google...

Any pointers would be greatly appreciated.

-DM

Re: PHP Sockets: Reconstructing Floating Point Numbers

DM wrote:
> Hi All,
>
> I'm writing a program that connects to a data stream containing a number of
> different variable types. Ie 4 & 8 byte integers, ASCII characters and 8
> byte floats.
>
> I use fgetc to capture the data in characters then I reconstruct the
> variables from these bytes.
>
> For integers this is easy, just multiply the bytes by orders of 256 to get
> the right offset values. The thing I can't figure out is how to reconstruct
> a double precision floating point number from its 8 component bytes.
>
> Is there an easy way to do this in PHP? I couldn't find anything useful
> through
> Google...
>
> Any pointers would be greatly appreciated.
>
> -DM
>
>
>

Hi DM,

This data stream, where is it coming from?
What program is producing it?
Also PHP?

Regards,
Erwin Moller

Re: PHP Sockets: Reconstructing Floating Point Numbers

"Erwin Moller"
wrote in
message news:46fb87ea$0$244$e4fe514c@news.xs4all.nl...
> DM wrote:
>> Hi All,
>>
>> I'm writing a program that connects to a data stream containing a number
>> of
>> different variable types. Ie 4 & 8 byte integers, ASCII characters and 8
>> byte floats.
>>
>> I use fgetc to capture the data in characters then I reconstruct the
>> variables from these bytes.
>>
>> For integers this is easy, just multiply the bytes by orders of 256 to
>> get
>> the right offset values. The thing I can't figure out is how to
>> reconstruct
>> a double precision floating point number from its 8 component bytes.
>>
>> Is there an easy way to do this in PHP? I couldn't find anything useful
>> through
>> Google...
>>
>> Any pointers would be greatly appreciated.
>>
>> -DM
>>
>
> Hi DM,
>
> This data stream, where is it coming from?
> What program is producing it?
> Also PHP?
>
> Regards,
> Erwin Moller

Hi Erwin, thanks for the reply.

The data stream is coming from a custom Win32 Visual C++ real-time server
program which pumps out up to 1,000,000 of these multi-type messages per
day.

I would like to grab and parse this data, then insert it into a postgreSQL
database under Linux rather than looking for a Windows solution.

I'm looking for a way to get this data into the postgres database so I can
utillise a whole suite of PHP diagnostic & analysis program I wrote for
another similar application.

I had a look at the IEEE standard for double precision floats, but it looks
like I would have to do around 10-15 calculations per message to manually
restore the value. This would be impractical for the amount of through-put I
am facing...

-DM

Re: PHP Sockets: Reconstructing Floating Point Numbers

DM wrote:
> "Erwin Moller"
> wrote in
> message news:46fb87ea$0$244$e4fe514c@news.xs4all.nl...
>> DM wrote:
>>> Hi All,
>>>
>>> I'm writing a program that connects to a data stream containing a number
>>> of
>>> different variable types. Ie 4 & 8 byte integers, ASCII characters and 8
>>> byte floats.
>>>
>>> I use fgetc to capture the data in characters then I reconstruct the
>>> variables from these bytes.
>>>
>>> For integers this is easy, just multiply the bytes by orders of 256 to
>>> get
>>> the right offset values. The thing I can't figure out is how to
>>> reconstruct
>>> a double precision floating point number from its 8 component bytes.
>>>
>>> Is there an easy way to do this in PHP? I couldn't find anything useful
>>> through
>>> Google...
>>>
>>> Any pointers would be greatly appreciated.
>>>
>>> -DM
>>>
>> Hi DM,
>>
>> This data stream, where is it coming from?
>> What program is producing it?
>> Also PHP?
>>
>> Regards,
>> Erwin Moller
>
> Hi Erwin, thanks for the reply.
>

Hi,

> The data stream is coming from a custom Win32 Visual C++ real-time server
> program which pumps out up to 1,000,000 of these multi-type messages per
> day.
>
> I would like to grab and parse this data, then insert it into a postgreSQL
> database under Linux rather than looking for a Windows solution.

Good. (I love Postgres)

>
> I'm looking for a way to get this data into the postgres database so I can
> utillise a whole suite of PHP diagnostic & analysis program I wrote for
> another similar application.
>
> I had a look at the IEEE standard for double precision floats, but it looks
> like I would have to do around 10-15 calculations per message to manually
> restore the value. This would be impractical for the amount of through-put I
> am facing...

Shame.
Are you sure 10 to 15 calculations is too much?
Too slow?
I mean, if you only get 1 milion messages/day.
Would that be too much?

Maybe you can simply throw the 8 bytes straight into Postgres, and write
the bytes->float function in there?
PL/PGsql is bloody fast.

Maybe a trick exists in Postgres to do this more directly.
You could try your luck at their mailinglist: pgsql-general@postgresql.org


You could also simply put the 8 bytes into postgres, and calculate them
to real floats when you need them for diagnostics & analysis app.
That way you are sure your datastream gets in, since you don't need the
conversion right away.

Good luck.

Regards,
Erwin


>
> -DM
>
>

Re: PHP Sockets: Reconstructing Floating Point Numbers

DM wrote:
> "Erwin Moller"
> wrote in
> message news:46fb87ea$0$244$e4fe514c@news.xs4all.nl...
>> DM wrote:
>>> Hi All,
>>>
>>> I'm writing a program that connects to a data stream containing a number
>>> of
>>> different variable types. Ie 4 & 8 byte integers, ASCII characters and 8
>>> byte floats.
>>>
>>> I use fgetc to capture the data in characters then I reconstruct the
>>> variables from these bytes.
>>>
>>> For integers this is easy, just multiply the bytes by orders of 256 to
>>> get
>>> the right offset values. The thing I can't figure out is how to
>>> reconstruct
>>> a double precision floating point number from its 8 component bytes.
>>>
>>> Is there an easy way to do this in PHP? I couldn't find anything useful
>>> through
>>> Google...
>>>
>>> Any pointers would be greatly appreciated.
>>>
>>> -DM
>>>
>> Hi DM,
>>
>> This data stream, where is it coming from?
>> What program is producing it?
>> Also PHP?
>>
>> Regards,
>> Erwin Moller
>
> Hi Erwin, thanks for the reply.
>
> The data stream is coming from a custom Win32 Visual C++ real-time server
> program which pumps out up to 1,000,000 of these multi-type messages per
> day.
>
> I would like to grab and parse this data, then insert it into a postgreSQL
> database under Linux rather than looking for a Windows solution.
>
> I'm looking for a way to get this data into the postgres database so I can
> utillise a whole suite of PHP diagnostic & analysis program I wrote for
> another similar application.
>
> I had a look at the IEEE standard for double precision floats, but it looks
> like I would have to do around 10-15 calculations per message to manually
> restore the value. This would be impractical for the amount of through-put I
> am facing...
>
> -DM
>
>

Personally, if I were looking at this much data, I'd be looking at a
compiled language such as C/C++. If the data is coming in in binary,
just stuff it in the appropriate bytes in a variable and you're done.

Converting floating point variables is not fun, as you have found.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

Re: PHP Sockets: Reconstructing Floating Point Numbers

DM wrote:
> "Erwin Moller"
> wrote in
> message news:46fb87ea$0$244$e4fe514c@news.xs4all.nl...
>> DM wrote:
>>> Hi All,
>>>
>>> I'm writing a program that connects to a data stream containing a number
>>> of
>>> different variable types. Ie 4 & 8 byte integers, ASCII characters and 8
>>> byte floats.
>>>
>>> I use fgetc to capture the data in characters then I reconstruct the
>>> variables from these bytes.
>>>
>>> For integers this is easy, just multiply the bytes by orders of 256 to
>>> get
>>> the right offset values. The thing I can't figure out is how to
>>> reconstruct
>>> a double precision floating point number from its 8 component bytes.
>>>
>>> Is there an easy way to do this in PHP? I couldn't find anything useful
>>> through
>>> Google...
>>>
>>> Any pointers would be greatly appreciated.
>>>
>>> -DM
>>>
>> Hi DM,
>>
>> This data stream, where is it coming from?
>> What program is producing it?
>> Also PHP?
>>
>> Regards,
>> Erwin Moller
>
> Hi Erwin, thanks for the reply.
>
> The data stream is coming from a custom Win32 Visual C++ real-time server
> program which pumps out up to 1,000,000 of these multi-type messages per
> day.
>
> I would like to grab and parse this data, then insert it into a postgreSQL
> database under Linux rather than looking for a Windows solution.
>
> I'm looking for a way to get this data into the postgres database so I can
> utillise a whole suite of PHP diagnostic & analysis program I wrote for
> another similar application.
>
> I had a look at the IEEE standard for double precision floats, but it looks
> like I would have to do around 10-15 calculations per message to manually
> restore the value. This would be impractical for the amount of through-put I
> am facing...

Not necessarily: especially if coded in C or assembler ;-)

It may be possible to 'patch' the postgres stuff to do this.

You could create a new function in there calld 'myfloat' or somesuch and
do 'set var=myfloat(inputstuff)' at the data base level.

Or add a new function to PHP. It has to be relatively trivial.

Or - best of all - write your data->SQL program in C.

If you can buffer database insertions enough, it will be blindingly quick.

From what I remember of floating point stuff, there is usually some
bytes for the mantissa, and some for the exponent. Its easy enough to do
shift left/right of the exponent to get it nto a multiplicand and a
floating point multiply by the exponent nets you the result.

IUf you know the range, its simply a matter of taking the mantissa and
adding lots of zeros to each end, and pitting the decimal point in the
right place, a defined by the exponent. That does fail on VERY large or
small numbers tho.

Tell us what the format is, and we can decide the optimal algorithm:

a million messages a day is only what? 12 a second?. Hardly a hard task
for a decent processor.



>
> -DM
>
>

Re: PHP Sockets: Reconstructing Floating Point Numbers

Jerry Stuckle wrote:
> DM wrote:
>> "Erwin Moller"
>> wrote
>> in message news:46fb87ea$0$244$e4fe514c@news.xs4all.nl...
>>> DM wrote:
>>>> Hi All,
>>>>
>>>> I'm writing a program that connects to a data stream containing a
>>>> number of
>>>> different variable types. Ie 4 & 8 byte integers, ASCII characters
>>>> and 8
>>>> byte floats.
>>>>
>>>> I use fgetc to capture the data in characters then I reconstruct the
>>>> variables from these bytes.
>>>>
>>>> For integers this is easy, just multiply the bytes by orders of 256
>>>> to get
>>>> the right offset values. The thing I can't figure out is how to
>>>> reconstruct
>>>> a double precision floating point number from its 8 component bytes.
>>>>
>>>> Is there an easy way to do this in PHP? I couldn't find anything
>>>> useful through
>>>> Google...
>>>>
>>>> Any pointers would be greatly appreciated.
>>>>
>>>> -DM
>>>>
>>> Hi DM,
>>>
>>> This data stream, where is it coming from?
>>> What program is producing it?
>>> Also PHP?
>>>
>>> Regards,
>>> Erwin Moller
>>
>> Hi Erwin, thanks for the reply.
>>
>> The data stream is coming from a custom Win32 Visual C++ real-time
>> server program which pumps out up to 1,000,000 of these multi-type
>> messages per day.
>>
>> I would like to grab and parse this data, then insert it into a
>> postgreSQL database under Linux rather than looking for a Windows
>> solution.
>>
>> I'm looking for a way to get this data into the postgres database so I
>> can utillise a whole suite of PHP diagnostic & analysis program I
>> wrote for another similar application.
>>
>> I had a look at the IEEE standard for double precision floats, but it
>> looks like I would have to do around 10-15 calculations per message to
>> manually restore the value. This would be impractical for the amount
>> of through-put I am facing...
>>
>> -DM
>>
>
> Personally, if I were looking at this much data, I'd be looking at a
> compiled language such as C/C++. If the data is coming in in binary,
> just stuff it in the appropriate bytes in a variable and you're done.
>
> Converting floating point variables is not fun, as you have found.
>
I'm leaning towards that too.
C, with its ability to cast anything to anything, makes a dynamite comms
type language.

Ive written apache/SQL shims in C before - its not really worse than PHP
- just a bit slower to debug.

Re: PHP Sockets: Reconstructing Floating Point Numbers

DM wrote:
> Hi All,
>
> I'm writing a program that connects to a data stream containing a number of
> different variable types. Ie 4 & 8 byte integers, ASCII characters and 8
> byte floats.
>
> I use fgetc to capture the data in characters then I reconstruct the
> variables from these bytes.
>
> For integers this is easy, just multiply the bytes by orders of 256 to get
> the right offset values. The thing I can't figure out is how to reconstruct
> a double precision floating point number from its 8 component bytes.
>
> Is there an easy way to do this in PHP? I couldn't find anything useful
> through
> Google...
>
> Any pointers would be greatly appreciated.
>

You can try unpack()



--
gosha bine

extended php parser ~ http://code.google.com/p/pihipi
blok ~ http://www.tagarga.com/blok

Re: PHP Sockets: Reconstructing Floating Point Numbers

"The Natural Philosopher" wrote in message
news:1190902272.21408.1@proxy02.news.clara.net...
> Jerry Stuckle wrote:
>> DM wrote:
>>> "Erwin Moller"
>>> wrote in
>>> message news:46fb87ea$0$244$e4fe514c@news.xs4all.nl...
>>>> DM wrote:
>>>>> Hi All,
>>>>>
>>>>> I'm writing a program that connects to a data stream containing a
>>>>> number of
>>>>> different variable types. Ie 4 & 8 byte integers, ASCII characters and
>>>>> 8
>>>>> byte floats.
>>>>>
>>>>> I use fgetc to capture the data in characters then I reconstruct the
>>>>> variables from these bytes.
>>>>>
>>>>> For integers this is easy, just multiply the bytes by orders of 256 to
>>>>> get
>>>>> the right offset values. The thing I can't figure out is how to
>>>>> reconstruct
>>>>> a double precision floating point number from its 8 component bytes.
>>>>>
>>>>> Is there an easy way to do this in PHP? I couldn't find anything
>>>>> useful through
>>>>> Google...
>>>>>
>>>>> Any pointers would be greatly appreciated.
>>>>>
>>>>> -DM
>>>>>
>>>> Hi DM,
>>>>
>>>> This data stream, where is it coming from?
>>>> What program is producing it?
>>>> Also PHP?
>>>>
>>>> Regards,
>>>> Erwin Moller
>>>
>>> Hi Erwin, thanks for the reply.
>>>
>>> The data stream is coming from a custom Win32 Visual C++ real-time
>>> server program which pumps out up to 1,000,000 of these multi-type
>>> messages per day.
>>>
>>> I would like to grab and parse this data, then insert it into a
>>> postgreSQL database under Linux rather than looking for a Windows
>>> solution.
>>>
>>> I'm looking for a way to get this data into the postgres database so I
>>> can utillise a whole suite of PHP diagnostic & analysis program I wrote
>>> for another similar application.
>>>
>>> I had a look at the IEEE standard for double precision floats, but it
>>> looks like I would have to do around 10-15 calculations per message to
>>> manually restore the value. This would be impractical for the amount of
>>> through-put I am facing...
>>>
>>> -DM
>>>
>>
>> Personally, if I were looking at this much data, I'd be looking at a
>> compiled language such as C/C++. If the data is coming in in binary,
>> just stuff it in the appropriate bytes in a variable and you're done.
>>
>> Converting floating point variables is not fun, as you have found.
>>
> I'm leaning towards that too.
> C, with its ability to cast anything to anything, makes a dynamite comms
> type language.
>
> Ive written apache/SQL shims in C before - its not really worse than PHP -
> just a bit slower to debug.
>

Thanks for the advice.

It looks like the popular consensus is to write the program in another
language. I'm very rusty at C but I'll give it a go.

-DM