SvUOK always fails on 64bit platform

I have a dll exporting a function for perl which accepts an unsigned
interger parameter. In the exported function I use SvUOK to check what
the returned values will be by supplying different values from perl
script. Here are what I get

32bit
(in perl) (SvUOK return value)
0 - 0x7fffffff false
0x80000000 - 0xffffffff true

64bit
(in perl) (SvUOK return value)
0-0xffffffff false

I'm wondering why there's such a difference on 32 and 64 bit platform.
My thought is if SvUOK returns true then the passed parameter should
be an unsigned interger but it is true only on 32 bit platform. Do I
misuse this macro? How should I correct it? Thanks.

Re: SvUOK always fails on 64bit platform

Quoth cyl :
> I have a dll exporting a function for perl which accepts an unsigned
> interger parameter. In the exported function I use SvUOK to check what
> the returned values will be by supplying different values from perl
> script. Here are what I get
>
> 32bit
> (in perl) (SvUOK return value)
> 0 - 0x7fffffff false
> 0x80000000 - 0xffffffff true

0x7fffffff here is the largest value that fits into a 32bit IV.

> 64bit
> (in perl) (SvUOK return value)
> 0-0xffffffff false

0xffffffff easily fits into a 64bit IV, so that's what perl uses. If you
try numbers like 1<<63 that fit into an UV but not an IV, you'll find
they start becoming SvUOK again, so:

64bit value SvUOK
0 .. (1<<63 - 1) false
(1<<63) .. (1<<64 - 1) true

exactly equivalent to the 32bit table.

Ben

Re: SvUOK always fails on 64bit platform

On Nov 24, 4:20 am, Ben Morrow wrote:
..
..
> 64bit value SvUOK
> 0 .. (1<<63 - 1) false
> (1<<63) .. (1<<64 - 1) true
>

There's an interesting little catch here. On many 64-bit builds of
perl you'll find that whilst SvUOK returns true for 1<<63, it returns
false for 2**63 (because, on perls that don't also have long double
support, 2**63 is an NV). If your perl is also built with long double
support, then this issue does not arise.

I personally find that behaviour quite annoying ... but I seem to be
the only one, and it has been put to me that this would *not* be
trivial to fix .... so I reckon the behaviour is here to stay :-)

Cheers,
Rob

Re: SvUOK always fails on 64bit platform

Quoth sisyphus :
> On Nov 24, 4:20 am, Ben Morrow wrote:
> > 64bit value SvUOK
> > 0 .. (1<<63 - 1) false
> > (1<<63) .. (1<<64 - 1) true
>
> There's an interesting little catch here. On many 64-bit builds of
> perl you'll find that whilst SvUOK returns true for 1<<63, it returns
> false for 2**63 (because, on perls that don't also have long double
> support, 2**63 is an NV). If your perl is also built with long double
> support, then this issue does not arise.

Yes: I noticed 2**63 was a NV, which is why I used << instead :). It
seems that since ** is pow(3), it counts as a floating-point function.

This all illustrates another point, which is that depending on Perl
passing a particular representation of data to XS is usually a bad
idea, *especially* with IV/UV/NV which aren't even distinguishable from
Perl-space.

> I personally find that behaviour quite annoying ... but I seem to be
> the only one, and it has been put to me that this would *not* be
> trivial to fix .... so I reckon the behaviour is here to stay :-)

Well, yeah: given that ** is a floating-point operation, you no longer
have integer precision in the result:

~% perl -le'print for (1<<55)-1, sprintf "%.0f", 2**55-1'
36028797018963967
36028797018963968

though I have to say I don't really understand why this does a
floating-point subtract, when there's a whole lot of careful code in
both pp_pow and pp_subtract that tries to do arithmetic in integers where
possible... Hmmm. The logic in pp_pow goes something like

if (base or power non-integral)
use pow(3)

if (power negative)
use pow(3)

if (base is a power of 2) {
calculate by hand, in NVs

This gives a NV-only result when the result doesn't fit in a NV
with integer precision.
}
else {
if (
x**power fits in an UV, where x is the smallest power of 2
greater than base
) {
calculate by hand, in UVs

This gives a IV/UV result.
}
else
use pow(3)
}
}

but I don't really understand why the 'power of 2' branch isn't moved
inside the second 'else', so small powers of 2 to small powers are
calculated in UVs where possible.

For that matter, even if you are going to calculate in NVs, the result
is always going to be accurate (since it's a power of 2) so when the
result fits in an IV/UV at all the result should be I/UOK. That seems
like an easy bug to fix, so I must be missing something...

Ben

Re: SvUOK always fails on 64bit platform

Quoth Ben Morrow :
>
> possible... Hmmm. The logic in pp_pow goes something like
>
> if (base or power non-integral)
> use pow(3)
>
> if (power negative)
> use pow(3)
>
> if (base is a power of 2) {
> calculate by hand, in NVs
>
> This gives a NV-only result when the result doesn't fit in a NV
> with integer precision.
> }
> else {
> if (
> x**power fits in an UV, where x is the smallest power of 2
> greater than base
> ) {
> calculate by hand, in UVs
>
> This gives a IV/UV result.
> }
> else
> use pow(3)
> }
> }
>
> but I don't really understand why the 'power of 2' branch isn't moved
> inside the second 'else', so small powers of 2 to small powers are
> calculated in UVs where possible.

Sorry to be responding to myself, but having checked perlbrowse I now
think this was simply an oversight. The if(power of 2) logic is older
than the inner if, and was added to give a sane result for 2**n on
machines that have slightly insane long doubles. The latter logic was
added later, to make things like 3**33 that fit in a 64bit UV give
integer results. It looks like the possibility of doing the same for
powers of 2 was overlooked. Maybe I'll report a bug after 5.10's out.

Ben

Re: SvUOK always fails on 64bit platform

On Nov 24, 3:08 pm, Ben Morrow wrote:
> Quoth Ben Morrow :
..
..
> Maybe I'll report a bug after 5.10's out.

Good luck with it. Somewhere, within the p5p archives there should be
buried a thread called "-Duse64bitint on Cygwin (Win32)" where I
raised the issue (without receiving any sympathy).

I also made a couple of attempts to raise the issue on perlmonks ( eg
http://www.perlmonks.org/index.pl?node_id=617840 ), where again, it
was not received with any sympathy.

Ben, I lack the expertise and self confidence to pursue the matter
with any conviction, but if you can get somewhere with this, that's
all the better.

Cheers,
Rob

Re: SvUOK always fails on 64bit platform

Ben Morrow schreef:
> sisyphus:
>> Ben Morrow:

>>> 64bit value SvUOK
>>> 0 .. (1<<63 - 1) false
>>> (1<<63) .. (1<<64 - 1) true
>>
>> There's an interesting little catch here. On many 64-bit builds of
>> perl you'll find that whilst SvUOK returns true for 1<<63, it returns
>> false for 2**63 (because, on perls that don't also have long double
>> support, 2**63 is an NV). If your perl is also built with long double
>> support, then this issue does not arise.
>
> Yes: I noticed 2**63 was a NV, which is why I used << instead :). It
> seems that since ** is pow(3), it counts as a floating-point function.
>
> This all illustrates another point, which is that depending on Perl
> passing a particular representation of data to XS is usually a bad
> idea, *especially* with IV/UV/NV which aren't even distinguishable
> from Perl-space.
>
>> I personally find that behaviour quite annoying ... but I seem to be
>> the only one, and it has been put to me that this would *not* be
>> trivial to fix .... so I reckon the behaviour is here to stay :-)
>
> Well, yeah: given that ** is a floating-point operation, you no longer
> have integer precision in the result:
>
> ~% perl -le'print for (1<<55)-1, sprintf "%.0f", 2**55-1'
> 36028797018963967
> 36028797018963968
>
> though I have to say I don't really understand why this does a
> floating-point subtract, when there's a whole lot of careful code in
> both pp_pow and pp_subtract that tries to do arithmetic in integers
> where possible... Hmmm. The logic in pp_pow goes something like
>
> if (base or power non-integral)
> use pow(3)
>
> if (power negative)
> use pow(3)
>
> if (base is a power of 2) {
> calculate by hand, in NVs
>
> This gives a NV-only result when the result doesn't fit in a
> NV with integer precision.
> }
> else {
> if (
> x**power fits in an UV, where x is the smallest power of 2
> greater than base
> ) {
> calculate by hand, in UVs
>
> This gives a IV/UV result.
> }
> else
> use pow(3)
> }
> }
>
> but I don't really understand why the 'power of 2' branch isn't moved
> inside the second 'else', so small powers of 2 to small powers are
> calculated in UVs where possible.
>
> For that matter, even if you are going to calculate in NVs, the result
> is always going to be accurate (since it's a power of 2) so when the
> result fits in an IV/UV at all the result should be I/UOK. That seems
> like an easy bug to fix, so I must be missing something...

Let's ask p5p.

--
Affijn, Ruud

"Gewoon is een tijger."

Re: SvUOK always fails on 64bit platform

[A complimentary Cc of this posting was sent to
cyl
], who wrote in article <6194303c-a628-48fa-9c95-39d4c1b58d98@p69g2000hsa.googlegroups.com>:
> I have a dll exporting a function for perl which accepts an unsigned
> interger parameter. In the exported function I use SvUOK to check what
> the returned values will be by supplying different values from perl
> script. Here are what I get
>
> 32bit
> (in perl) (SvUOK return value)
> 0 - 0x7fffffff false
> 0x80000000 - 0xffffffff true
>
> 64bit
> (in perl) (SvUOK return value)
> 0-0xffffffff false
>
> I'm wondering why there's such a difference on 32 and 64 bit platform.
> My thought is if SvUOK returns true then the passed parameter should
> be an unsigned interger but it is true only on 32 bit platform. Do I
> misuse this macro? How should I correct it? Thanks.

One other thing to mention: SvUOK() is not Perl. You are not supposed
to TOUCH any thing not Perl unless you are capable to understand
YOURSELF what it means... Kind of self-censoreship: if you have to
ask, you are not ready to USE it for quite some time.

SvUOK() is concerned with the INTERNAL REPRESENATION of a Perl value,
not with the value itself. This says NOTHING about the value, but
says something about "the history of how it was obtained". Some
operations will ALWAYS produce SvUV; some will never; some will
differentiate on the range of values (as you observe).

Hope this helps,
Ilya

Re: SvUOK always fails on 64bit platform

On 11月24日, 上午1時20分, Ben Morrow wrote:
> Quoth cyl :
> > 0 - 0x7fffffff false
> > 0x80000000 - 0xffffffff true
>
> 0x7fffffff here is the largest value that fits into a 32bit IV.
>
> > 64bit
> > (in perl) (SvUOKreturn value)
> > 0-0xffffffff false
>
> 0xffffffff easily fits into a 64bit IV, so that's what perl uses. If you

I thought 0xffffffff should be the max value of unsigned integer on
either 32 bit or 64 bit platform. Is this correct? Actually the
problem comes from the SWIG (http://www.swig.org) implementation. I
post the generated code below.

On 64bit platform, when I pass values larger than 0x7fffffff to
TestInt, the function SWIG_AsVal_unsigned_SS_long at the bottom will
be called with parameter ST(0). Since SvUOK returns false, SvIV is
used to convert the value. In this case, values latger than 0x7fffffff
become negative and an OverflowError is thrown. I'm wondering what
changes should I make to correct this problem. Will there be any
problem if I call SvUV directly without any type check?

XS(_wrap_TestInt) {
{
unsigned int arg1 ;
unsigned int result;
unsigned int val1 ;
int ecode1 = 0 ;
int argvi = 0;
dXSARGS;

if ((items < 1) || (items > 1)) {
SWIG_croak("Usage: TestInt(unsigned int);");
}
ecode1 = SWIG_AsVal_unsigned_SS_int SWIG_PERL_CALL_ARGS_2(ST(0),
&val1);
if (!SWIG_IsOK(ecode1)) {
SWIG_exception_fail(SWIG_ArgError(ecode1), "in method '"
"TestInt" "', argument " "1"" of type '" "unsigned int""'");
}
arg1 = (unsigned int)(val1);
result = (unsigned int)TestInt(arg1);
ST(argvi) = SWIG_From_unsigned_SS_int
SWIG_PERL_CALL_ARGS_1((unsigned int)(result)); argvi++ ;

XSRETURN(argvi);
fail:

SWIG_croak_null();
}
}

SWIGINTERN int
SWIG_AsVal_unsigned_SS_int SWIG_PERL_DECL_ARGS_2(SV * obj, unsigned
int *val)
{
unsigned long v;
int res = SWIG_AsVal_unsigned_SS_long SWIG_PERL_CALL_ARGS_2(obj,
&v);
if (SWIG_IsOK(res)) {
if ((v > UINT_MAX)) {
return SWIG_OverflowError;
} else {
if (val) *val = (unsigned int)(v);
}
}
return res;
}

SWIGINTERN int
SWIG_AsVal_unsigned_SS_long SWIG_PERL_DECL_ARGS_2(SV *obj, unsigned
long *val)
{
if (SvUOK(obj)) {
if (val) *val = SvUV(obj);
return SWIG_OK;
} else if (SvIOK(obj)) {
long v = SvIV(obj);
if (v >= 0) {
if (val) *val = v;
return SWIG_OK;
} else {
return SWIG_OverflowError;
}
} else {
int dispatch = 0;
const char *nptr = SvPV(obj, PL_na);
if (nptr) {
char *endptr;
unsigned long v = strtoul(nptr, &endptr,0);
if (errno == ERANGE) {
errno = 0;
return SWIG_OverflowError;
} else {
if (*endptr == '\0') {
if (val) *val = v;
return SWIG_Str2NumCast(SWIG_OK);
}
}
}
if (!dispatch) {
double d;
int res = SWIG_AddCast(SWIG_AsVal_double
SWIG_PERL_CALL_ARGS_2(obj,&d));
if (SWIG_IsOK(res) && SWIG_CanCastAsInteger(&d, 0, ULONG_MAX)) {
if (val) *val = (unsigned long)(d);
return res;
}
}
}
return SWIG_TypeError;
}

Re: SvUOK always fails on 64bit platform

Quoth cyl :
> On ?????, Ben Morrow wrote:
> > Quoth cyl :
> > > 0 - 0x7fffffff false
> > > 0x80000000 - 0xffffffff true
> >
> > 0x7fffffff here is the largest value that fits into a 32bit IV.
> >
> > > 64bit
> > > (in perl) (SvUOKreturn value)
> > > 0-0xffffffff false
> >
> > 0xffffffff easily fits into a 64bit IV, so that's what perl uses. If you
>
> I thought 0xffffffff should be the max value of unsigned integer on
> either 32 bit or 64 bit platform. Is this correct?

Yes, 0xffffffff is 2**32-1, and the largest unsigned 32bit value.
0x7fffffff is the largest *signed* 32bit value, and perl always prefers
to store things signed when it can. When a value fits into a signed
integer, it's stored as an IV and SvUOK is false.

The largest unsigned 64bit integer is 0xffffffffffffffff (that's 16 fs),
and the largest signed half that, 0x7fffffffffffffff. On a perl with
64bit IVs, any value less than this will be stored as an IV, and will
not be SvUOK.

> Actually the
> problem comes from the SWIG (http://www.swig.org) implementation. I
> post the generated code below.
>
> On 64bit platform,

Stop right there :). Do you mean a platform with 64bit ints, or a perl
with 64bit IVs? They are not equivalent: my perl here, for instance, is
configured for i386-freebsd-64int, so I have 64bit IVs even though my
ints are only 32bit. perl will use long long for IVs if you ask it to.

> when I pass values larger than 0x7fffffff to
> TestInt, the function SWIG_AsVal_unsigned_SS_long at the bottom will
> be called with parameter ST(0). Since SvUOK returns false,

Because the value fits into an IV.

> SvIV is used to convert the value.

*And then you cast it to (signed) long*, which is shorter than IV.
(Presumably you're running this on a 32bit platform with a 64bit perl:
you didn't say, but that's the only situation that fits.) Make the
temporary an IV, or, since you're checking SvIOK anyway, just use
SvIVX(obj) directly.

Why are you doing any of this? If you want to coerce a SV to an UV,
that's what SvUV does for you, including converting from a string or
float if necessary. I would have said the whole function would be better
written

UV uv = SvUV(obj);
*val = (unsigned long) uv;

if (*val != uv)
return SWIG_OverflowError;

return SWIG_OK;

if indeed you need that much error checking. Why aren't you just using
the typemap entry for unsigned long? That's what it's there for.

> In this case, values latger than
> 0x7fffffff become negative and an OverflowError is thrown. I'm
> wondering what changes should I make to correct this problem. Will
> there be any problem if I call SvUV directly without any type check?

No. SvUV is a coercion function. SvUVX is direct access to the 'UV' slot
of the SV (SVs don't really have an UV slot, but the IV slot sometimes
holds an UV instead), and using that without checking *is* dangerous.
Have you checked perlapi?


> SWIGINTERN int
> SWIG_AsVal_unsigned_SS_long SWIG_PERL_DECL_ARGS_2(SV *obj, unsigned
> long *val)
> {
> if (SvUOK(obj)) {
> if (val) *val = SvUV(obj);
> return SWIG_OK;
> } else if (SvIOK(obj)) {
> long v = SvIV(obj);
> if (v >= 0) {
> if (val) *val = v;
> return SWIG_OK;
> } else {
> return SWIG_OverflowError;
> }
> } else {

You really didn't need to post more code than this. It illustrates the
whole of your problem.

Ben

Re: SvUOK always fails on 64bit platform

[A complimentary Cc of this posting was sent to
cyl
], who wrote in article <46372675-8f8c-4703-9d68-e135639c5ea0@d27g2000prf.googlegroups.com>:
> SWIGINTERN int
> SWIG_AsVal_unsigned_SS_int SWIG_PERL_DECL_ARGS_2(SV * obj, unsigned
> int *val)
> {
> unsigned long v;
> int res = SWIG_AsVal_unsigned_SS_long SWIG_PERL_CALL_ARGS_2(obj,
> &v);
> if (SWIG_IsOK(res)) {
> if ((v > UINT_MAX)) {
> return SWIG_OverflowError;
> } else {
> if (val) *val = (unsigned int)(v);
> }
> }
> return res;
> }

IMO, you should disregard the direct criticism of the other reply to
your posting (the general musing there is correct, but applications to
your code look misplaced).

I think your problems stem from misunderstanding of C, not of
Perl/XS. Your code above:

int res = function_returning_long();

makes no sense in the context.

> SWIGINTERN int
> SWIG_AsVal_unsigned_SS_long SWIG_PERL_DECL_ARGS_2(SV *obj, unsigned
> long *val)
> {
> if (SvUOK(obj)) {
> if (val) *val = SvUV(obj);

THis is again: you put UV into an unsigned long slot. It makes sense
ONLY IF sizeof(UV) <= sizeof(long) - which will not hold on many
builds of Perl.

[Perl defines C macros with sizeof() of most useful types (and the
corresponding MAX/MIN), so you can write conditional code. Extract
SvUV to an UV variable, then compare it with UNSIGNED_LONG_MAX
(sp?) - and remember that the way to write comparison depends on
the relative sizes of types!]

Hope this helps,
Ilya

Re: SvUOK always fails on 64bit platform

Quoth Ilya Zakharevich :
> [A complimentary Cc of this posting was sent to
> cyl
> ], who wrote in article
> <46372675-8f8c-4703-9d68-e135639c5ea0@d27g2000prf.googlegroups.com>:
>
> IMO, you should disregard the direct criticism of the other reply to
> your posting (the general musing there is correct, but applications to
> your code look misplaced).

If anything I've said is incorrect or inapplicable I'll gladly accept
correction. I know you know the guts of perl better than I do :).

Ben

Re: SvUOK always fails on 64bit platform

[A complimentary Cc of this posting was sent to
Ben Morrow
], who wrote in article <76v325-9e4.ln1@osiris.mauzo.dyndns.org>:
> > ], who wrote in article
> > <46372675-8f8c-4703-9d68-e135639c5ea0@d27g2000prf.googlegroups.com>:

> > IMO, you should disregard the direct criticism of the other reply to
> > your posting (the general musing there is correct, but applications to
> > your code look misplaced).

> If anything I've said is incorrect or inapplicable I'll gladly accept
> correction. I know you know the guts of perl better than I do :).

I was not discussing "guts of perl". I was discussing your opinion
about "which places in the supplied C code need improvement". IMO, it
was too vague and misplaced.

Hope this helps,
Ilya

Re: SvUOK always fails on 64bit platform

I think I'll symply remove the SvUOK check and use SvUV directly. Hope
this can work. Thanks for all the valuable informations even though I
still have lots of question marks in my head.