threads, XSUB allocated memory, destructors, destruction

I have a perl module built out of XSUBs.
The functions malloc() space, build structures and return
pointers to perl. Perl calls their destructor routines with no
problems. The C structures go back to the perl interpreter, as
T_PTROBJ things, specified in a typemap file.

Now, if I use threads, I
make lots of new data
break my arrays into 2 pieces
threads->new( myfunc, @array_to_read_from);
threads->new( myfunc, @array_to_read_from);
thread[0]->join
thread[1]->join

Unfortunately, at the join() stage, each thread decides to
cleanup, and call the destructors for my @array_to_read_from.
Obviously free() gets called multiple times, terrible things
happen to the heap and everything dies.

What should I be doing in my XSUBs when I create objects (really
just pointers to malloc()'d space) to ask perl not to do this ?
I am sure I am missing something obvious, but I cannot find it in
perlxs or perlxstut man pages.
Many thanks for any advice.