Unix compress equivalent

Hi all

What is the equivalent in unix (more specifically python) to the compress()
function.

I am trying to make csv file for use with load data infile and am wondering
how to compress the strings that I would usually compress with compress() in
a regular sql statement. The field I am writing this into is longblob and I
need the compressed version here to be identical to what compress() would do

Thanks
olaf

----------------------------------------- Confidentiality Notice:
The following mail message, including any attachments, is for the
sole use of the intended recipient(s) and may contain confidential
and privileged information. The recipient is responsible to
maintain the confidentiality of this information and to use the
information only for authorized purposes. If you are not the
intended recipient (or authorized to receive information for the
intended recipient), you are hereby notified that any review, use,
disclosure, distribution, copying, printing, or action taken in
reliance on the contents of this e-mail is strictly prohibited. If
you have received this communication in error, please notify us
immediately by reply e-mail and destroy all copies of the original
message. Thank you.

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

Re: Unix compress equivalent

Or even better, can I tell load data infile or somewhere in the table
definition to compress whatever is written to the file?

Thanks
Olaf


On 5/8/09 12:29 PM, "Olaf Stein" wrote:

> Hi all
>
> What is the equivalent in unix (more specifically python) to the compress()
> function.
>
> I am trying to make csv file for use with load data infile and am wondering
> how to compress the strings that I would usually compress with compress() in
> a regular sql statement. The field I am writing this into is longblob and I
> need the compressed version here to be identical to what compress() would do
>
> Thanks
> olaf
>
> ----------------------------------------- Confidentiality Notice:
> The following mail message, including any attachments, is for the
> sole use of the intended recipient(s) and may contain confidential
> and privileged information. The recipient is responsible to
> maintain the confidentiality of this information and to use the
> information only for authorized purposes. If you are not the
> intended recipient (or authorized to receive information for the
> intended recipient), you are hereby notified that any review, use,
> disclosure, distribution, copying, printing, or action taken in
> reliance on the contents of this e-mail is strictly prohibited. If
> you have received this communication in error, please notify us
> immediately by reply e-mail and destroy all copies of the original
> message. Thank you.


--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

Re: Unix compress equivalent

hi Olaf,
in unix you have small programms that do one thing and not more.
What you want to archive is a compressed output files.

the most easy way is:

| gzip -c >outfile

depending on your data replace gzip with zoo,lha,bzip2,compress,.....

re,
wh


Olaf Stein schrieb:
> Or even better, can I tell load data infile or somewhere in the table
> definition to compress whatever is written to the file?
>
> Thanks
> Olaf
>
>
> On 5/8/09 12:29 PM, "Olaf Stein" wrote:
>
>> Hi all
>>
>> What is the equivalent in unix (more specifically python) to the compress()
>> function.
>>
>> I am trying to make csv file for use with load data infile and am wondering
>> how to compress the strings that I would usually compress with compress() in
>> a regular sql statement. The field I am writing this into is longblob and I
>> need the compressed version here to be identical to what compress() would do
>>
>> Thanks
>> olaf
>>
>> ----------------------------------------- Confidentiality Notice:
>> The following mail message, including any attachments, is for the
>> sole use of the intended recipient(s) and may contain confidential
>> and privileged information. The recipient is responsible to
>> maintain the confidentiality of this information and to use the
>> information only for authorized purposes. If you are not the
>> intended recipient (or authorized to receive information for the
>> intended recipient), you are hereby notified that any review, use,
>> disclosure, distribution, copying, printing, or action taken in
>> reliance on the contents of this e-mail is strictly prohibited. If
>> you have received this communication in error, please notify us
>> immediately by reply e-mail and destroy all copies of the original
>> message. Thank you.
>
>

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

Re: Unix compress equivalent

Hi Olaf,

If you only need to compress the column *while* loading it from the csv file then use load data infile with a user variable to do the compression at load time,.

mysql> create table t(uncompressed varchar(4000), compressed varbinary(1000));
Query OK, 0 rows affected (0.07 sec)

mysql> \! echo -e 'abcde\nfghijk\n' > /tmp/test.csv

mysql> load data infile '/tmp/test.csv' into table t (@raw) set uncompressed = @raw, compressed = compress(@raw);
Query OK, 3 rows affected (0.00 sec)
Records: 3 Deleted: 0 Skipped: 0 Warnings: 0

mysql> select uncompressed, @a:=hex(compress(uncompressed)), @b:=hex(compressed), @a = @b from t;
+--------------+--------------------------------------+----- ---------------------------------+---------+
| uncompressed | @a:=hex(compress(uncompressed)) | @b:=hex(compressed) | @a = @b |
+--------------+--------------------------------------+----- ---------------------------------+---------+
| abcde | 05000000789C4B4C4A4E49050005C801F0 | 05000000789C4B4C4A4E49050005C801F0 | 1 |
| fghijk | 06000000789C4B4BCFC8CCCA060008870274 | 06000000789C4B4BCFC8CCCA060008870274 | 1 |
| | | | 1 |
+--------------+--------------------------------------+----- ---------------------------------+---------+
3 rows in set (0.04 sec)


Cheers,
-Janek

On Fri, 2009-05-08 at 12:29 -0400, Olaf Stein wrote:
> Hi all
>
> What is the equivalent in unix (more specifically python) to the compress()
> function.
>
> I am trying to make csv file for use with load data infile and am wondering
> how to compress the strings that I would usually compress with compress() in
> a regular sql statement. The field I am writing this into is longblob and I
> need the compressed version here to be identical to what compress() would do
>
> Thanks
> olaf
>
> ----------------------------------------- Confidentiality Notice:
> The following mail message, including any attachments, is for the
> sole use of the intended recipient(s) and may contain confidential
> and privileged information. The recipient is responsible to
> maintain the confidentiality of this information and to use the
> information only for authorized purposes. If you are not the
> intended recipient (or authorized to receive information for the
> intended recipient), you are hereby notified that any review, use,
> disclosure, distribution, copying, printing, or action taken in
> reliance on the contents of this e-mail is strictly prohibited. If
> you have received this communication in error, please notify us
> immediately by reply e-mail and destroy all copies of the original
> message. Thank you.
>


--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

Re: Unix compress equivalent

Thanks
This is exactly what I needed

Olaf


On 5/11/09 7:15 AM, "Janek Bogucki" wrote:

> Hi Olaf,
>
> If you only need to compress the column *while* loading it from the csv file
> then use load data infile with a user variable to do the compression at load
> time,.
>
> mysql> create table t(uncompressed varchar(4000), compressed varbinary(1000));
> Query OK, 0 rows affected (0.07 sec)
>
> mysql> \! echo -e 'abcde\nfghijk\n' > /tmp/test.csv
>
> mysql> load data infile '/tmp/test.csv' into table t (@raw) set uncompressed =
> @raw, compressed = compress(@raw);
> Query OK, 3 rows affected (0.00 sec)
> Records: 3 Deleted: 0 Skipped: 0 Warnings: 0
>
> mysql> select uncompressed, @a:=hex(compress(uncompressed)),
> @b:=hex(compressed), @a = @b from t;
> +--------------+--------------------------------------+----- ------------------
> ---------------+---------+
> | uncompressed | @a:=hex(compress(uncompressed)) | @b:=hex(compressed)
> | @a = @b |
> +--------------+--------------------------------------+----- ------------------
> ---------------+---------+
> | abcde | 05000000789C4B4C4A4E49050005C801F0 |
> 05000000789C4B4C4A4E49050005C801F0 | 1 |
> | fghijk | 06000000789C4B4BCFC8CCCA060008870274 |
> 06000000789C4B4BCFC8CCCA060008870274 | 1 |
> | | |
> | 1 |
> +--------------+--------------------------------------+----- ------------------
> ---------------+---------+
> 3 rows in set (0.04 sec)
>
>
> Cheers,
> -Janek
>
> On Fri, 2009-05-08 at 12:29 -0400, Olaf Stein wrote:
>> Hi all
>>
>> What is the equivalent in unix (more specifically python) to the compress()
>> function.
>>
>> I am trying to make csv file for use with load data infile and am wondering
>> how to compress the strings that I would usually compress with compress() in
>> a regular sql statement. The field I am writing this into is longblob and I
>> need the compressed version here to be identical to what compress() would do
>>
>> Thanks
>> olaf
>>
>> ----------------------------------------- Confidentiality Notice:
>> The following mail message, including any attachments, is for the
>> sole use of the intended recipient(s) and may contain confidential
>> and privileged information. The recipient is responsible to
>> maintain the confidentiality of this information and to use the
>> information only for authorized purposes. If you are not the
>> intended recipient (or authorized to receive information for the
>> intended recipient), you are hereby notified that any review, use,
>> disclosure, distribution, copying, printing, or action taken in
>> reliance on the contents of this e-mail is strictly prohibited. If
>> you have received this communication in error, please notify us
>> immediately by reply e-mail and destroy all copies of the original
>> message. Thank you.
>>
>


--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org