Curl(): maximum length of POST arguments?

I'm trying to query a form using curl, so that I can manipulate the data it
returns before pushing it to the browser.

If I just use a normal POST form it work fine. However if I use curl() then
the script gives an error. One things that springs to mind is the length of
the POST data. Its really long. So long if I try to use file_get_contents()
it complains.

Is there any restriction on how long the CURLOPT_POSTFIELDS can be?

curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode($query));

Re: Curl(): maximum length of POST arguments?

On Sat, 01 Jul 2006 11:21:10 +0100, Bonge Boo! wrote:

>I'm trying to query a form using curl, so that I can manipulate the data it
>returns before pushing it to the browser.
>
>If I just use a normal POST form it work fine. However if I use curl() then
>the script gives an error. One things that springs to mind is the length of
>the POST data. Its really long. So long if I try to use file_get_contents()
>it complains.

What's the error?

>Is there any restriction on how long the CURLOPT_POSTFIELDS can be?
>
>curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode($query));

Shouldn't be in cURL, but you could well be hitting PHP memory limits. How big
is $query?

--
Andy Hassall :: andy@andyh.co.uk :: http://www.andyh.co.uk
http://www.andyhsoftware.co.uk/space :: disk and FTP usage analysis tool

Re: Curl(): maximum length of POST arguments?

On 1/7/06 12:51, in article sboca2l84durjq31cabk3j3l7q7l67ibbo@4ax.com,
"Andy Hassall" wrote:

>> I'm trying to query a form using curl, so that I can manipulate the data it
>> returns before pushing it to the browser.
>>
>> If I just use a normal POST form it work fine. However if I use curl() then
>> the script gives an error. One things that springs to mind is the length of
>> the POST data. Its really long. So long if I try to use file_get_contents()
>> it complains.
>
> What's the error?

I think it might have been a urlencode error on my part.

>> Is there any restriction on how long the CURLOPT_POSTFIELDS can be?
>>
>> curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode($query));
>
> Shouldn't be in cURL, but you could well be hitting PHP memory limits. How
> big
> is $query?

Not that huge. About 400 characters.

A copy of a GET string below

http://www.blahbh.com/zippo/Search.ASP?WCI=AddToPortFolio%26 PAGE%3D1%26DEL%3
D%26PropertyID%3D00595876%26GTYP%3D0%26ID%3D0009185%26PER%3D 10%26XSLT%3D-1%2
6GID%3D580%26SQL%3D2+AND+Property.EstateAgentID+%3D+580+AND+ %28Property.Inst
ructionLevel+%3C+2%29+AND+%28%28Property.Price+%2A+CurrencyC onversion.Value%
29+%3C%3D+99000000+AND+%28Property.Price+%2A+CurrencyConvers ion.Value%29+%3E
%3D+0%29%29+AND+Property.Price+%3C%3D+300000

I think it must be a misunderstanding on my part about using curl(). If you
have a moment, and don't mind, could you take a look through this and look
for any screamers?

$query .= "&PAGE=1";
$query .= "&DEL=";
$query .= "&PropertyID=00595876";
$query .= ">YP=0";
$query .= "&ID=0000000";
$query .= "&PER=10";
$query .= "&XSLT=-1";
$query .= "&GID=400";
$query .= "&SQL=2 AND Property.EstateAgentID = 580 AND
(Property.InstructionLevel < 2) AND ((Property.Price *
CurrencyConversion.Value) <= 99000000 AND (Property.Price *
CurrencyConversion.Value) >= 0)) AND Property.Price <= 300000";

// create a new curl resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_REFERER, "http://www.blahbl.com/
zippo/Search.ASP?WCI=AddToPortFolio");
curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode($dezrez_query));
// return curl as a string

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$login_contents = curl_exec($ch);
print $login_contents;
// close curl resource
curl_close($ch);

Re: Curl(): maximum length of POST arguments?

Bonge Boo! wrote:

> Is there any restriction on how long the CURLOPT_POSTFIELDS can be?

The length of the POST depends a bit on your browser, some versions of MSIE
don't allow large POST, limiting the length to something like 1-2kb.

Re: Curl(): maximum length of POST arguments?

Bonge Boo! wrote:
> $query .= "&PAGE=1";
> $query .= "&DEL=";
> $query .= "&PropertyID=00595876";
>

The problem might be that you are urlencoding the entire query string,
including the seperators; try encoding the parameter values only:

$query .= '&FOO=' . urlencode($value);


JW

Re: Curl(): maximum length of POST arguments?

On Sat, 01 Jul 2006 14:53:46 +0200, "J.O. Aho" wrote:

>Bonge Boo! wrote:
>
>> Is there any restriction on how long the CURLOPT_POSTFIELDS can be?
>
>The length of the POST depends a bit on your browser, some versions of MSIE
>don't allow large POST, limiting the length to something like 1-2kb.

That's GET, surely.

POST limited to 2kb would be a bit useless, particularly since it's used for
file uploads.

--
Andy Hassall :: andy@andyh.co.uk :: http://www.andyh.co.uk
http://www.andyhsoftware.co.uk/space :: disk and FTP usage analysis tool

Re: Curl(): maximum length of POST arguments?

On Sat, 01 Jul 2006 13:41:48 +0100, Bonge Boo! wrote:

>On 1/7/06 12:51, in article sboca2l84durjq31cabk3j3l7q7l67ibbo@4ax.com,
>"Andy Hassall" wrote:
>
>>> I'm trying to query a form using curl, so that I can manipulate the data it
>>> returns before pushing it to the browser.
>>>
>>> If I just use a normal POST form it work fine. However if I use curl() then
>>> the script gives an error. One things that springs to mind is the length of
>>> the POST data. Its really long. So long if I try to use file_get_contents()
>>> it complains.
>>
>> What's the error?
>
>I think it might have been a urlencode error on my part.

What's the error? As in the error message?

>>> Is there any restriction on how long the CURLOPT_POSTFIELDS can be?
>>>
>>> curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode($query));
>>
>> Shouldn't be in cURL, but you could well be hitting PHP memory limits. How
>> big
>> is $query?
>
>Not that huge. About 400 characters.
>
>A copy of a GET string below
>
>http://www.blahbh.com/zippo/Search.ASP?WCI=AddToPortFolio%2 6PAGE%3D1%26DEL%3
>D%26PropertyID%3D00595876%26GTYP%3D0%26ID%3D0009185%26PER%3 D10%26XSLT%3D-1%2
>6GID%3D580%26SQL%3D2+AND+Property.EstateAgentID+%3D+580+AND +%28Property.Inst
>ructionLevel+%3C+2%29+AND+%28%28Property.Price+%2A+Currency Conversion.Value%
>29+%3C%3D+99000000+AND+%28Property.Price+%2A+CurrencyConver sion.Value%29+%3E
>%3D+0%29%29+AND+Property.Price+%3C%3D+300000

Are you sure the server accepts POST for the page?

>I think it must be a misunderstanding on my part about using curl(). If you
>have a moment, and don't mind, could you take a look through this and look
>for any screamers?
>
>$query .= "&PAGE=1";
> $query .= "&DEL=";
> $query .= "&PropertyID=00595876";
> $query .= ">YP=0";
> $query .= "&ID=0000000";
> $query .= "&PER=10";
> $query .= "&XSLT=-1";
> $query .= "&GID=400";
> $query .= "&SQL=2 AND Property.EstateAgentID = 580 AND
>(Property.InstructionLevel < 2) AND ((Property.Price *
>CurrencyConversion.Value) <= 99000000 AND (Property.Price *
>CurrencyConversion.Value) >= 0)) AND Property.Price <= 300000";
>
>// create a new curl resource
> $ch = curl_init();
> // set URL and other appropriate options
> curl_setopt($ch, CURLOPT_URL, $url);
> curl_setopt($ch, CURLOPT_HEADER, 0);
> curl_setopt($ch, CURLOPT_POST, 1);
>curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
> curl_setopt($ch, CURLOPT_REFERER, "http://www.blahbl.com/
>zippo/Search.ASP?WCI=AddToPortFolio");
>curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode($dezrez_query));

You've just encoded all the "&" and "=" separators; you should be encoding the
keys and values separately and joining them up with "&" and "=".

--
Andy Hassall :: andy@andyh.co.uk :: http://www.andyh.co.uk
http://www.andyhsoftware.co.uk/space :: disk and FTP usage analysis tool

Re: Curl(): maximum length of POST arguments?

Andy Hassall wrote:
> On Sat, 01 Jul 2006 14:53:46 +0200, "J.O. Aho" wrote:
>
>> Bonge Boo! wrote:
>>
>>> Is there any restriction on how long the CURLOPT_POSTFIELDS can be?
>> The length of the POST depends a bit on your browser, some versions of MSIE
>> don't allow large POST, limiting the length to something like 1-2kb.
>
> That's GET, surely.
>
> POST limited to 2kb would be a bit useless, particularly since it's used for
> file uploads.
>

Yeah, you are right

http://support.microsoft.com/default.aspx?scid=kb;EN-US;q208 427



//Aho

Re: Curl(): maximum length of POST arguments?

On 1/7/06 13:55, in article 44a670bd$0$12275$dbd49001@news.euronet.nl,
"Janwillem Borleffs" wrote:

>> $query .= "&PAGE=1";
>> $query .= "&DEL=";
>> $query .= "&PropertyID=00595876";
>>
>
> The problem might be that you are urlencoding the entire query string,
> including the seperators; try encoding the parameter values only:
>
> $query .= '&FOO=' . urlencode($value);

Ok. I've now correctly encode my url string. If I cut and paste the complete
generated string into a Firefox browser, then goto that page, the expected
result is returned. Some info is entered into a database and an xml string
returned.

If I use Safrai or IE with that same string , the xml comes back, but the
item isn't entered into the database. If I use file_get_contents() with that
same string, again, it fails to work in the same way.

Hum. Rather stuck with this. Unfortunately I'm trying to hack someone else's
system to work how I want it to, so I don't know what is happening at their
end.

Can't get curl() give me any useful feedback on why it failing to give the
expected result.

Darn.

Re: Curl(): maximum length of POST arguments?

On 1/7/06 14:06, in article ggsca2t750pabo4q693ea91jec7jljojft@4ax.com,
"Andy Hassall" wrote:

>> http://www.blahbh.com/zippo/Search.ASP?WCI=AddToPortFolio%26 PAGE%3D1%26DEL%3
>> D%26PropertyID%3D00595876%26GTYP%3D0%26ID%3D0009185%26PER%3D 10%26XSLT%3D-1%2
>> 6GID%3D580%26SQL%3D2+AND+Property.EstateAgentID+%3D+580+AND+ %28Property.Inst
>> ructionLevel+%3C+2%29+AND+%28%28Property.Price+%2A+CurrencyC onversion.Value%
>> 29+%3C%3D+99000000+AND+%28Property.Price+%2A+CurrencyConvers ion.Value%29+%3E
>> %3D+0%29%29+AND+Property.Price+%3C%3D+300000
>
> Are you sure the server accepts POST for the page?

Absolutely. All the sample pages I have been examining, which work, use
POST.

>> I think it must be a misunderstanding on my part about using curl(). If you
>> have a moment, and don't mind, could you take a look through this and look
>> for any screamers?
>>
>> $query .= "&PAGE=1";
>> $query .= "&DEL=";
>> $query .= "&PropertyID=00595876";
>> $query .= ">YP=0";
>> $query .= "&ID=0000000";
>> $query .= "&PER=10";
>> $query .= "&XSLT=-1";
>> $query .= "&GID=400";
>> $query .= "&SQL=2 AND Property.EstateAgentID = 580 AND
>> (Property.InstructionLevel < 2) AND ((Property.Price *
>> CurrencyConversion.Value) <= 99000000 AND (Property.Price *
>> CurrencyConversion.Value) >= 0)) AND Property.Price <= 300000";
>>
>> // create a new curl resource
>> $ch = curl_init();
>> // set URL and other appropriate options
>> curl_setopt($ch, CURLOPT_URL, $url);
>> curl_setopt($ch, CURLOPT_HEADER, 0);
>> curl_setopt($ch, CURLOPT_POST, 1);
>> curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
>> curl_setopt($ch, CURLOPT_REFERER, "http://www.blahbl.com/
>> zippo/Search.ASP?WCI=AddToPortFolio");
>> curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode($dezrez_query));
>
> You've just encoded all the "&" and "=" separators; you should be encoding
> the
> keys and values separately and joining them up with "&" and "=".

Done that. Still fails. I get returned a 1, which I assume indicates curl
succeeded, but nothing else returned to the $string I pass the result of the
curl to.

Re: Curl(): maximum length of POST arguments?

On Sun, 02 Jul 2006 16:35:51 +0100, Bonge Boo! wrote:

>>> // create a new curl resource
>>> $ch = curl_init();
>>> // set URL and other appropriate options
>>> curl_setopt($ch, CURLOPT_URL, $url);
>>> curl_setopt($ch, CURLOPT_HEADER, 0);
>>> curl_setopt($ch, CURLOPT_POST, 1);
>>> curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
>>> curl_setopt($ch, CURLOPT_REFERER, "http://www.blahbl.com/
>>> zippo/Search.ASP?WCI=AddToPortFolio");
>>> curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode($dezrez_query));
>>
>> You've just encoded all the "&" and "=" separators; you should be encoding
>> the
>> keys and values separately and joining them up with "&" and "=".
>
>Done that. Still fails. I get returned a 1, which I assume indicates curl
>succeeded,

That depends on the options.

> but nothing else returned to the $string I pass the result of the
>curl to.

With the settings above, cURL would just print the results to the browser, not
return any content. See CURLOPT_RETURNTRANSFER.

If you want more information, ask for it using curl_getinfo(), curl_error(),
and CURLOPT_VERBOSE.

--
Andy Hassall :: andy@andyh.co.uk :: http://www.andyh.co.uk
http://www.andyhsoftware.co.uk/space :: disk and FTP usage analysis tool

Re: Curl(): maximum length of POST arguments?

On 2/7/06 16:52, in article 9kqfa250r72epe0990gl6qklv9heu644fa@4ax.com,
"Andy Hassall" wrote:

>
>>>> // create a new curl resource
>>>> $ch = curl_init();
>>>> // set URL and other appropriate options
>>>> curl_setopt($ch, CURLOPT_URL, $url);
>>>> curl_setopt($ch, CURLOPT_HEADER, 0);
>>>> curl_setopt($ch, CURLOPT_POST, 1);
>>>> curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
>>>> curl_setopt($ch, CURLOPT_REFERER, "http://www.blahbl.com/
>>>> zippo/Search.ASP?WCI=AddToPortFolio");
>>>> curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode($dezrez_query));
>>>
>>> You've just encoded all the "&" and "=" separators; you should be encoding
>>> the
>>> keys and values separately and joining them up with "&" and "=".
>>
>> Done that. Still fails. I get returned a 1, which I assume indicates curl
>> succeeded,
>
> That depends on the options.
>
>> but nothing else returned to the $string I pass the result of the
>> curl to.
>
> With the settings above, cURL would just print the results to the browser,
> not
> return any content. See CURLOPT_RETURNTRANSFER.
>
> If you want more information, ask for it using curl_getinfo(), curl_error(),
> and CURLOPT_VERBOSE.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);

curl_setopt($ch, CURLOPT_REFERER, "http://www.blah.com/bibble/Search.ASP");
curl_setopt($ch, CURLOPT_POSTFIELDS, $query);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$login_contents = curl_exec($ch);
$error = curl_error($ch);
print $error;
print $login_contents;
curl_close($ch);

Pretty sure there is no curl error. Trying variations of this doesn't give
me any feedback apart from that 1.

Capturing the data sent back and forth using curl and also the normal form
posts, it looks like some cookie() are being sent as well during the form
posts, which might explain why its not working in some browsers and isn't in
others. Ithought I had it pinned down to a id cookie for the domain, but
apparantly not.

I think I might as well give up on this before I it becomes an even bigger
time-sucker.

Thanks to all for your brains