curl-and-php
Re: post then get help
Date: Tue, 8 Jul 2008 15:54:50 -0500
it seems there isn't a 301 or 302. here is the actual flow from firefox
http live headers (i omitted the header and other junk)
http://<url>/doctypesearch.aspxPOST /doctypesearch.aspx HTTP/1.1
Host: <url>
...some stuff
Referer: http://<url>/doctypesearch.aspx
Cookie: ASP.NET_SessionId=rdt0fp2ippebau55gwta2y55
Content-Type: application/x-www-form-urlencodedContent-Length: 164
__VIEWSTATE=dDwyMDY1NDQwMTMxOztsPGNoa1VudmVyaWZpZWQ7Pj5APoilxKEX3nDtx3%2Bj9TB4hNF%2Flw%3D%3D&txtStart=7%2F7%2F2008&txtEnd=7%2F7%2F2008&btnSearchDoc=Search&docTypes=
HTTP/1.x 200 OK
...some stuff
----------------------------------------------------------
http://<url>/results.aspxGET /results.aspx HTTP/1.1
Host: <url>
...some stuff
Referer: http://<url>/doctypesearch.aspx
Cookie: ASP.NET_SessionId=rdt0fp2ippebau55gwta2y55HTTP/1.x 200 OK
...some stuff
----------------------------------------------------------
http://<url>/css/calibrate_page1_files/sub_topstrip.jpgGET
/css/calibrate_page1_files/sub_topstrip.jpg HTTP/1.1
Host: <url>
...some stuffHTTP/1.x 404 Not Found
...some stuff
----------------------------------------------------------
http://<url>/treeView.aspx?initCriteria=GET /treeView.aspx?initCriteria=
HTTP/1.1
Host: <url>
...some stuff
HTTP/1.x 200 OK
...some stuff
----------------------------------------------------------
http://<url>/ig_common/20043/scripts/ig_WebGrid_srt.jsGET
/ig_common/20043/scripts/ig_WebGrid_srt.js HTTP/1.1
Host: <url>
...some stuffHTTP/1.x 200 OK
...some stuff
---------------------------------------------------------- i don't know
how to handle the above. a 200ok is received instead of 301 or 301 does
that mean i should just grab the header? and remove the if brackets?
----- Original Message -----
From: "Stephen Pynenburg"
To: "curl with PHP"
Subject: Re: post then get help
Date: Tue, 8 Jul 2008 16:25:35 -0400
Ya, my mistake - I left the return variable the same as the input
variable. In this section:
if ($http_code == 301 || $http_code == 302)
{
echo "code is 301 or 302";
$matches = array();
preg_match('/Location:(.*?)\n/', $header, $matches);
$url = @parse_url(trim(array_pop($matches)));
}
return $url;
Change the variable $url to something else, like $url2, etc.
Also, it's possible that
if ($http_code == 301 || $http_code == 302)
Isn't evaluating properly, so you might just want to take out that
whole conditional block.
-Stephen
2008/7/8 ryan pal <ryanpal_at_mail.com>:
i see what you are referring to now. when testing your code it
returns me the original url that i send the curl_POST_redirect
function, not the redirect one.
----- Original Message -----
From: "Stephen Pynenburg"
To: "curl with PHP"
Subject: Re: post then get help
Date: Mon, 7 Jul 2008 22:28:29 -0400
I was thinking - do the POST request, get the first returned
redirection URL, then use that URL for standard (GET)
requests, using the full curl_redir_exec function:
<?PHP
//Use this function to POST the page and parse the redirect
location header out of the response
function curl_POST_grab_redirect($url,$postfields){
$ch = curl_init(); //initialize curl handle
curl_setopt($ch, CURLOPT_URL, $url); //set url to post to
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, "/tmp");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //return into
variable
curl_setopt($ch, CURLOPT_TIMEOUT, 24); //timeout after 25s
curl_setopt($ch, CURLOPT_POST, 1); //set POST method
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields); //add post
fields
curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
//curl_setopt($ch, CURLINFO_HEADER_OUT, true);
$result = curl_exec($ch); //run the whole process
list($header, $data) = explode("\n\n", $data, 2);
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
//echo "HTTP CODE: $http_code<br>";
if ($http_code == 301 || $http_code == 302)
{
echo "code is 301 or 302";
$matches = array();
preg_match('/Location:(.*?)\n/', $header, $matches);
$url = @parse_url(trim(array_pop($matches)));
}
return $url;
curl_close($ch);
unset($ch); //clean up
}
//POST and get the url
$theurl = curl_POST_grab_redirect("http://yoursite.com/page.php","key=value&so=on&so=forth");
//proceed with curl_redir_exec
$ch2 = curl_init();
curl_setopt($ch2, CURLOPT_GET, 1); //this is set by default
anyway
curl_setopt($ch2, CURLOPT_URL, $theurl); //from the function
/* Add any other options as needed */
$res = curl_redir_exec($ch2);
?>
2008/7/7 ryan pal <ryanpal_at_mail.com>:
In order to first post I must tell curl to use POST,
which I have done with:
curl_setopt(
$ch, CURLOPT_POSTFIELDS, $postfields);
Since I need to do a GET after this, don't I have to do a
CURLPT_HTTPGET after or else it will still try to post?
I believe that's the issue I'm having from the
beginning. How to do a POST then a get. I'm under the
impression that I have to write code to switch between
the two. I'm also a little confused of I should code
what you describe.
Here is my main function along with the one you
describe. How would I modify it to do GET?
list($header, $data) = explode("\n\n", $data, 2);
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
//echo "HTTP CODE: $http_code<br>";
if ($http_code == 301 || $http_code == 302)
{
curl_setopt(
$ch, CURLOPT_POST, 0); //turn off POST method
curl_setopt(
$ch, CURLOPT_HTTPGET, 1); //set GET method
//curl_setopt($ch, CURLOPT_DEBUGFUNCTION,1);
curl_setopt(
$ch, CURLOPT_POSTFIELDS, NULL);
echo "code is 301 or 302";
$matches = array();
preg_match(
'/Location:(.*?)\n/', $header, $matches);
$url = @parse_url(trim(array_pop($matches)));
}
function
curl_navigate($url,$postfields){
$ch = curl_init(); //initialize curl handle
curl_setopt(
$ch, CURLOPT_URL,$url); //set url to post to
curl_setopt(
$ch, CURLOPT_VERBOSE, 1);
curl_setopt(
$ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, "/tmp");
curl_setopt(
$ch, CURLOPT_RETURNTRANSFER, 1); //return into variable
curl_setopt(
$ch, CURLOPT_TIMEOUT, 24); //timeout after 25s
curl_setopt(
$ch, CURLOPT_POST, 1); //set POST method
curl_setopt(
$ch, CURLOPT_POSTFIELDS, $postfields); //add post fields
curl_setopt(
$ch, CURLOPT_AUTOREFERER, 1);
//curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_redir_exec(
$ch);
$result = curl_exec($ch); //run the whole process
return $result;
curl_close(
$ch);
}
----- Original Message -----
From: "Stephen Pynenburg"
To: "curl with PHP"
Subject: Re: post then get help
Date: Mon, 7 Jul 2008 21:43:07 -0400
Hey,
After the first POST, you only need that little
snippet from the function to get the new Location
header.
Once you've got the new URL from your special POST
request, you can actually just use the full function
for the rest of your requests.
About using GET, unless you explicitly tell cURL to
use POST or PUT or something, it defaults to a GET
request, so you shouldn't have to worry.
-Stephen
2008/7/7 ryan pal <ryanpal_at_mail.com>:
stephen, i see you displayed only a little bit of
the redirect function. are you suggesting i omit
the rest? also, how do you do a get request? use
the CURLOPT_HTTPGET? oh and one more thing.
should this be a seperate function since you
mention i should do the POST as normal? thanks,ryan
----- Original Message -----
From: "Stephen Pynenburg"
To: "curl with PHP"
Subject: Re: post then get help
Date: Mon, 7 Jul 2008 18:39:08 -0400
My recommendation would be to do your first
POST request normally, then take the data you
get and use a little bit of the redir_exec
function:
list($header, $data) = explode("\n\n", $data,
2);
$http_code = curl_getinfo($ch,
CURLINFO_HTTP_CODE);
if ($http_code == 301 || $http_code == 302) {
$matches = array();
preg_match('/Location:(.*?)\n/ ', $header,
$matches);
$url = @parse_url(trim(array_pop($matches)));
To get the next URL, then just do a while
loop, with a cURL GET request in it, until
there are no more URLs.
-Stephen
2008/7/7 ryan pal <ryanpal_at_mail.com>:
i'm trying to figure out how i can do a
"post" then a "get". due to restrictions
on my server, i have to use the redirect
script found on php.net: function
curl_redir_exec($ch,$debug="")
{
static $curl_loops = 0;
static $curl_max_loops = 20; if
($curl_loops++ >=
$curl_max_loops)
{
$curl_loops = 0;
return FALSE;
}
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,
true);
$data = curl_exec($ch);
list($header, $data) = explode("\n\n",
$data, 2);
$http_code = curl_getinfo($ch,
CURLINFO_HTTP_CODE);
if ($http_code == 301 || $http_code ==
302) {
$matches = array();
preg_match('/Location:(.*?)\n/', $header,
$matches);
$url =
@parse_url(trim(array_pop($matches)));
//print_r($url);
if (!$url)
{
//couldn't process the url to redirect to
$curl_loops = 0;
return $data;
}
$last_url = parse_url(curl_getinfo($ch,
CURLINFO_EFFECTIVE_URL));
/* if (!$url['scheme'])
$url['scheme'] = $last_url['scheme'];
if (!$url['host'])
$url['host'] = $last_url['host'];
if (!$url['path'])
$url['path'] = $last_url['path'];*/
$new_url = $url['scheme'] . '://' .
$url['host'] .
$url['path'] .
($url['query']?'?'.$url['query']:'');
curl_setopt($ch, CURLOPT_URL, $new_url);
// debug('Redirecting to', $new_url);
return
curl_redir_exec($ch);
} else {
$curl_loops=0;
return $data;
} i have my curl_setopt to POST, but need
to do GET also. how do setup my code so
that i post and then do a get for the
next few redirects?
-- Be Yourself @ mail.com!
Choose From 200+ Email Addresses
Get a Free Account at www.mail.com!
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
-- Be Yourself @ mail.com!
Choose From 200+ Email Addresses
Get a Free Account at www.mail.com!
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
-- Be Yourself @ mail.com!
Choose From 200+ Email Addresses
Get a Free Account at www.mail.com!
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
-- Be Yourself @ mail.com!
Choose From 200+ Email Addresses
Get a Free Account at www.mail.com!
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
-- Be Yourself @ mail.com! Choose From 200+ Email Addresses Get a Free Account at www.mail.com
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2008-07-08