cURL / Mailing Lists / curl-library / Single Mail

curl-library

Connection times out after first curl_easy_perform call

From: John Vorwald <john_vorwald_at_msn.com>
Date: Wed, 13 Dec 2006 03:51:16 -0500

I am trying to review data on a financial site.
I have two question. Is the procedure that I'm using secure? And second,
how do I remain connected after the first curl_easy_perform call? Do I need
to output more information to resolve this?

I developed curl command line statements by review the output of "Live HTTP
headers" while using the site with Mozilla. Two commands are required. The
first command logs on and obtains the url of the balance site. The second
command obtains the data from the balance site.

The first command is

curl --data-binary
"VTI-GROUP=0&partId=USER&password=PASS&acrover=7&fileDigit=A&Enter=Login" \
     --user-agent "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1)
Gecko/20061010 Firefox/2.0" \
     --referer https://tspweb2.tspsec.tsp.gov/login_at.htm \
     --output step4b.out \
     --trace-ascii step4b.trace \
     --dump-header step4b.hdr \
     
"https://tspweb2.tspsec.tsp.gov/NASApp/tsp/authenticate.do?_name=nfclogin&SessionKey$=3ABC123"

From the output of the first command, extract the link to the balance site
(remove phrase "amp;"
The second command, with the link altered, is

curl --user-agent "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1)
Gecko/20061010 Firefox/2.0" \
     --referer
"https://tspweb2.tspsec.tsp.gov/NASApp/tsp/authenticate.do?_name=nfclogin&SessionKey$=2ABC123"
  \
     --output step5b.out \
     --trace-ascii step5b.trace \
     --dump-header step5b.hdr \
     
"https://tspweb2.tspsec.tsp.gov/NASApp/tsp/accountBalance.do?GXHC_GX_jst=5aa774646d6164&GXHC_JSESSIONID=64915053759914652&subaction=view&_name=acctbal&SessionKey$=3PTyXRUqwoVWQtwQYsXMRw&Acct$=CIV"

The program to implement these two commands is below. The output of the
program is

BEFORE SECOND CALL TO CURL

* Resolving host timed out: accountBalance.do
* Closing connection #0

AFTER SECOND CALL TO CURL

Net Date Line: Date: Wed, 13 Dec 2006 08:45:21 GMT

EST: 13-12-2006 3:45:21
Press any key to continue . . .

BEFORE FIRST CALL TO CURL

* About to connect() to tspweb2.tspsec.tsp.gov port 443
* Trying 38.118.64.250... * connected
* Connected to tspweb2.tspsec.tsp.gov (38.118.64.250) port 443
* successfully set certificate verify locations:
* CAfile: cacert.pem
  CApath: C:\Documents and Settings\John\My Documents\retirement\cpp\rev12\
* SSL connection using AES256-SHA
* Server certificate:
* subject: /C=US/ST=District of Columbia/L=District of
Columbia/O=Federal
Retirement Thrift Investment Board/OU=Thrift Savings Plan Account
Access/OU=Ter
ms of use at www.verisign.com/rpa (c)05/CN=tspweb2.tspsec.tsp.gov
* start date: 2006-07-19 00:00:00 GMT
* expire date: 2007-08-16 23:59:59 GMT
* common name: tspweb2.tspsec.tsp.gov (matched)
* issuer: /O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign
Interna
tional Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY
LTD.
(c)97 VeriSign
* SSL certificate verify ok.
>POST /NASApp/tsp/authenticate.do?_name=nfclogin&SessionKey$=3ABC123
>HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1)
Gecko/2006
1010 Firefox/2.0
Host: tspweb2.tspsec.tsp.gov
Accept: */*
Referer: https://tspweb2.tspsec.tsp.gov/login_at.htm
Content-Length: 76
Content-Type: application/x-www-form-urlencoded

VTI-GROUP=0&partId=USER&password=PASS&acrover=7&fileDigit=A&Enter=Login* Cu
rl_xxx_rcvs returned -1, block = TRUE
< HTTP/1.1 200 200 OK
HTTP/1.1 200 200 OK

< Server: Netscape-Enterprise/4.1
Server: Netscape-Enterprise/4.1

< Date: Wed, 13 Dec 2006 08:45:21 GMT
Date: Wed, 13 Dec 2006 08:45:21 GMT

< Content-type: text/html;charset=8859_1
Content-type: text/html;charset=8859_1

< Connection: close
Connection: close

* Closing connection #0

AFTER FIRST CALL TO CURL

BEFORE SECOND CALL TO CURL

* Resolving host timed out: accountBalance.do
* Closing connection #0

AFTER SECOND CALL TO CURL

The code snippet is

std::string account_link;

size_t BalanceHistory_CURL_WriteOutput(void *ptr, size_t size, size_t nmemb,
                                 void *stream)
{

  /* Write to file */
  fwrite(ptr, size, nmemb, (FILE *)stream);

  /* Extract the account link */
  std::string str( (const char*) ptr, nmemb);
  const size_t start_sub(str.find("accountBalance.do",0));
  if (start_sub != std::string::npos)
  {
     const size_t end_sub(str.find('"',start_sub));
     account_link = str.substr(start_sub, (end_sub-start_sub));

     fflush((FILE *)stream );
     fprintf((FILE *)stream,"\n\nFOUND ACCOUNT
LINK:\n%s",account_link.c_str());

     size_t s2;
     while ((s2=account_link.find("amp;",0))!=std::string::npos)
       account_link.erase(s2,4);

     fprintf((FILE *)stream,"\nMODIFIED ACCOUNT LINK TO
BE:\n%s\n\n",account_link.c_str());
  }

  return(nmemb*size);
}

size_t BalanceHistory_CURL_WriteHeader(void *ptr, size_t size, size_t nmemb,
                                 void *stream)
{
  std::cout << ((char *)ptr) << std::endl;
  return(nmemb*size);
}

CURLcode TSP_BALANCE_HISTORY::BalanceHistory_CURL_Fetch(CURL *curl, const
PASSWORD &password,
        const char *OutFileName=NULL)
{

//curl --data-binary
"VTI-GROUP=0&partId=SSN&password=_password_&acrover=7&fileDigit=A&Enter=Login"
\
// --user-agent "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.1) Gecko/20061010 Firefox/2.0" \
// --referer https://tspweb2.tspsec.tsp.gov/login_at.htm \
//
"https://tspweb2.tspsec.tsp.gov/NASApp/tsp/authenticate.do?_name=nfclogin&SessionKey$=3ABC123"

  /* Eumulate Mozila */
  curl_easy_setopt(curl, CURLOPT_USERAGENT,
                   "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.1) Gecko/20061010 Firefox/2.0");

  curl_easy_setopt(curl, CURLOPT_HEADER, 1); // Header in output (1-yes,
0-no)

  // SET SSN AND PIN

  // curl_easy_setopt(curl, CURLOPT_POSTFIELDS,
  //
"VTI-GROUP=0&partId=SSN&password=PIN&acrover=7&fileDigit=A&Enter=Login");

  std::string postStr0=((std::string)"VTI-GROUP=0&partId=")
                       +password.get_user_string()
                       +((std::string)"&password=")
                       +password.get_password_string()
                       +((std::string)"&acrover=7&fileDigit=A&Enter=Login");

        char * postStr = new char[postStr0.length()+1];
        sprintf(postStr,"%s",postStr0.c_str());

  curl_easy_setopt(curl, CURLOPT_POST, 1);
        curl_easy_setopt(curl,CURLOPT_POSTFIELDS,postStr);

  // Set referer
  const char *referer("https://tspweb2.tspsec.tsp.gov/login_at.htm");
  curl_easy_setopt(curl, CURLOPT_REFERER, referer);

  // Set SSL Option and certificate file
  curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1);
  curl_easy_setopt(curl, CURLOPT_CAPATH, "C:\\Documents and
Settings\\John\\My Documents\\retirement\\cpp\\rev12\\");
  curl_easy_setopt(curl, CURLOPT_CAINFO, "cacert.pem");

  // Set Header and Output functions
  curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
*BalanceHistory_CURL_WriteOutput);
  curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION,
*BalanceHistory_CURL_WriteHeader);

  FILE *outfile;
  CURLcode res;

  if (OutFileName == NULL || strlen(OutFileName) < 2)
    curl_easy_setopt(curl, CURLOPT_NOBODY, 1);
  else
  {
    outfile = fopen(OutFileName, "wb");
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, outfile);
  }

  const char
*url1("https://tspweb2.tspsec.tsp.gov/NASApp/tsp/authenticate.do?_name=nfclogin&SessionKey$=3ABC123");
  curl_easy_setopt(curl, CURLOPT_URL, url1);
  curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
  std::cout << std::endl << std::endl << "BEFORE FIRST CALL TO CURL " <<
std::endl << std::endl;
  res = curl_easy_perform(curl);
  std::cout << std::endl << std::endl << "AFTER FIRST CALL TO CURL " <<
std::endl << std::endl;

  if (res == CURLE_OK)
  {
     // Second call using new refere and balance url
    const char *url2(account_link.c_str());
          // url2 = new char[account_link.length()+1];
          // sprintf(url2,"%s",account_link.c_str());
    curl_easy_setopt(curl, CURLOPT_REFERER, url1);
    curl_easy_setopt(curl, CURLOPT_URL, url2);
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
    std::cout << std::endl << std::endl << "BEFORE SECOND CALL TO CURL " <<
std::endl << std::endl;
    res = curl_easy_perform(curl);
    std::cout << std::endl << std::endl << "AFTER SECOND CALL TO CURL " <<
std::endl << std::endl;
  }

  if (outfile != NULL)
    fclose(outfile);

        delete [] postStr;

  // std::cout << std::endl << curl_version( ) << std::endl;
  return res; /* (CURLE_OK) */
}

void TSP_BALANCE_HISTORY::get_history(const PASSWORD &password)
{
  CURL *curl;

  /* Init CURL before usage */
  curl_global_init(CURL_GLOBAL_ALL);
  curl = curl_easy_init();
  if (curl) {
    BalanceHistory_CURL_Fetch(curl, password, "tsp_login.txt");
    curl_easy_cleanup(curl);
  }
  return;
}
Received on 2006-12-13