curl / Mailing Lists / curl-users / Single Mail


curl/python -- stderr

From: bruce <>
Date: Thu, 30 Mar 2017 14:03:03 -0400


I know this is a bit off topic. It's python + curl...!

Trying to understand the "correct" way to run a sys command ("curl")
and to get the potential stderr. Checking Stackoverflow (SO), implies
that I should be able to use a raw/text cmd, with "shell=true".

If I leave the stderr out, and just use
the test works...

Any pointers on what I might inspect to figure out why this hangs on
the proc.communicate process/line??

I'm showing a very small chunk of the test, but its the relevant piece.



  cmd='[r" curl -sS '
  #cmd=cmd+'-A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0)
Gecko/20100101 Firefox/38.0"'
  cmd=cmd+"-A '"+user_agent+"'"
  ##cmd=cmd+' --cookie-jar '+cname+' --cookie '+cname+' '
  cmd=cmd+' --cookie-jar '+ff+' --cookie '+ff+' '
  #cmd=cmd+'-e "'+referer+'" -d "'+tt+'" '
  #cmd=cmd+'-e "'+referer+'" '
  cmd=cmd+"-L '"+url1+"'"+'"]'
  #cmd=cmd+'-L "'+xx+'" '




the cmd is generated to be:
cmd=[r" curl -sS -A 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT
6.1; Trident/5.0; yie8)' --cookie-jar
--cookie /crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c88e.lwp
   -L ''"]

test code hangs, ctrl-C generates the following:
^CTraceback (most recent call last):
  File "/crawl_tmp/", line 3363, in <module>
  File "/crawl_tmp/", line 978, in
  File "/usr/lib64/python2.6/", line 732, in communicate
    stdout, stderr = self._communicate(input, endtime)
  File "/usr/lib64/python2.6/", line 1328, in _communicate
    stdout, stderr = self._communicate_with_poll(input, endtime)
  File "/usr/lib64/python2.6/", line 1400, in
    ready = poller.poll(self._remaining_time(endtime))

This works from the cmdline:
curl -sS -A 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1;
Trident/5.0; yie8)' --cookie-jar
--cookie /crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c88e.lwp
   -L ''
Received on 2017-03-30