Menu

#496 Can not upload a zip file in OpenVMS

closed-later
ftp (93)
6
2014-08-17
2005-03-03
Anonymous
No

Hi there,

I have tried to upload a zip file with the following
command and always received a 'Uploaded unaligned
file size' error and the zip file is just partially uploaded to
ther server

curl -v -u user:password --upload-file myfile.zip
ftp://myftpserver

I tried to use the same option to upload a binary file
(e.g. wordpad.exe) or a text file, it all worked fine.

The Curl I am using was download from the link "VMS
Alpha 7.13.0 binary Marty Kuhrt 1777 KB " and I am
using OpenVMS here.

I also tested the windows version of Curl and the zip file
can be uploaded without any problems.

By the way, if I add the -crlf option, the zip file then can
be uploaded without the error, but the file size is
increased by a coulpe of bytes and I can't use winzip to
open it as it said the file is not in the right format.

Any ideas why this would happen?

Thanks

Discussion

  • Daniel Stenberg

    Daniel Stenberg - 2005-03-04
    • priority: 5 --> 6
     
  • Daniel Stenberg

    Daniel Stenberg - 2005-03-04

    Logged In: YES
    user_id=1110

    Uh, in what way is a zip file _not_ a binary file?

     
  • Nobody/Anonymous

    Logged In: NO

    I can tell It was using Binary transfer mode during the
    transmission as I can see the 'Type Set to I ' command on
    the screen.

    It seens to me that this issue is to do with the LF character.
    If I force to use -crlf option, the file can be transferred but with
    the size being increased. But if you don't specify this option,
    curl seems to drop the LF character or do some funny thing
    to it.

     
  • Daniel Stenberg

    Daniel Stenberg - 2005-03-04

    Logged In: YES
    user_id=1110

    I don't know much about VMS and file modes/types, but I
    don't see how a zip file differs from an exe or a text file,
    since curl sends the data using binary mode and uses simple
    file operations to read (binary) data from the file and
    sends it.

    curl doesn't even support ftp ascii transfers properly (and
    you're not using that).

    I don't know what to do here.

     
  • Nobody/Anonymous

    Logged In: NO

    My soluction for this is:

    1) Add --crlf option before sending the zip file
    2) Build a program to read through the received file and
    convert each CRLF back to LF and save it as a new file
    3) Use winzip to open this new file.

    It seems to be working in this way.

    Ching

     
  • Daniel Stenberg

    Daniel Stenberg - 2005-03-11

    Logged In: YES
    user_id=1110

    Still, if a binary transfer doesn't send the file binary
    without any kind of translation, the bug is in your ftp
    server. libcurl does no translation at all.

     
  • Daniel Stenberg

    Daniel Stenberg - 2005-03-11
    • status: open --> open-works-for-me
     
  • Nobody/Anonymous

    Logged In: NO

    well, I don't think so. Because, if you add the --crlf option,
    curl will actually replace each 'lf' to 'crlf' before sending them -
    you can see this from the source code. Also, I can upload
    the file to my ftp server with ftp client built in VMS without
    any problem. So it means the problem is in curl, not the ftp
    server.
    BTW, I found this problem is not only to do with the zip file, it
    is actually to with any file that has 'lf' character in it.

    Ching

     
  • Nobody/Anonymous

    Logged In: NO

    libcurl may no do the translation, but it may chop the content
    after the last 'lf' because of the file size calculation bug - this
    is just my guess.

    Ching

     
  • Daniel Stenberg

    Daniel Stenberg - 2005-03-14

    Logged In: YES
    user_id=1110

    If you use --crlf it will modify the contents, yes. But then
    you asked for it and curl can't be blamed.

    You said you are doing a binary transfer and then you should
    NOT need any translations and thus --crlf is plain wrong to
    use. Why do you need to translate a binary file?

    What "file size calculation bug" are you referring to?

     
  • Nobody/Anonymous

    Logged In: NO

    Well, the problem is that Curl for VMS version does not have
    a same behavior as Curl for windows or any other ftp client
    software in terms of uploading the files, which has LF
    character insides (not necessary a binary file or ascii file).
    You can easily repeat this issue if you try it on a VMS
    machine.

    Also I am not complaining the --crtl option, on the contrary, I
    am saying this is the only way that we can play around with
    the problem.

    If you read the source code, you can see that with the --crtl
    option, curl explicitly read the file byte by byte to do the
    translation if needed, and then put them into a new buffer to
    send. While, without --crtl option, it just reads the file block
    by block to the buffer and somehow it can be partialy sent if
    there is a LF.

     
  • Marty Kuhrt

    Marty Kuhrt - 2005-03-22

    Logged In: YES
    user_id=663072

    Send me an email at vms_curl@kuhrt.net and we can work on
    this problem. Make sure to put VMS somewhere in the subject
    line so I know it is not just a generic curl announcement
    message.

    cURL on VMS _shouldn't_ be any different than any other OS,
    since I don't think there have been any changes to the code
    path that are VMS specific (at least nothing I've worked on,
    anyway). zip files on VMS should be fixed length 512 byte
    record files, which is the same for any executable, so if
    executables transfer OK it is odd that zip files don't.

    What IP stack are you using on VMS, and what ftp server are
    you connecting to do the upload? Most Windows boxes I've
    worked with have ftp clients, but not ftp servers, so if you
    are uploading to a Windows box the ftp server you are using
    might be important.

     
  • Marty Kuhrt

    Marty Kuhrt - 2005-03-22

    Logged In: YES
    user_id=663072

    P.S. I tried to use the same command you used in your
    initial problem report with the same version of curl and I
    _did_ get the

    %CURL-E-PARTIAL_FILE, partial file

    error, but the files were exactly the same, but that was VMS
    to VMS using Multinet as the IP stack. The file came out
    the same with the diff command and I could unzip it, too. YMMV.

     
  • Daniel Stenberg

    Daniel Stenberg - 2005-03-29

    Logged In: YES
    user_id=1110

    So, any suggestions on what to do with this report?

     
  • Nobody/Anonymous

    Logged In: NO

    My ftp server is called globalscape ftp server running in
    windows. (you can download it from www.globalscape.com).

    The problem I have here is that once file contains LF, it has
    the error (.not the type of the file.)

    BTW, does Curl 7.13 work in VMS 7.2.1 ?

    Thanks
    Ching

     
  • Daniel Stenberg

    Daniel Stenberg - 2005-04-05
    • status: open-works-for-me --> closed-later
     
  • Daniel Stenberg

    Daniel Stenberg - 2005-04-05

    Logged In: YES
    user_id=1110

    This is now added as KNOWN_BUGS #22. Closing this report.

     
  • John Malmberg

    John Malmberg - 2013-07-16

    ZIP files should be binary files and are routinely transferred by FTP that way.
    Transferring a ZIP as an ASCII file will corrupt them.

    I need to do some tests with ZIP files, but I suspect this is similar to CURL BUG 758, in this case the ZIP file format is not set to STREAM_LF for some reason so that an incorrect upload size is calculated.

    The protocol for FTP STRU support of VMS attributes appears to be implemented in the open source CKERMIT and the MAD GOAT FTP product, and the HG FTP product which is the successor to HG FTP. In that case the FTP client detects that the server is VMS and transfers the VMS files always as a binary file, but then sets the VMS specific record attributes so that the file structure on the server matches the client.

     
  • John Malmberg

    John Malmberg - 2013-07-23

    The fix for the original bug has been checked in to curl and should appear in curl 7.32.

    Curl will now convert and upload in the two variable length text file types that are produced by DCL and text editors as STREAM_LF files. Curl will also convert and upload STREAM_CR files as STREAM_LF files. This conversion is done by the OpenVMS C runtime.

    Curl will now upload all other file types as STREAM_LF files which respects the logical EOF in the final block.

    FTP STRU support of VMS will need to be tackled as a separate issue.