Bug 6848

Summary: Change in behaviour with -f flag between versions 3.23 and 4.14
Product: GridFTP Reporter: Paula Landart <paula.landart@terradue.com>
Component: GridFTPAssignee: Mike Link <mlink@mcs.anl.gov>
Status: NEW    
Severity: minor CC: bresnaha@mcs.anl.gov, kettimut@mcs.anl.gov, paula.landart@terradue.com
Priority: P3    
Version: 4.2.0   
Target Milestone: ---   
Hardware: PC   
OS: Linux   

Description From 2009-09-01 05:46:33
We recently upgraded to globus-url-copy 4.14 as part of a general upgrade to
globus toolkit 4.2.1, and noted a change in behaviour with the "-f" flag to
globus-url-copy (-f = read input URLs from a file). When the URLs point to
files contained in different directories, e.g.:

  file:///home/testuser/tmp/dir_a/testfile.txt 
gsiftp://machine3:2811/var/tmp/dir_a/from_machine1.txt
  file:///home/testuser/tmp/dir_b/testfile.txt 
gsiftp://machine3:2811/var/tmp/dir_b/from_machine1.txt
  globus-url-copy 4.14 only creates the first remote directory, "dir_a", and
then attempts to transfer "dir_b/testfile.txt" without creating the remote
directory first.

Debug output showing the difference is below.


--------- output ----------------
## globus-url-copy: 4.14
## command:
globus-url-copy -dbg -vb -cd -f remotefiles

## content of remotefiles:
file:///home/testuser/tmp/dir_a/testfile.txt 
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from
_machine1.txt
file:///home/testuser/tmp/dir_a/testfile_1.txt 
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/fr
om_machine1_1.txt
file:///home/testuser/tmp/dir_b/testfile.txt 
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from
_machine1.txt

## remote files received:
/var/tmp
/var/tmp/dir_a
/var/tmp/dir_a/from_machine1_1.txt


## output:

debug: starting to mkdir gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
debug: connecting to gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
220 machine3.domain.com GridFTP Server 3.15 (gcc64dbgpthr, 1222656151-78)
[Globus Toolkit 4.2.1] r
eady.

debug: authenticating with gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
230 User testuser logged in.

debug: sending command to gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
SITE HELP

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
214-The following commands are recognized:
    ALLO    APPE    REST    CWD     CDUP    DCAU    EPSV    FEAT
    ERET    MDTM    STAT    ESTO    HELP    LIST    MODE    NLST
    MLSD    PASV    RNFR    MLST    NOOP    OPTS    STOR    PASS
    PBSZ    PORT    PROT    SITE    EPRT    RETR    SPOR    SCKS
    TREV    PWD     QUIT    SBUF    SIZE    SPAS    STRU    SYST
    RNTO    TYPE    USER    LANG    MKD     RMD     DELE    CKSM
214 End

debug: sending command to gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
FEAT

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
211-Extensions supported
 AUTHZ_ASSERT
 UTF8
 LANG EN
 DCAU
 PARALLEL
 SIZE
 MLST
Type*;Size*;Modify*;Perm*;Charset;UNIX.mode*;UNIX.owner*;UNIX.group*;Unique*;UNIX.slink*;
 ERET
 ESTO
 SPAS
 SPOR
 REST STREAM
 MDTM
 PASV AllowDelayed;
211 End.

debug: sending command to gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
TYPE I
debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
200 Type set to I.

debug: sending command to gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
PBSZ 1048576

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
200 PBSZ=1048576

debug: sending command to gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
MKD /var/tmp/dir_a/

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
257 Directory "/var/tmp/dir_a/" created successfully.

debug: operation complete
debug: starting to put
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt
debug: sending command to
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
PASV

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
227 Entering Passive Mode (62,123,237,245,95,63)

debug: sending command to
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
ALLO 7

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
200 ALLO command successful.

debug: sending command to
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
STOR /var/tmp/dir_a/from_machine1_1.txt

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
150 Begining transfer.

debug: writing buffer 0xb4c3e008, length 7, offset=0, eof=true
debug: data callback, no error, buffer 0xb4c3e008, length 7, offset=0, eof=true
debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
226 Transfer Complete.

debug: operation complete
debug: starting to put
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt
debug: sending command to
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
PASV

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
227 Entering Passive Mode (62,123,237,245,81,150)

debug: sending command to
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
ALLO 7

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
200 ALLO command successful.

debug: sending command to
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
STOR /var/tmp/dir_b/from_machine1.txt

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
500-Command failed. : globus_l_gfs_file_open failed.
500-globus_xio: Unable to open file /var/tmp/dir_b/from_machine1.txt
500-globus_xio: System error in open: No such file or directory
500-globus_xio: A system call failed: No such file or directory
500 End.

debug: fault on connection to
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
globus_ftp_client: the server responded with an error
debug: data callback, error globus_ftp_client: the server responded with an
error, buffer 0xb4c3e0
08, length 0, offset=0, eof=true
debug: operation complete

error: globus_ftp_client: the server responded with an error
500 500-Command failed. : globus_l_gfs_file_open failed.
500-globus_xio: Unable to open file /var/tmp/dir_b/from_machine1.txt
500-globus_xio: System error in open: No such file or directory
500-globus_xio: A system call failed: No such file or directory
500 End.

Source: file:///home/testuser/tmp/dir_a/
Dest:   gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
  testfile_1.txt  ->  from_machine1_1.txt

Source: file:///home/testuser/tmp/dir_b/
Dest:   gsiftp://machine3.domain.com:2811/var/tmp/dir_b/
  testfile.txt  ->  from_machine1.txt

## globus-url-copy: 3.23
## command:
globus-url-copy -dbg -vb -cd -f remotefiles 

## content of remotefiles:
file:///home/testuser/tmp/dir_a/testfile.txt 
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from _machine1.txt
file:///home/testuser/tmp/dir_a/testfile_1.txt 
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/fr
om_machine1_1.txt
file:///home/testuser/tmp/dir_b/testfile.txt 
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from
_machine1.txt

## remote files, transfer successful:
/var/tmp/dir_a
/var/tmp/dir_a/from_machine1.txt
/var/tmp/dir_a/from_machine1_1.txt
/var/tmp/dir_b
/var/tmp/dir_b/from_machine1.txt


## output:

debug: starting to mkdir gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
debug: connecting to gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
220 machine3.domain.com GridFTP Server 3.15 (gcc64dbgpthr, 1222656151-78)
[Globus Toolkit 4.2.1] r
eady.

debug: authenticating with gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
230 User testuser logged in.


debug: sending command:
SITE HELP

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
214-The following commands are recognized:
    ALLO    APPE    REST    CWD     CDUP    DCAU    EPSV    FEAT
    ERET    MDTM    STAT    ESTO    HELP    LIST    MODE    NLST
    MLSD    PASV    RNFR    MLST    NOOP    OPTS    STOR    PASS
    PBSZ    PORT    PROT    SITE    EPRT    RETR    SPOR    SCKS
    TREV    PWD     QUIT    SBUF    SIZE    SPAS    STRU    SYST
    RNTO    TYPE    USER    LANG    MKD     RMD     DELE    CKSM
214 End

debug: sending command:
FEAT

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
211-Extensions supported
 AUTHZ_ASSERT
 UTF8
 LANG EN
 DCAU
 PARALLEL
 SIZE
 MLST
Type*;Size*;Modify*;Perm*;Charset;UNIX.mode*;UNIX.owner*;UNIX.group*;Unique*;UNIX.slink*;
 ERET
 ESTO
 SPAS
 SPOR
 REST STREAM
 MDTM
 PASV AllowDelayed;
211 End.

debug: sending command:
TYPE I
debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
200 Type set to I.

debug: sending command:
PBSZ 1048576

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
200 PBSZ=1048576

debug: sending command:
MKD /var/tmp/dir_a/

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
257 Directory "/var/tmp/dir_a/" created successfully.

debug: operation complete
debug: starting to put
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1.txt
debug: sending command:
PASV

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1.txt:
227 Entering Passive Mode (62,123,237,245,95,63)

debug: sending command:
ALLO 7

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1.txt:
200 ALLO command successful.

debug: sending command:
STOR /var/tmp/dir_a/from_machine1.txt

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1.txt:
150 Begining transfer.

debug: data callback, no error, buffer 0xb7e4a008, length 7, offset=0, eof=true
debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1.txt:
226 Transfer Complete.

debug: operation complete
debug: starting to mkdir gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
debug: sending command:
MKD /var/tmp/dir_a/

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a/:
500-Command failed : System error in mkdir: File exists
500-A system call failed: File exists
500 End.

debug: operation complete
debug: starting to mkdir gsiftp://machine3.domain.com:2811/var
debug: sending command:
MKD /var

debug: response from gsiftp://machine3.domain.com:2811/var:
500-Command failed : System error in mkdir: File exists
500-A system call failed: File exists
500 End.

debug: operation complete
debug: starting to mkdir gsiftp://machine3.domain.com:2811/var/tmp
debug: sending command:
MKD /var/tmp

debug: response from gsiftp://machine3.domain.com:2811/var/tmp:
500-Command failed : System error in mkdir: File exists
500-A system call failed: File exists
500 End.

debug: operation complete
debug: starting to mkdir gsiftp://machine3.domain.com:2811/var/tmp/dir_a
debug: sending command:
MKD /var/tmp/dir_a

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_a:
500-Command failed : System error in mkdir: File exists
500-A system call failed: File exists
500 End.

debug: operation complete
debug: starting to put
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt
debug: sending command:
PASV

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
227 Entering Passive Mode (62,123,237,245,81,150)

debug: sending command:
ALLO 7

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
200 ALLO command successful.

debug: sending command:
STOR /var/tmp/dir_a/from_machine1_1.txt

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
150 Begining transfer.

debug: data callback, no error, buffer 0xb7e4a008, length 7, offset=0, eof=true
debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_a/from_machine1_1.txt:
226 Transfer Complete.

debug: operation complete
debug: starting to mkdir gsiftp://machine3.domain.com:2811/var/tmp/dir_b/
debug: sending command:
MKD /var/tmp/dir_b/

debug: response from gsiftp://machine3.domain.com:2811/var/tmp/dir_b/:
257 Directory "/var/tmp/dir_b/" created successfully.

debug: operation complete
debug: starting to put
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt
debug: sending command:
PASV

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
227 Entering Passive Mode (62,123,237,245,88,249)

debug: sending command:
ALLO 7

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
200 ALLO command successful.

debug: sending command:
STOR /var/tmp/dir_b/from_machine1.txt

debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
150 Begining transfer.

debug: data callback, no error, buffer 0xb7e4a008, length 7, offset=0, eof=true
debug: response from
gsiftp://machine3.domain.com:2811/var/tmp/dir_b/from_machine1.txt:
226 Transfer Complete.

debug: operation complete
Source: file:///home/testuser/tmp/dir_a/
Dest:   gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
  testfile.txt  ->  from_machine1.txt

Source: file:///home/testuser/tmp/dir_a/
Dest:   gsiftp://machine3.domain.com:2811/var/tmp/dir_a/
  testfile_1.txt  ->  from_machine1_1.txt


Source: file:///home/testuser/tmp/dir_b/
Dest:   gsiftp://machine3.domain.com:2811/var/tmp/dir_b/
  testfile.txt  ->  from_machine1.txt