Scripted FTP transfer - SCO

This is a discussion on Scripted FTP transfer - SCO ; Hi all. I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've written numerous FTP transfer scripts before, but haven't really approached it this way. What I'd like to do is basically retrieve some files from a ...

+ Reply to Thread
Results 1 to 14 of 14

Thread: Scripted FTP transfer

  1. Scripted FTP transfer

    Hi all.

    I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've
    written numerous FTP transfer scripts before, but haven't really approached
    it this way.

    What I'd like to do is basically retrieve some files from a remote system,
    then move them at the remote end into an archive folder. Sounds pretty
    simple.

    In my script, I'm trying to write a FTP command file on the fly to redirect
    into my FTP command as follows:-

    ===========
    :
    ftphost=ftp.XXXXXX.com
    ftpuser=XXXXXX
    ftppass=XXXXXX
    ftpdir="/"
    ftparcdir="/arcdir"
    work=/tmp/xfer.$$
    #
    # get list of files waiting
    ftp -g -n ${ftphost} > ${work} 2>&1 << !
    user ${ftpuser} ${ftppass}
    cd ${ftpdir}
    dir
    bye
    !

    # process file list into ftp commands
    exec 3< ${work}
    while read line <&3
    do
    filename=`echo ${line} | awk '{print $4}'`
    fileext=`echo ${filename} | awk -F\. '{print $2}' | dd conv=ucase
    2>/dev/null`
    if [ "${fileext}" = "XML" ]
    then
    echo "get ${filename}" >> ${work}.2
    echo "ren ${filename} ${ftparcdir}/${filename}" >> ${work}.3
    fi
    done

    # write ftp script file to retrieve all XML files
    #echo "OPEN ${ftphost}" > /tmp/ftp.$$
    echo "USER ${ftpuser} ${ftppass}" >> /tmp/ftp.$$
    echo "binary" >> /tmp/ftp.$$
    echo "cd ${ftpdir}" >> /tmp/ftp.$$
    cat ${work}.2 >> /tmp/ftp.$$
    cat ${work}.3 >> /tmp/ftp.$$
    echo "bye" >> /tmp/ftp.$$

    # run script to transfer all XML files and archive them
    ftp -i -g -n ${ftphost} < /tmp/ftp.$$ > /tmp/ftp$$.log 2>&1
    rm -f ${work} ${work}.2 ${work}.3

    cat /tmp/ftp$$.log
    rm -f /tmp/ftp$$.log
    rm -f /tmp/ftp.$$

    exit 0

    ===========

    The script runs and creates my command file, but I just cannot get it to
    accept the commands from the temporary file. I have removed the redirect to
    the ftp$$.log file, and stripped out the relevant commands and run them
    directly from the command line.

    The output I get (either on the screen, or in the ftp$$.log file) is:-

    ?Invalid command
    ?Invalid command
    Not connected.
    Not connected.
    Not connected.
    Not connected.
    Not connected.
    Not connected.


    This is the method for ftp which I have found on Google, but cannot
    understand why it isn't working on my Unixware system. I get exactly the
    same results on an Openserver 5.0.6 system too.

    Any ideas what I'm doing wrong here?


    I suppose I could also do a "mget *.xml" in this instance, but not knowing
    the case of the files (it could be another Unix or Linux system at the other
    end - I don't know), this may not work.



    Many thanks,

    Julian.



  2. Re: Scripted FTP transfer

    On Apr 22, 2:34 am, "Julian Brett" wrote:
    > Hi all.
    >
    > I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've
    > written numerous FTP transfer scripts before, but haven't really approached
    > it this way.
    >
    > What I'd like to do is basically retrieve some files from a remote system,
    > then move them at the remote end into an archive folder. Sounds pretty
    > simple.
    >
    > In my script, I'm trying to write a FTP command file on the fly to redirect
    > into my FTP command as follows:-
    >
    > ===========
    > :
    > ftphost=ftp.XXXXXX.com
    > ftpuser=XXXXXX
    > ftppass=XXXXXX
    > ftpdir="/"
    > ftparcdir="/arcdir"
    > work=/tmp/xfer.$$
    > #
    > # get list of files waiting
    > ftp -g -n ${ftphost} > ${work} 2>&1 << !
    > user ${ftpuser} ${ftppass}
    > cd ${ftpdir}
    > dir
    > bye
    > !
    >
    > # process file list into ftp commands
    > exec 3< ${work}
    > while read line <&3
    > do
    > filename=`echo ${line} | awk '{print $4}'`
    > fileext=`echo ${filename} | awk -F\. '{print $2}' | dd conv=ucase
    > 2>/dev/null`
    > if [ "${fileext}" = "XML" ]
    > then
    > echo "get ${filename}" >> ${work}.2
    > echo "ren ${filename} ${ftparcdir}/${filename}" >> ${work}.3
    > fi
    > done
    >
    > # write ftp script file to retrieve all XML files
    > #echo "OPEN ${ftphost}" > /tmp/ftp.$$
    > echo "USER ${ftpuser} ${ftppass}" >> /tmp/ftp.$$
    > echo "binary" >> /tmp/ftp.$$
    > echo "cd ${ftpdir}" >> /tmp/ftp.$$
    > cat ${work}.2 >> /tmp/ftp.$$
    > cat ${work}.3 >> /tmp/ftp.$$
    > echo "bye" >> /tmp/ftp.$$
    >
    > # run script to transfer all XML files and archive them
    > ftp -i -g -n ${ftphost} < /tmp/ftp.$$ > /tmp/ftp$$.log 2>&1
    > rm -f ${work} ${work}.2 ${work}.3
    >
    > cat /tmp/ftp$$.log
    > rm -f /tmp/ftp$$.log
    > rm -f /tmp/ftp.$$
    >
    > exit 0
    >
    > ===========
    >
    > The script runs and creates my command file, but I just cannot get it to
    > accept the commands from the temporary file. I have removed the redirect to
    > the ftp$$.log file, and stripped out the relevant commands and run them
    > directly from the command line.
    >
    > The output I get (either on the screen, or in the ftp$$.log file) is:-
    >
    > ?Invalid command
    > ?Invalid command
    > Not connected.
    > Not connected.
    > Not connected.
    > Not connected.
    > Not connected.
    > Not connected.
    >
    > This is the method for ftp which I have found on Google, but cannot
    > understand why it isn't working on my Unixware system. I get exactly the
    > same results on an Openserver 5.0.6 system too.
    >
    > Any ideas what I'm doing wrong here?
    >
    > I suppose I could also do a "mget *.xml" in this instance, but not knowing
    > the case of the files (it could be another Unix or Linux system at the other
    > end - I don't know), this may not work.
    >
    > Many thanks,
    >
    > Julian.


    Sounds like a problem with the command file (/tmp/ftp.$$). Post
    that. From the "Not connected" errors I'm guessing it's a problem
    with establishing user credentials. Are you able to FTP interactively
    to the host from this box?

    BTW if this script is designed to run in a particular context I'd
    use .netrc files to contain the user name and password instead of
    echoing them in the script. It's somewhat more secure, makes the
    password easier to change, and simplifies your script.

    Better yet is SCP, but you didn't ask about that.

    R Robert
    *** Software

  3. Re: Scripted FTP transfer

    ThreeStar wrote:

    > Sounds like a problem with the command file (/tmp/ftp.$$). Post
    > that. From the "Not connected" errors I'm guessing it's a problem
    > with establishing user credentials. Are you able to FTP interactively
    > to the host from this box?
    >
    > BTW if this script is designed to run in a particular context I'd
    > use .netrc files to contain the user name and password instead of
    > echoing them in the script. It's somewhat more secure, makes the
    > password easier to change, and simplifies your script.
    >
    > Better yet is SCP, but you didn't ask about that.
    >
    > R Robert
    > *** Software


    SCP is *AWFUL* for this sort of thing. It mishandles symlinks, and SCP or SFTP
    access provides user access to the rest of the server's file system, with the
    user's normal privileges. That presents a real security issue.

    I'm a big proponent of WebDAV over HTTPS fur just this sort of thing, and for
    rsync over SSH with a restricted validation script as needed.

  4. Re: Scripted FTP transfer

    On Tue, Apr 22, 2008, Nico Kadel-Garcia wrote:
    >ThreeStar wrote:
    >
    >> Sounds like a problem with the command file (/tmp/ftp.$$). Post
    >> that. From the "Not connected" errors I'm guessing it's a problem
    >> with establishing user credentials. Are you able to FTP interactively
    >> to the host from this box?
    >>
    >> BTW if this script is designed to run in a particular context I'd
    >> use .netrc files to contain the user name and password instead of
    >> echoing them in the script. It's somewhat more secure, makes the
    >> password easier to change, and simplifies your script.
    >>
    >> Better yet is SCP, but you didn't ask about that.
    >>
    >> R Robert
    >> *** Software

    >
    >SCP is *AWFUL* for this sort of thing. It mishandles symlinks, and SCP or SFTP
    >access provides user access to the rest of the server's file system, with the
    >user's normal privileges. That presents a real security issue.
    >
    >I'm a big proponent of WebDAV over HTTPS fur just this sort of thing, and for
    >rsync over SSH with a restricted validation script as needed.


    Depending on the nature of the data, we often use rsync's modules
    to handle things like this, restricting access by IP address.
    Doing it that way does not require things like ssh identities with
    empty pass phrases or other potential security issues.

    Bill
    --
    INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC
    URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way
    Voice: (206) 236-1676 Mercer Island, WA 98040-0820
    Fax: (206) 232-9186

    Each individual of the society has a right to be protected in the
    enjoyment of his life, liberty, and property no part of the property
    of any individual can, with justice, be taken from him, or applied to
    public uses, without his own consent -- John Adams

  5. Re: Scripted FTP transfer

    On Apr 22, 5:34 am, "Julian Brett" wrote:
    > Hi all.
    >
    > I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've
    > written numerous FTP transfer scripts before, but haven't really approached
    > it this way.



    Don't do this. There are better ways to transfer files. If it must
    be ftp, use Perl:
    http://aplawrence.com/Unixart/perlnetftp.html

    Much easier, much more control, much more reliable.

  6. Re: Scripted FTP transfer

    On Apr 22, 8:06 am, Nico Kadel-Garcia wrote:
    > ThreeStar wrote:
    > > Sounds like a problem with the command file (/tmp/ftp.$$). Post
    > > that. From the "Not connected" errors I'm guessing it's a problem
    > > with establishing user credentials. Are you able to FTP interactively
    > > to the host from this box?

    >
    > > BTW if this script is designed to run in a particular context I'd
    > > use .netrc files to contain the user name and password instead of
    > > echoing them in the script. It's somewhat more secure, makes the
    > > password easier to change, and simplifies your script.

    >
    > > Better yet is SCP, but you didn't ask about that.

    >
    > > R Robert
    > > *** Software

    >
    > SCP is *AWFUL* for this sort of thing. It mishandles symlinks, and SCP or SFTP
    > access provides user access to the rest of the server's file system, with the
    > user's normal privileges. That presents a real security issue.
    >
    > I'm a big proponent of WebDAV over HTTPS fur just this sort of thing, and for
    > rsync over SSH with a restricted validation script as needed.


    AFAIK the only issue is that SCP copies symbolic links as files, which
    isn't necessarily a bad thing.

    I assume one would use SCP keys only for a user with restricted
    permissions. So it doesn't necessarily follow that SCP exposes the
    whole server.

    Better security is better, even if it's not the best security
    (whatever that means). SCP fits the simple scenario the poster laid
    out, and avoids sending login credentials over the Internet in clear
    text like FTP does. WebDAV and the other proposed solutions may be
    better yet, although we don't know enough about the poster's equipment
    or requirements or abilities to say for sure.

    --RLR

  7. Re: Scripted FTP transfer

    Rather then specify the username and password in your script, I would
    suggest you put the ftp destination name/IP address, user name and password
    details in to a .netrc file in the home directory of the user name that is
    going to run the ftp script, so....

    If you are running this from root, the .netrc will be created in /

    The format of .netrc would be:

    machine name/IP address login username password password

    set chmod to 600 for .netrc. Then run your script minus the ftpuser and
    ftppass variables

    Dave

    "Julian Brett" wrote in message
    news:fukbc8$mej$1$8300dec7@news.demon.co.uk...
    > Hi all.
    >
    > I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server.
    > I've written numerous FTP transfer scripts before, but haven't really
    > approached it this way.
    >
    > What I'd like to do is basically retrieve some files from a remote system,
    > then move them at the remote end into an archive folder. Sounds pretty
    > simple.
    >
    > In my script, I'm trying to write a FTP command file on the fly to
    > redirect into my FTP command as follows:-
    >
    > ===========
    > :
    > ftphost=ftp.XXXXXX.com
    > ftpuser=XXXXXX
    > ftppass=XXXXXX
    > ftpdir="/"
    > ftparcdir="/arcdir"
    > work=/tmp/xfer.$$
    > #
    > # get list of files waiting
    > ftp -g -n ${ftphost} > ${work} 2>&1 << !
    > user ${ftpuser} ${ftppass}
    > cd ${ftpdir}
    > dir
    > bye
    > !
    >
    > # process file list into ftp commands
    > exec 3< ${work}
    > while read line <&3
    > do
    > filename=`echo ${line} | awk '{print $4}'`
    > fileext=`echo ${filename} | awk -F\. '{print $2}' | dd conv=ucase
    > 2>/dev/null`
    > if [ "${fileext}" = "XML" ]
    > then
    > echo "get ${filename}" >> ${work}.2
    > echo "ren ${filename} ${ftparcdir}/${filename}" >> ${work}.3
    > fi
    > done
    >
    > # write ftp script file to retrieve all XML files
    > #echo "OPEN ${ftphost}" > /tmp/ftp.$$
    > echo "USER ${ftpuser} ${ftppass}" >> /tmp/ftp.$$
    > echo "binary" >> /tmp/ftp.$$
    > echo "cd ${ftpdir}" >> /tmp/ftp.$$
    > cat ${work}.2 >> /tmp/ftp.$$
    > cat ${work}.3 >> /tmp/ftp.$$
    > echo "bye" >> /tmp/ftp.$$
    >
    > # run script to transfer all XML files and archive them
    > ftp -i -g -n ${ftphost} < /tmp/ftp.$$ > /tmp/ftp$$.log 2>&1
    > rm -f ${work} ${work}.2 ${work}.3
    >
    > cat /tmp/ftp$$.log
    > rm -f /tmp/ftp$$.log
    > rm -f /tmp/ftp.$$
    >
    > exit 0
    >
    > ===========
    >
    > The script runs and creates my command file, but I just cannot get it to
    > accept the commands from the temporary file. I have removed the redirect
    > to the ftp$$.log file, and stripped out the relevant commands and run them
    > directly from the command line.
    >
    > The output I get (either on the screen, or in the ftp$$.log file) is:-
    >
    > ?Invalid command
    > ?Invalid command
    > Not connected.
    > Not connected.
    > Not connected.
    > Not connected.
    > Not connected.
    > Not connected.
    >
    >
    > This is the method for ftp which I have found on Google, but cannot
    > understand why it isn't working on my Unixware system. I get exactly the
    > same results on an Openserver 5.0.6 system too.
    >
    > Any ideas what I'm doing wrong here?
    >
    >
    > I suppose I could also do a "mget *.xml" in this instance, but not knowing
    > the case of the files (it could be another Unix or Linux system at the
    > other end - I don't know), this may not work.
    >
    >
    >
    > Many thanks,
    >
    > Julian.
    >




  8. Re: Scripted FTP transfer



    On Tue, 22 Apr 2008, ThreeStar wrote:

    >
    > Better security is better, even if it's not the best security
    > (whatever that means). SCP fits the simple scenario the poster laid
    > out, and avoids sending login credentials over the Internet in clear
    > text like FTP does.


    There are FTP implementations available that support encryption. They
    won't work through many firewalls, but as long as only one end has an
    external firewall, they can be used.

  9. Re: Scripted FTP transfer

    On Tue, 22 Apr 2008, Joe Dunning wrote:

    > There are FTP implementations available that support encryption. They
    > won't work through many firewalls, but as long as only one end has an
    > external firewall, they can be used.


    Are you referring to FTP over SSL, sometimes known as FTPS? If not, what?

    Regards,
    .....Bob Rasmussen, President, Rasmussen Software, Inc.

    personal e-mail: ras@anzio.com
    company e-mail: rsi@anzio.com
    voice: (US) 503-624-0360 (9:00-6:00 Pacific Time)
    fax: (US) 503-624-0760
    web: http://www.anzio.com
    street address: Rasmussen Software, Inc.
    10240 SW Nimbus, Suite L9
    Portland, OR 97223 USA

  10. Re: Scripted FTP transfer

    In article , Julian Brett wrote:
    > Hi all.
    >
    > I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've
    > written numerous FTP transfer scripts before, but haven't really approached
    > it this way.




    This sort of thing in a script file has always worked for me
    under various unixen:



    cd_dir="somedir"
    file="somefile.txt"
    user="myusername"
    host="myhost.com"
    echo -n "Password for ${user} at ${host}: "
    read pass

    echo "
    open ${host}
    user ${user} ${pass}
    #passive
    bin
    hash
    prompt
    cd ${cd_dir}
    pwd
    ls
    chmod 700 *
    mdele *
    put ${file}
    chmod 000 *
    ls
    " | ftp -n


  11. Re: Scripted FTP transfer

    On Apr 24, 3:04 am, bco...@example.com wrote:
    > In article , Julian Brett wrote:
    > > Hi all.

    >
    > > I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've
    > > written numerous FTP transfer scripts before, but haven't really approached
    > > it this way.

    >
    > This sort of thing in a script file has always worked for me
    > under various unixen:
    >
    > cd_dir="somedir"
    > file="somefile.txt"
    > user="myusername"
    > host="myhost.com"
    > echo -n "Password for ${user} at ${host}: "
    > read pass
    >
    > echo "
    > open ${host}
    > user ${user} ${pass}
    > #passive
    > bin
    > hash
    > prompt
    > cd ${cd_dir}
    > pwd
    > ls
    > chmod 700 *
    > mdele *
    > put ${file}
    > chmod 000 *
    > ls
    > " | ftp -n


    Really? And how do you know when it fails and WHERE it failed?

    You don't, and that's why I recommend doing this with Perl. It's easy
    (I have sample scripts at the links I gave above) and you have total
    control of errors..


  12. Re: Scripted FTP transfer

    Joe Dunning wrote:
    >
    >
    > On Tue, 22 Apr 2008, ThreeStar wrote:
    >
    >>
    >> Better security is better, even if it's not the best security
    >> (whatever that means). SCP fits the simple scenario the poster laid
    >> out, and avoids sending login credentials over the Internet in clear
    >> text like FTP does.

    >
    > There are FTP implementations available that support encryption. They
    > won't work through many firewalls, but as long as only one end has an
    > external firewall, they can be used.


    Most of them are not, in fact, FTP. 'sftp' is one such, and it's basically SCP
    with an interface, suffering most of the flaws of SCP and rcp access. The
    interface is far less flexible than FTP, it normally has no chroot capability
    and thus no way to restrict clients from browsing the rest of the filesystem,
    and it mishandles symlinks.

    These are reasons that I like WebDAV over HTTPS: it has most if not all of the
    features of FTP, it runs on the HTTPS port, and is easily implemented with a
    basic Apache installation with a great deal of flexibility.

  13. Re: Scripted FTP transfer

    In article <7f8d4cf6-5438-46b4-8b2d-00f17f53f84d@c65g2000hsa.googlegroups.com>, Tony Lawrence wrote:
    > On Apr 24, 3:04 am, bco...@example.com wrote:


    >> put ${file}
    >> chmod 000 *
    >> ls
    >> " | ftp -n

    >
    > Really? And how do you know when it fails and WHERE it failed?
    >


    Your right. This is only usefull as a quick and dirty one time script.
    Its not secure and adding any sort of robust error checking turns the
    simple script into something complicated enought that something like Perl
    quickly becomes the better solution.

    However, that said, sometimes one really wants or needs to do FTP and this
    is simple enough to remember and just type in.






  14. Re: Scripted FTP transfer



    On Sat, 26 Apr 2008, Nico Kadel-Garcia wrote:

    > Joe Dunning wrote:
    >>
    >>
    >> On Tue, 22 Apr 2008, ThreeStar wrote:
    >>
    >> >
    >> > Better security is better, even if it's not the best security
    >> > (whatever that means). SCP fits the simple scenario the poster laid
    >> > out, and avoids sending login credentials over the Internet in clear
    >> > text like FTP does.

    >>
    >> There are FTP implementations available that support encryption. They
    >> won't work through many firewalls, but as long as only one end has an
    >> external firewall, they can be used.

    >
    > Most of them are not, in fact, FTP.


    Most, maybe. However, on my Gentoo boxes, FTP (net-ftp/ftp) is built with
    SSl support. However, this can be problematic, because, if one needs the
    NAT firewall to react properly to the ftp commands going across the
    firewall, it won't happen with an encrypted session.

+ Reply to Thread