Problem in Kermit trying to get a file while sending it at the same time - Protocols

This is a discussion on Problem in Kermit trying to get a file while sending it at the same time - Protocols ; I ran across this error as the scenerio describes. Both are trying to connect using SSH and CKERMIT. I am running a Kermit script called DROPOFF trying to send TESTFILE to a remote server (it is a 2meg file) at ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: Problem in Kermit trying to get a file while sending it at the same time

  1. Problem in Kermit trying to get a file while sending it at the same time

    I ran across this error as the scenerio describes. Both are trying to
    connect using SSH and CKERMIT.

    I am running a Kermit script called DROPOFF trying to send TESTFILE to
    a remote server (it is a 2meg file) at the same time I am running a
    Kermit PICKUP trying to mget files (including the TESTFILE) from this
    same server in the same directory.

    What is happening is, PICKUP ends first and has a partial file of
    TESTFILE and DROPOFF gets 74% done (according to Kermit) and then
    comes back with the error message "Last error: FAILURE: Error writing
    data" and it keeps around the partial file (74% of it).

    I'm not sure why this is happening and how to avoid it. It is a
    realtime scenerio where somebody may be dropping off a file in a
    directory, while the other script is trying to get all the files out
    of this directory.

    BTW: I am using Kermit so I can use the mget /delete option.

  2. Re: Problem in Kermit trying to get a file while sending it at the same time

    In article ,
    newexpectuser wrote:
    : I ran across this error as the scenerio describes. Both are trying to
    : connect using SSH and CKERMIT.
    :
    : I am running a Kermit script called DROPOFF trying to send TESTFILE to
    : a remote server (it is a 2meg file) at the same time I am running a
    : Kermit PICKUP trying to mget files (including the TESTFILE) from this
    : same server in the same directory.
    :
    : What is happening is, PICKUP ends first and has a partial file of
    : TESTFILE and DROPOFF gets 74% done (according to Kermit) and then
    : comes back with the error message "Last error: FAILURE: Error writing
    : data" and it keeps around the partial file (74% of it).
    :
    : I'm not sure why this is happening and how to avoid it. It is a
    : realtime scenerio where somebody may be dropping off a file in a
    : directory, while the other script is trying to get all the files out
    : of this directory.
    :
    You answered your own question:

    : I am running a Kermit script called DROPOFF trying to send TESTFILE to
    : a remote server (it is a 2meg file) at the same time I am running a
    : Kermit PICKUP trying to mget files (including the TESTFILE) from this
    : same server in the same directory.

    The server does not have permission to write to a file that another
    process has open for reading. Nor would you want it to! Please read
    about transaction processing here:

    http://www.columbia.edu/kermit/case10.html

    and here:

    http://www.columbia.edu/kermit/ftpscripts.html#tp

    to see the kinds of things you need to do.

    - Frank

  3. Re: Problem in Kermit trying to get a file while sending it at the same time

    Looking at Case Study #10, this tells me there would need to be two
    scripts running, 1 on the Branch machine to "send" the files and 1 on
    the HQ machine to "set receive" the files.

    My case is a little different, my HQ machine is the one running the
    Kermit Script (logging onto the Branch machine, changing to the
    directory where the file resides on the remote machine and doing a
    mget * command), so this example may not help me, unless I broke up
    the scripts into two.

    The issue with doing that is the Branch machine is a secure ftp server
    and cannot communicate over to the HQ machine for security reasons.
    Frank da Cruz wrote in message news:...
    > In article ,
    > newexpectuser wrote:
    > : I ran across this error as the scenerio describes. Both are trying to
    > : connect using SSH and CKERMIT.
    > :
    > : I am running a Kermit script called DROPOFF trying to send TESTFILE to
    > : a remote server (it is a 2meg file) at the same time I am running a
    > : Kermit PICKUP trying to mget files (including the TESTFILE) from this
    > : same server in the same directory.
    > :
    > : What is happening is, PICKUP ends first and has a partial file of
    > : TESTFILE and DROPOFF gets 74% done (according to Kermit) and then
    > : comes back with the error message "Last error: FAILURE: Error writing
    > : data" and it keeps around the partial file (74% of it).
    > :
    > : I'm not sure why this is happening and how to avoid it. It is a
    > : realtime scenerio where somebody may be dropping off a file in a
    > : directory, while the other script is trying to get all the files out
    > : of this directory.
    > :
    > You answered your own question:
    >
    > : I am running a Kermit script called DROPOFF trying to send TESTFILE to
    > : a remote server (it is a 2meg file) at the same time I am running a
    > : Kermit PICKUP trying to mget files (including the TESTFILE) from this
    > : same server in the same directory.
    >
    > The server does not have permission to write to a file that another
    > process has open for reading. Nor would you want it to! Please read
    > about transaction processing here:
    >
    > http://www.columbia.edu/kermit/case10.html
    >
    > and here:
    >
    > http://www.columbia.edu/kermit/ftpscripts.html#tp
    >
    > to see the kinds of things you need to do.
    >
    > - Frank


  4. Re: Problem in Kermit trying to get a file while sending it at the same time

    In article ,
    newexpectuser wrote:
    : Looking at Case Study #10, this tells me there would need to be two
    : scripts running, 1 on the Branch machine to "send" the files and 1 on
    : the HQ machine to "set receive" the files.
    :
    : My case is a little different, my HQ machine is the one running the
    : Kermit Script (logging onto the Branch machine, changing to the
    : directory where the file resides on the remote machine and doing a
    : mget * command), so this example may not help me, unless I broke up
    : the scripts into two.
    :
    You can have any arrangement you want: caller is HQ vs Branch; the server
    can be the caller or the callee, etc. Just reverse the example: tell
    the server on the Branch machine to "set send move-to ..." or whatever
    before it goes into server mode, and then tell the client at HQ to
    "mget" the files.

    : The issue with doing that is the Branch machine is a secure ftp server
    : and cannot communicate over to the HQ machine for security reasons.
    :
    I'm sure it's possible to have Kermit do whatever you want to it do.
    Read the manual and the updates and the case studies; study the sample
    scripts.

    - Frank

  5. Re: Problem in Kermit trying to get a file while sending it at the same time

    My main issue is the file(s) I want are on a secure ftp server. This
    process has other companies dropping off their files off in a certain
    directory, so if they are dropping off a large file in this directory
    at the same time I am trying to get the files from this directory, it
    seems I am getting only a partial file.

    I had thought remembering some protcols cannot "see" files in a
    directory until they are 100% finished being dropped off, I'm not sure
    if Kermit can handle this situation.

    Also, the way I am testing the method of dropping off the file is
    using a kermit script to connect to the server and using the send
    command (to send from my local area) after doing a rcd command.
    > Looking at Case Study #10, this tells me there would need to be two
    > scripts running, 1 on the Branch machine to "send" the files and 1 on
    > the HQ machine to "set receive" the files.
    >
    > My case is a little different, my HQ machine is the one running the
    > Kermit Script (logging onto the Branch machine, changing to the
    > directory where the file resides on the remote machine and doing a
    > mget * command), so this example may not help me, unless I broke up
    > the scripts into two.
    >
    > The issue with doing that is the Branch machine is a secure ftp server
    > and cannot communicate over to the HQ machine for security reasons.
    > Frank da Cruz wrote in message news:...
    > > In article ,
    > > newexpectuser wrote:
    > > : I ran across this error as the scenerio describes. Both are trying to
    > > : connect using SSH and CKERMIT.
    > > :
    > > : I am running a Kermit script called DROPOFF trying to send TESTFILE to
    > > : a remote server (it is a 2meg file) at the same time I am running a
    > > : Kermit PICKUP trying to mget files (including the TESTFILE) from this
    > > : same server in the same directory.
    > > :
    > > : What is happening is, PICKUP ends first and has a partial file of
    > > : TESTFILE and DROPOFF gets 74% done (according to Kermit) and then
    > > : comes back with the error message "Last error: FAILURE: Error writing
    > > : data" and it keeps around the partial file (74% of it).
    > > :
    > > : I'm not sure why this is happening and how to avoid it. It is a
    > > : realtime scenerio where somebody may be dropping off a file in a
    > > : directory, while the other script is trying to get all the files out
    > > : of this directory.
    > > :
    > > You answered your own question:
    > >
    > > : I am running a Kermit script called DROPOFF trying to send TESTFILE to
    > > : a remote server (it is a 2meg file) at the same time I am running a
    > > : Kermit PICKUP trying to mget files (including the TESTFILE) from this
    > > : same server in the same directory.
    > >
    > > The server does not have permission to write to a file that another
    > > process has open for reading. Nor would you want it to! Please read
    > > about transaction processing here:
    > >
    > > http://www.columbia.edu/kermit/case10.html
    > >
    > > and here:
    > >
    > > http://www.columbia.edu/kermit/ftpscripts.html#tp
    > >
    > > to see the kinds of things you need to do.
    > >
    > > - Frank


  6. Re: Problem in Kermit trying to get a file while sending it at the same time

    In article ,
    newexpectuser wrote:
    : My main issue is the file(s) I want are on a secure ftp server. This
    : process has other companies dropping off their files off in a certain
    : directory, so if they are dropping off a large file in this directory
    : at the same time I am trying to get the files from this directory, it
    : seems I am getting only a partial file.
    :
    Please read the the FTP scripting tutorial, including the "transaction
    processing" section. I've referred you to it before. It addresses
    exactly this problem.

    http://www.columbia.edu/kermit/ftpscripts.html

    - Frank

  7. Re: Problem in Kermit trying to get a file while sending it at the same time

    I think I understand the following in the article:

    "Notice that failure leaves the partial file (if any) in the working
    directory, where the central-site watcher process does not look for
    it. Thus transient failures do no harm. The script can be run again
    later.

    The /DELETE switch on the PUT command removes the source file after,
    and only if, it was uploaded successfully; this prevents it from being
    uploaded again (you could also have it moved or renamed). This way,
    even if the script is run again for the same file, it will fail
    immediately because the file is no longer there. Or, if a file of the
    same name is in the same place, it is a new file that should be
    uploaded.

    Now we can move the uploaded file from the server's working directory
    to its ready directory (the syntax assumes a UNIX-like file system on
    server):


    ftp rename \m(nameonly) ../ready/\m(nameonly)"


    This solves my problem if I have a partial file, it will be transfered
    into a directory (working) where my central-site server won't be
    looking for it. Once this file has been dropped off 100%, then the
    next time around I run to send files, I will then pickup a full file
    in my working directory.

    What I don't understand is with the above "ftp rename \m(nameonly)
    .../ready/\m(nameonly)", how does this avoid moving the partial file
    from the working directory to the ready directory, there will still be
    a file in the working directory albeit a partial file and this command
    will move it along to the ready directory.




    Frank da Cruz wrote in message news:...
    > In article ,
    > newexpectuser wrote:
    > : My main issue is the file(s) I want are on a secure ftp server. This
    > : process has other companies dropping off their files off in a certain
    > : directory, so if they are dropping off a large file in this directory
    > : at the same time I am trying to get the files from this directory, it
    > : seems I am getting only a partial file.
    > :
    > Please read the the FTP scripting tutorial, including the "transaction
    > processing" section. I've referred you to it before. It addresses
    > exactly this problem.
    >
    > http://www.columbia.edu/kermit/ftpscripts.html
    >
    > - Frank


  8. Re: Problem in Kermit trying to get a file while sending it at the same time

    In article ,
    newexpectuser wrote:
    :
    : (quoting from http://www.columbia.edu/kermit/ftpscripts.html):
    :
    : "...Now we can move the uploaded file from the server's working directory
    : to its ready directory (the syntax assumes a UNIX-like file system on
    : server):
    :
    : ftp rename \m(nameonly) ../ready/\m(nameonly)"
    :
    : This solves my problem if I have a partial file, it will be transfered
    : into a directory (working) where my central-site server won't be
    : looking for it. Once this file has been dropped off 100%, then the
    : next time around I run to send files, I will then pickup a full file
    : in my working directory.
    :
    : What I don't understand is with the above "ftp rename \m(nameonly)
    : ../ready/\m(nameonly)", how does this avoid moving the partial file
    : from the working directory to the ready directory, there will still be
    : a file in the working directory albeit a partial file and this command
    : will move it along to the ready directory.
    :
    The lines preceding the "ftp rename" command are:

    ftp put /delete \m(filename)
    if fail exit 1 ftp put \m(filename): \v(ftp_message)

    Thus the "ftp rename" command won't be reached if "ftp put" fails.
    This is exactly the reason that we recommend that each critical step
    be checked with IF FAIL -- you don't want the script to proceed in
    cases where the subsequent commands require the preceding ones to
    succeed.

    Of course conditional command in the IF statement need not be EXIT; it
    can be anything you want, even a series of commands, e.g.:

    if fail {
    command1
    command2
    ...
    } else { ; The ELSE part is optional
    command1
    command2
    ...
    }

    - Frank

  9. Re: Problem in Kermit trying to get a file while sending it at the same time

    anthonypieper@cs.com (newexpectuser) wrote in message news:...
    > I think I understand the following in the article:
    >
    > "Notice that failure leaves the partial file (if any) in the working
    > directory, where the central-site watcher process does not look for
    > it. Thus transient failures do no harm. The script can be run again
    > later.
    >
    > The /DELETE switch on the PUT command removes the source file after,
    > and only if, it was uploaded successfully; this prevents it from being
    > uploaded again (you could also have it moved or renamed). This way,
    > even if the script is run again for the same file, it will fail
    > immediately because the file is no longer there. Or, if a file of the
    > same name is in the same place, it is a new file that should be
    > uploaded.
    >
    > Now we can move the uploaded file from the server's working directory
    > to its ready directory (the syntax assumes a UNIX-like file system on
    > server):
    >
    >
    > ftp rename \m(nameonly) ../ready/\m(nameonly)"
    >
    >
    > This solves my problem if I have a partial file, it will be transfered
    > into a directory (working) where my central-site server won't be
    > looking for it. Once this file has been dropped off 100%, then the
    > next time around I run to send files, I will then pickup a full file
    > in my working directory.
    >
    > What I don't understand is with the above "ftp rename \m(nameonly)
    > ../ready/\m(nameonly)", how does this avoid moving the partial file
    > from the working directory to the ready directory, there will still be
    > a file in the working directory albeit a partial file and this command
    > will move it along to the ready directory.


    Both Frank and Jeffrey have been responding to your questions and have
    really already provided the answers to the above, but I have a sense
    that you still don't get it, so here's a shot at what I think the
    confusion is.

    Partial files occur in two ways. One way is that a transfer is
    interrupted because of a communications error, a disconnect or
    whatever. This may leave a partially transferred file on the
    receiver. The material you quote above is dealing with this situation
    and here the sender deals with it by transferring the file to a
    "working" directory, testing this transfer (with "if failure" or "if
    success") and only after successful complete transfer, moving the file
    on the receiver to the "ready" directory.

    The other kind of partial file that I think you're concerned about is
    a file that is in the process of being written by some other process
    when the sender considers it as a candidate for transfer. In this
    case, it is up to the underlying operating system to either hide the
    presense of the partial file until the creating process is done with
    it, or to deny access to the file by another process until the
    creating process in done with it. All real operating systems do this.
    If the process that is creating this candidate file is a file
    transfer process that could actually terminate before the file is
    complete, then it too needs to use a "working directory"/"ready
    directory" scheme.

    --
    Mark Sapiro msapiro -at- value net The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan

+ Reply to Thread