Re: Behavior of flock() oriented procedures - Unix

This is a discussion on Re: Behavior of flock() oriented procedures - Unix ; Aaron Hsu wrote: > I am trying to make some wrappers for file locking in order to use them > in another language. However, I cannot figure out the proper semantics > of flock oriented procedures. Specifically, the use of ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: Re: Behavior of flock() oriented procedures

  1. Re: Behavior of flock() oriented procedures

    Aaron Hsu wrote:
    > I am trying to make some wrappers for file locking in order to use them
    > in another language. However, I cannot figure out the proper semantics
    > of flock oriented procedures. Specifically, the use of open() and what
    > happens when a file descriptor is locked.


    > If I lock a file descriptor using either flock() or open(), then
    > obviously the file will remain locked in this manner at least until I
    > close the file, but is there a standard behavior for what happens to the
    > lock *after* I close the file descriptor? If I close the file
    > descriptore without explicitly calling flock() with the unlock message,
    > does this mean that the file will still have the lock on it, or does
    > this mean that the file will automatically be unlocked?


    If the process that holds the lock closes the file the lock vanishes
    without any explicit unlocking. All forms of locking work that way.

    > It would work better for my purposes if the lock remained until an
    > explicit unlock command,


    But the moment you close the file the file descriptor you accessed
    the file with loses its meaning (and the number gets re-used the
    for the next file you open). So you can't unlock it anymore!

    > but I want to know if this is behavior that I can
    > expect over the UNIX platforms in general, or if this is different?


    It might be nice for your purposes. But consider the consequences
    if a lock would continue to exist until an explicit "unlock" would
    be done. Open files get closed automatically the moment the process
    that opened them exits, and therefor also the lock vanishes. If the
    lock would continue to exist there would be no process anymore that
    could send such an "unlock" command. It would be extremely simple
    to write a "denial of serveic program" that locks a lot of impor-
    tant files and the exits.

    You could argue that on exit of a process an implicit unlock should
    be done to avoid that scenario. But if you have a process that for-
    gets to unlock a file but continues to runs for the next ten weeks
    that still would keep every other process from at least writing to
    the file for that time. Only someone with root priveleges could
    either kill that process or at least undo the lock.

    > The main platforms of Interest would be OpenBSD, Linux, Mac OS X, and
    > Solaris, but I would want to consider others as well, such as AIX, if
    > only to know if there was a lot of difference.


    I don't think its any different on any of these systems, sorry. But
    perhaps if you write what exactly you want to achieve someone may
    come up with an idea that is feasible...

    Regards, Jens
    --
    \ Jens Thoms Toerring ___ jt@toerring.de
    \__________________________ http://toerring.de

  2. Re: Behavior of flock() oriented procedures

    On Jun 2, 2:09 am, j...@toerring.de (Jens Thoms Toerring) wrote:
    > Aaron Hsu wrote:
    >> is there a standard behavior for what happens to the
    >> lock *after* I close the file descriptor? If I close the file
    >> descriptore without explicitly calling flock() with the unlock message,
    >> does this mean that the file will still have the lock on it, or does
    >> this mean that the file will automatically be unlocked?

    >
    > If the process that holds the lock closes the file the lock vanishes
    > without any explicit unlocking. All forms of locking work that way.


    .... apart from the ones that don't, eg: dotlock.

  3. Re: Behavior of flock() oriented procedures

    viza wrote:
    > On Jun 2, 2:09 am, j...@toerring.de (Jens Thoms Toerring) wrote:
    > > Aaron Hsu wrote:
    > >> is there a standard behavior for what happens to the
    > >> lock *after* I close the file descriptor? If I close the file
    > >> descriptore without explicitly calling flock() with the unlock message,
    > >> does this mean that the file will still have the lock on it, or does
    > >> this mean that the file will automatically be unlocked?

    > >
    > > If the process that holds the lock closes the file the lock vanishes
    > > without any explicit unlocking. All forms of locking work that way.


    > ... apart from the ones that don't, eg: dotlock.


    Ok, qualify that as "kernel managed" locking methods;-) Of course,
    if you implement youe own locking mechanism you can do a lot of
    things differently but they all will have a number of problems.
    The worst is probably that you can't sleep and get woken when a
    lock becomes available, you have to poll. And, of course, hardly
    any programs except the ones you wrote yourself will be aware
    that there's such a locking mechanism and in blissful ignorance
    will disregard it.
    Regards, Jens
    --
    \ Jens Thoms Toerring ___ jt@toerring.de
    \__________________________ http://toerring.de

  4. Re: Behavior of flock() oriented procedures

    On Jun 2, 6:55 am, arcf...@sacrideo.us (Aaron Hsu) wrote:
    > Jens Thoms Toerring wrote:
    >
    > I am trying to add an additional feature to Scheme's normal procedures
    > of OPEN-INPUT-FILE and family of CALL-WITH and WITH- procedures which
    > deal with files. The input/output datatype in Scheme is a port, but
    > unfortunately, a port does not explicitly have a unique file descriptor
    > available to implementations. I could use a file descriptor to make a
    > generic port on some Scheme implementations, but this isn't very nice
    > because it bypasses the native file ports support on implementations
    > that have faster file ports.
    >
    > If indeed I need to save the file descriptor to use it later, then I
    > think I have no choice but to return a file lock object which later is
    > used to "unlock" the object. This will unfortunately create two
    > connections to the file, and make for some nasty interactions if we are
    > cose to the maximum limit of opened files. I have run into this before,
    > so I know it can be a problem.


    The types of locking I have used are:

    fcntl, lockf - no good because the lock is lost when you close *any*
    descriptor for the file.

    flock - the lock is lost when you close *all* descriptors dup()ed from
    the one you obtained it for.

    dotlock (to lock /file create /file.lock) - persistent but slow -
    also if your program gets killed the lock remains forever, also, some
    programs ignore dotlocks that they *guess* are stale.

    cclient - (similar to dotlock, but create /tmp/.dev#.ino# and then
    flock it) - more or less like dotlock but used where you might not
    have write permission in the directories where you want to lock files.

    So if you want persistent locking you need to use a file system based
    one like dotlock - but these don't work if O_EXCL doesn't work, and
    are slower.

    If you are only trying to lock files so that other copies of your own
    program don't mess with them, then it might be best to implement your
    own locking method.

    Regarding your point about too many open files, does it help if your
    "lock object" is a dup()ed descriptor rather than an open()ed one?

    viza

  5. Re: Behavior of flock() oriented procedures

    Aaron Hsu wrote:
    > Jens Thoms Toerring wrote:
    > > I don't think its any different on any of these systems, sorry. But
    > > perhaps if you write what exactly you want to achieve someone may
    > > come up with an idea that is feasible...


    > I am trying to add an additional feature to Scheme's normal procedures
    > of OPEN-INPUT-FILE and family of CALL-WITH and WITH- procedures which
    > deal with files. The input/output datatype in Scheme is a port, but
    > unfortunately, a port does not explicitly have a unique file descriptor
    > available to implementations. I could use a file descriptor to make a
    > generic port on some Scheme implementations, but this isn't very nice
    > because it bypasses the native file ports support on implementations
    > that have faster file ports.


    I have to admit that I am rather ignorant also when it comes to
    Scheme and thus have a hard time understanding what you write...
    I am just wondering how you manage to obtain a lock if you don't
    have the file descriptor or at least don't want to use it. And is
    it a typical way in Scheme that files get closed when they are
    needed again later?

    > If indeed I need to save the file descriptor to use it later, then I
    > think I have no choice but to return a file lock object which later is
    > used to "unlock" the object. This will unfortunately create two
    > connections to the file, and make for some nasty interactions if we are
    > cose to the maximum limit of opened files. I have run into this before,
    > so I know it can be a problem.


    As far as I understand locking you can't keep a look on a file you
    closed. If it would be organized differently in a way that you keep
    the lock even on closed files until the process exits the burden
    would be shifted to the kernel. It would have to keep a list of all
    files your process closed while still holding a lock and take this
    list into account each time a file gets opened (by your or any other
    process. A lot of extra work for a rather uncommon (and potentially
    dangerous) scenario.
    Regards, Jens
    --
    \ Jens Thoms Toerring ___ jt@toerring.de
    \__________________________ http://toerring.de

  6. Re: Behavior of flock() oriented procedures

    viza wrote:

    > So if you want persistent locking you need to use a file system
    > based one like dotlock - but these don't work if O_EXCL doesn't
    > work, and are slower.


    For more reliable dotlocking than O_EXCL create a new file
    with a unique name, then link(2) the dot lock file. If link()
    succeeds, you're in, else check the link count on the new file
    with stat(2), if it's 2 you've got your lock too.

    With regard to stale locks: use a predefined timeout. If the
    process holding the lock doesn't update mtime on the lock
    file within the timeout other candidates may assume it's no
    longer valid.


    Regards,
    Marcel
    --
    printf -v email $(echo \ 155 141 162 143 145 154 155 141 162 \
    143 145 154 100 157 162 141 156 147 145 56 156 154 | tr \ \\)
    # O Herr, lass Hirn vom Himmel fallen! #

  7. Re: Behavior of flock() oriented procedures

    On Jun 2, 6:58 pm, Marcel Bruinsma wrote:
    > viza wrote:
    > > So if you want persistent locking you need to use a file system
    > > based one like dotlock - but these don't work if O_EXCL doesn't
    > > work, and are slower.

    >
    > For more reliable dotlocking than O_EXCL create a new file
    > with a unique name


    ....which is fundamentally impossible without O_EXCL...

    > then link(2) the dot lock file. If link()
    > succeeds, you're in, else check the link count on the new file
    > with stat(2), if it's 2 you've got your lock too.


    This is a absolutely fugly botch. Don't get me wrong - it decreases
    the chances of thinking you have a lock when you don't by a very large
    factor, but it most certainly does not make it impossible.

    > With regard to stale locks: use a predefined timeout. If the
    > process holding the lock doesn't update mtime on the lock
    > file within the timeout other candidates may assume it's no
    > longer valid.


    This is a million billion times worse. Never under any circumstances
    do this. Let the program send an email to root saying "I think that
    this lock might be stale but I can't possibly be sure". This should
    be very low volume on any working system, since locks shouldn't get
    left behind unless a program crashes.

    Unilaterally timing-out other processes locks is a recipe for
    disaster. How are processes supposed to be sure that they get
    scheduled in time to update it? request real-time priority? And even
    then, who decides what is an ok timeout?

    Really the only correct solution to this is to have open() always fail
    if O_EXCL is specified and cannot be guaranteed. This is required by
    POSIX, but some people think that it is acceptable to just greatly
    reduce the probability of trashing data.

    I know that what you've suggested is widely accepted, but it shouldn't
    be!

    viza

+ Reply to Thread