Is implementing a mutex in shared memory the best idea - SGI

This is a discussion on Is implementing a mutex in shared memory the best idea - SGI ; Hello, Briefly, here is a bug that I may be confronted with and told to spend some time on. There are several embedded "time critical" programs on different machines (SGI, and Linux [PC's and mainframes]) that communicate through a shared ...

+ Reply to Thread
Results 1 to 14 of 14

Thread: Is implementing a mutex in shared memory the best idea

  1. Is implementing a mutex in shared memory the best idea

    Hello,

    Briefly, here is a bug that I may be confronted with and told to
    spend some time on.

    There are several embedded "time critical" programs on different
    machines (SGI, and Linux [PC's and mainframes]) that communicate
    through a shared memory file to a server program The server is
    currently on a SGI system. But, any changes to the code should be
    accomplished, so that the source code is transportable to a Linux
    machine. All the processes run at the same speed, and were completely
    written by individuals where I work.

    Questions:

    (1) Explicitly, what instructions can I use to prevent all the
    clients from clobbering each other in their desire to use the single
    communication link? I.E: The big problem is sometimes these client
    programs overwrite the communication area at about the same time and
    thus the network hangs up! So, when the network becomes useless like
    this, I have to restart the server which is not a good thing since all
    previous data is gone.

    (2) Via a shared memory location (using the same shared memory file
    as above and memory mapping technique), could I define a mutex to guard
    the critical section of server code? Are there any problems in doing
    this (such as speed reduction), and explicitly how would I fix them? I
    have never implemented a mutex like this before.

    (3) Are there any other alternatives?

    (4) What are the advantages and disadvantages of any choice?

    (5) What is my best option?

    Help,
    Christopher Lusardi


  2. Re: Is implementing a mutex in shared memory the best idea

    In article <1139243236.509983.192710@f14g2000cwb.googlegroups. com>,
    wrote:
    > There are several embedded "time critical" programs on different
    >machines (SGI, and Linux [PC's and mainframes]) that communicate
    >through a shared memory file to a server program The server is
    >currently on a SGI system. But, any changes to the code should be
    >accomplished, so that the source code is transportable to a Linux
    >machine.


    > (1) Explicitly, what instructions can I use to prevent all the
    >clients from clobbering each other in their desire to use the single
    >communication link? I.E: The big problem is sometimes these client
    >programs overwrite the communication area at about the same time and
    >thus the network hangs up!


    It sounds to me as if your needs would be well served by a semaphore.
    semaphores have been around a long time, and should by now be
    implemented in any operating system that supports shared memory.
    ipcrm semop semctl semget -- or the posix versions that start
    with sem_* such as sem_init sem_wait


    > (2) Via a shared memory location (using the same shared memory file
    >as above and memory mapping technique), could I define a mutex to guard
    >the critical section of server code?


    If you are already using POSIX Threads (pthreads), then see
    pthread_mutex_init pthread_mutex_lock and kin.


    Perhaps I am reading too much into your words, but it seems to me
    that you were looking for a way to implement mutexes yourself,
    based upon some kind of code directly accessing one of the shared
    locations.

    Unfortunately, the C programming language does not provide atomic
    operations -- no "test and set" operation for example. The closest C
    comes to that is that it does provide sig_atomic_t as a type that can
    be safely accessed (i.e., read from) even in the presence of
    interrupts.

    If you chase down the various clauses in the C standard, about all you
    can do is use a volatile sig_atomic_t variable manipulated only within
    signal handlers that get invoked when you raise() signals... and even
    then, as C was not designed specifically to be thread-safe, you can't
    be sure that two different processors cannot enter the same signal
    handler code at the same time. Trying to implement mutexes via signal
    handlers is the best you can do in standard C, but it doesn't always
    work in practice -- and it is almost certain to be slow.

    So... you are better off finding a library function (perhaps such as
    the pthread mutexes) that is implemented by all of the operating
    systems that are of interest to you. If you are not using pthreads
    already, you will find semaphores to be lower coding overhead to add in.

  3. Re: Is implementing a mutex in shared memory the best idea


    Walter Roberson wrote:
    > In article <1139243236.509983.192710@f14g2000cwb.googlegroups. com>,
    > wrote:
    > > There are several embedded "time critical" programs on different
    > >machines (SGI, and Linux [PC's and mainframes]) that communicate
    > >through a shared memory file to a server program The server is
    > >currently on a SGI system. But, any changes to the code should be
    > >accomplished, so that the source code is transportable to a Linux
    > >machine.

    >
    > It sounds to me as if your needs would be well served by a semaphore.
    > semaphores have been around a long time, and should by now be
    > implemented in any operating system that supports shared memory.
    > ipcrm semop semctl semget -- or the posix versions that start
    > with sem_* such as sem_init sem_wait
    >


    First, I am not using threading at all.

    O'kay you may be helping to unconfuse me and I like your conversation!

    But, it surely can not be possible to use one semaphore and no "shared
    memory file" and
    no memory mapping across multible computers. My goal in life is to have
    each computer ask for service from another routine on a different
    computer or it may be on the same computer without each request being
    overwritten by other requests.

    As it stands now, I would create a shared memory file, and memory map a
    semaphore to that file. Different programs on different computers would
    do this in a very simliar fashion.

    Confused still,
    Christopher Lusardi


  4. Re: Is implementing a mutex in shared memory the best idea

    In article <1139413618.611698.93900@g14g2000cwa.googlegroups.c om>,
    clusardi2k@aol.com wrote:

    : As it stands now, I would create a shared memory file, and memory map a
    : semaphore to that file. Different programs on different computers would
    : do this in a very simliar fashion.

    Are you trying to use a shared memory zone across multiple computers? There's a
    number of very good reasons why that won't work.


    Cheers - Tony 'Nicoya' Mantler

    --
    Tony 'Nicoya' Mantler -- Master of Code-fu -- nicoya@ubb.ca
    -- http://nicoya.feline.pp.se/ -- http://www.ubb.ca/ --

  5. Re: Is implementing a mutex in shared memory the best idea


    Tony 'Nicoya' Mantler wrote:
    > In article <1139413618.611698.93900@g14g2000cwa.googlegroups.c om>,
    > clusardi2k@aol.com wrote:
    >
    > : As it stands now, I would create a shared memory file, and memory map a
    > : semaphore to that file. Different programs on different computers would
    > : do this in a very simliar fashion.
    >
    > Are you trying to use a shared memory zone across multiple computers? There's a
    > number of very good reasons why that won't work.


    The shared file is on a shared drive.

    Chris Lusardi


  6. Re: Is implementing a mutex in shared memory the best idea

    In article <1139504768.583152.133990@g14g2000cwa.googlegroups. com>,
    clusardi2k@aol.com writes:

    >The shared file is on a shared drive.


    NFS? Don't.
    I would change the whole approach and transfer the data by other means.


  7. Re: Is implementing a mutex in shared memory the best idea


    An answer, below, is an answer given with an attempt at a simple
    description! First, I already have one shared memory file which I use
    for communication! It's fine.

    Tor wrote:
    > In article <1139504768.583152.133990@g14g2000cwa.googlegroups. com>,
    > clusardi2k@aol.com writes:
    >
    > >The shared file is on a shared drive.

    >
    > NFS? Don't.
    > I would change the whole approach and transfer the data by other means.


    So no, I woun't get the o'kay to stop using my already existing shared
    memory file. For now, it's use the shared memory file on the shared
    disk as a means of communication between separate programs on different
    computers. I do this by memory mapping a C struct to that shared memory
    file.

    But, to install a new semaphore, I may have to create a new shared
    memory file on the shared drive and then memory map all the program's
    new internal semaphore to this shared memory file. All programs will do
    this so that the semaphore is viable across all computers. I currently
    am not using a semaphore so I have a clobbering problem that I want to
    fix with the new semaphore.

    In detail with maybe a simple useful pseudo program, how would you
    change the whole approach of using a semaphore across SGI, Linux (and
    later possibly Window) machines?

    Chris Lusardi


  8. Re: Is implementing a mutex in shared memory the best idea

    In article <1139504768.583152.133990@g14g2000cwa.googlegroups. com>,
    clusardi2k@aol.com wrote:

    : Tony 'Nicoya' Mantler wrote:
    : > In article <1139413618.611698.93900@g14g2000cwa.googlegroups.c om>,
    : > clusardi2k@aol.com wrote:
    : >
    : > : As it stands now, I would create a shared memory file, and memory map a
    : > : semaphore to that file. Different programs on different computers would
    : > : do this in a very simliar fashion.
    : >
    : > Are you trying to use a shared memory zone across multiple computers?
    : > There's a
    : > number of very good reasons why that won't work.
    :
    : The shared file is on a shared drive.

    There is really only one situation where you should be using a file on a shared
    drive to transfer data between cluster nodes. That is when you have large
    amounts of bulk data, and you're using a clustered file system on a SAN that's
    significantly faster than your network.

    HOWEVER.

    You should never attempt to use such a shared file as a primary communications
    channel. It should only be considered a side channel, to be used not unlike the
    "data" channel in an FTP connection.

    As an example:

    Cluster node A,B,C connect to server node X via TCP sockets. All cluster nodes
    share a cxfs file system mounted as /clusterdata.

    Node X fills the file "/clusterdata/NodeA-Raw" with raw data from some data
    source. Once filled, node X then sends a message over the TCP socket to node A
    telling it to read and process "/clusterdata/NodeA-Raw".

    Node A processes "/clusterdata/NodeA-Raw" into "/clusterdata/NodeA-Cooked". Once
    completed, node A sends a message over the TCP socket to node X telling it to
    read and integrate "/clusterdata/NodeA-Cooked", and to refill
    "/clusterdata/NodeA-Raw" with new data.

    Meanwhile, node X also fills "/clusterdata/NodeB-Raw", "/clusterdata/NodeC-Raw"
    with data, separately signals B and C via TCP to start processing, etc etc.

    You could also supplement this arrangement with *read only* shared data, like
    "/clusterdata/Shared-TransformMatrix", which would *only* be updated by Node X,
    and would *only* be updated when all nodes are idle; either by waiting until all
    nodes have processed and returned all current data, or by signalling through the
    TCP channel that the nodes should abort current processing and wait for new data.


    Now, if you don't have large amounts of bulk data or if your network is
    sufficiently fast to pass them that way, then I would strongly encourage you to
    use a network-only approach. I would also strongly suggest a network-only
    approach if you don't have a clustered FS SAN at all and are using a network
    filesystem like CIFS, SMB or NFS.

    If you need a high level of inter-node communication (like, if node calculations
    are interdependant on the results of other node calculations) then you'll need a
    different approach. You should post more details of the problem you're trying to
    calculate if this is the case.


    Cheers - Tony 'Nicoya' Mantler

    --
    Tony 'Nicoya' Mantler -- Master of Code-fu -- nicoya@ubb.ca
    -- http://nicoya.feline.pp.se/ -- http://www.ubb.ca/ --

  9. Re: Is implementing a mutex in shared memory the best idea


    Tony 'Nicoya' Mantler wrote:
    > Now, if you don't have large amounts of bulk data or if your network is
    > sufficiently fast to pass them that way, then I would strongly encourage you to
    > use a network-only approach. I would also strongly suggest a network-only
    > approach if you don't have a clustered FS SAN at all and are using a network
    > filesystem like CIFS, SMB or NFS.


    Do you have any simple implementations that you can refer me to?

    >
    > If you need a high level of inter-node communication (like, if node calculations
    > are interdependant on the results of other node calculations) then you'll need a
    > different approach. You should post more details of the problem you're trying to
    > calculate if this is the case.
    >


    For the foreseeable future, it appears that we will be keeping our
    original shared file on the shared drive to do communication between
    separate programs on different computers (SGI and Linux). But, to come
    up with a temporary fix for our clobbering problem, is it o'kay to
    create a new shared memory file on a shared drive and memory map
    everybody's semaphone to it? This new file will only be used for the
    semaphore. In simple English, please explain to me what specific
    problems will I run into?

    Christopher Lusardi


  10. Re: Is implementing a mutex in shared memory the best idea

    In article <1139584154.246953.309630@g44g2000cwa.googlegroups. com>,
    clusardi2k@aol.com wrote:

    : Tony 'Nicoya' Mantler wrote:
    : > Now, if you don't have large amounts of bulk data or if your network is
    : > sufficiently fast to pass them that way, then I would strongly encourage you to
    : > use a network-only approach. I would also strongly suggest a network-only
    : > approach if you don't have a clustered FS SAN at all and are using a network
    : > filesystem like CIFS, SMB or NFS.
    :
    : Do you have any simple implementations that you can refer me to?

    Most people use MPI or something similar.


    : For the foreseeable future, it appears that we will be keeping our
    : original shared file on the shared drive to do communication between
    : separate programs on different computers (SGI and Linux). But, to come
    : up with a temporary fix for our clobbering problem, is it o'kay to
    : create a new shared memory file on a shared drive and memory map
    : everybody's semaphone to it? This new file will only be used for the
    : semaphore. In simple English, please explain to me what specific
    : problems will I run into?

    The problem you'll run into is that semaphores are not guaranteed to work
    between multiple systems. In fact, mmap() isn't guaranteed to be coherent across
    systems either without careful use of msync() and some sort of mutex-like
    exclusion to prevent multiple writers.

    What you are asking to do is simply not viable nor advisable.


    Cheers - Tony 'Nicoya' Mantler

    --
    Tony 'Nicoya' Mantler -- Master of Code-fu -- nicoya@ubb.ca
    -- http://nicoya.feline.pp.se/ -- http://www.ubb.ca/ --

  11. Re: Is implementing a mutex in shared memory the best idea

    In <1139584154.246953.309630@g44g2000cwa.googlegroups. com>, on
    02/10/2006
    at 07:09 AM, clusardi2k@aol.com said:

    >For the foreseeable future, it appears that we will be keeping our
    >original shared file on the shared drive to do communication between
    >separate programs on different computers (SGI and Linux). But, to
    >come up with a temporary fix for our clobbering problem, is it o'kay
    >to create a new shared memory file on a shared drive and memory map
    >everybody's semaphone to it?


    If the file is mapped into different machines then it is *NOT* shared
    memory and you need to worry about synchronization. But it's not my
    dog.

    --
    Shmuel (Seymour J.) Metz, SysProg and JOAT

    Unsolicited bulk E-mail subject to legal action. I reserve the
    right to publicly post or ridicule any abusive E-mail. Reply to
    domain Patriot dot net user shmuel+news to contact me. Do not
    reply to spamtrap@library.lspace.org


  12. Re: Is implementing a mutex in shared memory the best idea

    In article <1139584154.246953.309630@g44g2000cwa.googlegroups. com>,
    wrote:

    >For the foreseeable future, it appears that we will be keeping our
    >original shared file on the shared drive to do communication between
    >separate programs on different computers (SGI and Linux). But, to come
    >up with a temporary fix for our clobbering problem, is it o'kay to
    >create a new shared memory file on a shared drive and memory map
    >everybody's semaphone to it? This new file will only be used for the
    >semaphore.


    You latched on to my mention of semaphores a bit too soon ;-)
    I did not realize at the time that your "shared memory" consisted
    of a common nfs mounted filesystem.

    If you have a shared file and you want to ensure that only one
    process at a time is writing to it, then you should use rpc.lockd .
    You could just request locking the entire file, or if you can
    divide your work more finely, you could lock particular records.

    If for some reason you cannot use rpc.lockd then you can fake
    a lock by using open() with O_CREAT|O_EXCL in the option and
    naming a file in a shared nfs directory. If another process already
    has the file open then the open() will fail, so your process could
    sleep and try again later. If the open() succeeds then you own
    the lock and can go ahead and write to the shared mmap()'d file.

    The make-shift lock will persist as long as the file exists with
    that name. If the writing process is somehow able to detect
    that it is now safe for other processes to write to the shared
    area, then the writing process could be the one that removes
    the file. It might, though, turn out to make more sense to
    have the reading server process remove the lock file when it has finished
    processing the data. You will, though, need to find some way for
    the writing process to signal to the reading process that it has
    finished putting in the request.

  13. Re: Is implementing a mutex in shared memory the best idea


    Walter Roberson wrote:
    > In article <1139584154.246953.309630@g44g2000cwa.googlegroups. com>,
    > wrote:
    >
    > >For the foreseeable future, it appears that we will be keeping our
    > >original shared file on the shared drive to do communication between
    > >separate programs on different computers (SGI and Linux). But, to come
    > >up with a temporary fix for our clobbering problem, is it o'kay to
    > >create a new shared memory file on a shared drive and memory map
    > >everybody's semaphone to it? This new file will only be used for the
    > >semaphore.

    >
    > If you have a shared file and you want to ensure that only one
    > process at a time is writing to it, then you should use rpc.lockd .
    > You could just request locking the entire file, or if you can
    > divide your work more finely, you could lock particular records.
    >


    So, to clarify, rpc.lockd will allow different programs on different
    computers (Linux, SGI, and Windows) to an equivalent of a semaphore
    (mutex specifically)?

    Are there any better alternatives to rpc.lockd?

    Thanks,
    Christopher Lusardi


  14. Re: Is implementing a mutex in shared memory the best idea


    Walter Roberson wrote:
    > In article <1139584154.246953.309630@g44g2000cwa.googlegroups. com>,
    > wrote:
    >
    > >For the foreseeable future, it appears that we will be keeping our
    > >original shared file on the shared drive to do communication between
    > >separate programs on different computers (SGI and Linux). But, to come
    > >up with a temporary fix for our clobbering problem, is it o'kay to
    > >create a new shared memory file on a shared drive and memory map
    > >everybody's semaphone to it? This new file will only be used for the
    > >semaphore.

    >
    > If you have a shared file and you want to ensure that only one
    > process at a time is writing to it, then you should use rpc.lockd .
    > You could just request locking the entire file, or if you can
    > divide your work more finely, you could lock particular records.



    So, to clarify, rpc.lockd will allow different programs on different
    computers (Linux, SGI, and Windows) to do an equivalent of a semaphore
    (mutex specifically)? Where one program can lock a file and other
    programs on different computers would wait until the file becomes
    unlocked. Only when the file becomes unlocked will a program continue.

    Questions: With rpc.lockd is it possible to have a timeout on the
    amount of waiting before the file becomes unlocked. For example, if a
    program doesn't have to have the locked file then it can give up trying
    to access it.

    Are there any better alternatives to rpc.lockd?


    Thanks,
    Christopher Lusardi


+ Reply to Thread