help me solve a problem in converting a non-parallel program into parallel program... - Linux

This is a discussion on help me solve a problem in converting a non-parallel program into parallel program... - Linux ; Hi all, I need your help. I have a Matlab program, which uses the Matlab symbolic toolbox and does exclusively symbolic computations. It calls Maple's library internally. My program is currently non-parallel. I need to run it on a super ...

+ Reply to Thread
Results 1 to 16 of 16

Thread: help me solve a problem in converting a non-parallel program into parallel program...

  1. help me solve a problem in converting a non-parallel program into parallel program...

    Hi all,

    I need your help.

    I have a Matlab program, which uses the Matlab symbolic toolbox and does
    exclusively symbolic computations. It calls Maple's library internally. My
    program is currently non-parallel. I need to run it on a super computer. The
    super computer does not have Matlab and it is Linux based and supports MPI.

    I didn't figure out how to convert non-parallel program into parallel
    exectuables on Linux with MPI.

    Since I failed to get a nice solution of the above problem, now I am willing
    to make a comprise, I am going to make it coarse-grained distrubuted
    computing:

    (1) It is easy to compile Matlab non-parallel program into non-parallel
    exectuables;
    (2) I am willing to launch 100 independent copies fo these exectuables on
    100 CPUs;

    Is there a mechanism in Linux, or MPI, or third party applications that can
    facilitate luanching of such 100 programs and finally automatically collect
    back the 100 results from these 100 runs?

    I am thinking this is too hard to do well by me myself manually. I need some
    kind of automation to manage this whole thing.

    Please shed some lights on me! Thanks a lot!





  2. Re: help me solve a problem in converting a non-parallel programinto parallel program...

    Mike wrote:
    > I have a Matlab program, which uses the Matlab symbolic toolbox and does
    > exclusively symbolic computations. It calls Maple's library internally. My
    > program is currently non-parallel. I need to run it on a super computer. The
    > super computer does not have Matlab and it is Linux based and supports MPI.
    >
    > I didn't figure out how to convert non-parallel program into parallel
    > exectuables on Linux with MPI.
    >
    > Since I failed to get a nice solution of the above problem, now I am willing
    > to make a comprise, I am going to make it coarse-grained distrubuted
    > computing:
    >
    > (1) It is easy to compile Matlab non-parallel program into non-parallel
    > exectuables;
    > (2) I am willing to launch 100 independent copies fo these exectuables on
    > 100 CPUs;
    >
    > Is there a mechanism in Linux, or MPI, or third party applications that can
    > facilitate luanching of such 100 programs


    Yes, it's called a loop, and the shell can do it. As in:

    for i in 1 2 3 4 5 6 7 8 9
    do
    ./myprogram $i &
    done

    Or:

    while read line
    do
    ./myprogram $line &
    done < my_file_with_parameters_for_each_desired_instance

    > and finally automatically collect
    > back the 100 results from these 100 runs?


    Sure, collecting the results from 100 programs is easy. If they all
    write everything to standard output, you'd just do something like
    this:

    while read outputfile param1 param2
    do
    ./myprogram $param1 $param2 > $outputfile &
    done < my_file_with_output_filenames_and_parameters

    # now wait for them *all* to finish
    wait

    echo "all of the programs are done now!"

    Of course, you can put all that in a file and have the shell run it
    as a script so you don't have to type it all interactively (and so
    you can do it again if you make an error or something).

    - Logan

  3. Re: help me solve a problem in converting a non-parallel program into parallel program...


    "Logan Shaw" wrote in message
    news:45a090dc$0$5175$4c368faf@roadrunner.com...
    > Mike wrote:
    >> I have a Matlab program, which uses the Matlab symbolic toolbox and does
    >> exclusively symbolic computations. It calls Maple's library internally.
    >> My
    >> program is currently non-parallel. I need to run it on a super computer.
    >> The
    >> super computer does not have Matlab and it is Linux based and supports
    >> MPI.
    >>
    >> I didn't figure out how to convert non-parallel program into parallel
    >> exectuables on Linux with MPI.
    >>
    >> Since I failed to get a nice solution of the above problem, now I am
    >> willing
    >> to make a comprise, I am going to make it coarse-grained distrubuted
    >> computing:
    >>
    >> (1) It is easy to compile Matlab non-parallel program into non-parallel
    >> exectuables;
    >> (2) I am willing to launch 100 independent copies fo these exectuables on
    >> 100 CPUs;
    >>
    >> Is there a mechanism in Linux, or MPI, or third party applications that
    >> can
    >> facilitate luanching of such 100 programs

    >
    > Yes, it's called a loop, and the shell can do it. As in:
    >
    > for i in 1 2 3 4 5 6 7 8 9
    > do
    > ./myprogram $i &
    > done
    >
    > Or:
    >
    > while read line
    > do
    > ./myprogram $line &
    > done < my_file_with_parameters_for_each_desired_instance
    >
    >> and finally automatically collect
    >> back the 100 results from these 100 runs?

    >
    > Sure, collecting the results from 100 programs is easy. If they all
    > write everything to standard output, you'd just do something like
    > this:
    >
    > while read outputfile param1 param2
    > do
    > ./myprogram $param1 $param2 > $outputfile &
    > done < my_file_with_output_filenames_and_parameters
    >
    > # now wait for them *all* to finish
    > wait
    >
    > echo "all of the programs are done now!"
    >
    > Of course, you can put all that in a file and have the shell run it
    > as a script so you don't have to type it all interactively (and so
    > you can do it again if you make an error or something).
    >
    > - Logan


    Logan, thanks for your help!

    Do you mean I just need to launch the 100 programs using a loop and then the
    supercomputer will automatically dispatch the jobs to 100 CPUs?

    Your shell script looks serial to me... (as you see, I've got no previous
    experience with parallel computing...)

    Thanks!



  4. Re: help me solve a problem in converting a non-parallel program into parallel program...

    "Mike" writes:

    > "Logan Shaw" wrote in message
    > news:45a090dc$0$5175$4c368faf@roadrunner.com...
    >> Mike wrote:
    >>> I have a Matlab program, which uses the Matlab symbolic toolbox and does
    >>> exclusively symbolic computations. It calls Maple's library internally.
    >>> My
    >>> program is currently non-parallel. I need to run it on a super computer.
    >>> The
    >>> super computer does not have Matlab and it is Linux based and supports
    >>> MPI.
    >>>
    >>> I didn't figure out how to convert non-parallel program into parallel
    >>> exectuables on Linux with MPI.
    >>>
    >>> Since I failed to get a nice solution of the above problem, now I am
    >>> willing
    >>> to make a comprise, I am going to make it coarse-grained distrubuted
    >>> computing:
    >>>
    >>> (1) It is easy to compile Matlab non-parallel program into non-parallel
    >>> exectuables;
    >>> (2) I am willing to launch 100 independent copies fo these exectuables on
    >>> 100 CPUs;
    >>>
    >>> Is there a mechanism in Linux, or MPI, or third party applications that
    >>> can
    >>> facilitate luanching of such 100 programs

    >>
    >> Yes, it's called a loop, and the shell can do it. As in:
    >>
    >> for i in 1 2 3 4 5 6 7 8 9
    >> do
    >> ./myprogram $i &
    >> done
    >>
    >> Or:
    >>
    >> while read line
    >> do
    >> ./myprogram $line &
    >> done < my_file_with_parameters_for_each_desired_instance
    >>
    >>> and finally automatically collect
    >>> back the 100 results from these 100 runs?

    >>
    >> Sure, collecting the results from 100 programs is easy. If they all
    >> write everything to standard output, you'd just do something like
    >> this:
    >>
    >> while read outputfile param1 param2
    >> do
    >> ./myprogram $param1 $param2 > $outputfile &
    >> done < my_file_with_output_filenames_and_parameters
    >>
    >> # now wait for them *all* to finish
    >> wait
    >>
    >> echo "all of the programs are done now!"
    >>
    >> Of course, you can put all that in a file and have the shell run it
    >> as a script so you don't have to type it all interactively (and so
    >> you can do it again if you make an error or something).
    >>
    >> - Logan

    >
    > Logan, thanks for your help!
    >
    > Do you mean I just need to launch the 100 programs using a loop and then the
    > supercomputer will automatically dispatch the jobs to 100 CPUs?
    >
    > Your shell script looks serial to me... (as you see, I've got no previous
    > experience with parallel computing...)


    the '&' operator starts a process in the background and the shell
    continues without waiting it to return.

    look at these examples:

    phix@igloo:~> { sleep 10 ; echo I am done } &
    [1] 10674
    phix@igloo:~> echo test
    test
    phix@igloo:~> I am done

    [1] + done {; sleep 10; echo I am done; }
    phix@igloo:~>
    phix@igloo:~> { { sleep 5; echo last } & echo first } > out ; wait ; cat out
    [2] 10710
    [2] + done {; sleep 5; echo later; }
    first
    last

    --
    Philippe Amarenco, aka Phix
    epita 2007 - EpX - ex-{GISTR,ACU,LSE}

  5. Re: help me solve a problem in converting a non-parallel program into parallel program...


    "Mike д
    "
    > Hi all,
    >
    > I need your help.
    >
    > I have a Matlab program, which uses the Matlab symbolic toolbox and does
    > exclusively symbolic computations. It calls Maple's library internally. My
    > program is currently non-parallel. I need to run it on a super computer. The
    > super computer does not have Matlab and it is Linux based and supports MPI.
    >
    > I didn't figure out how to convert non-parallel program into parallel
    > exectuables on Linux with MPI.
    >
    > Since I failed to get a nice solution of the above problem, now I am willing
    > to make a comprise, I am going to make it coarse-grained distrubuted
    > computing:
    >
    > (1) It is easy to compile Matlab non-parallel program into non-parallel
    > exectuables;
    > (2) I am willing to launch 100 independent copies fo these exectuables on
    > 100 CPUs;
    >
    > Is there a mechanism in Linux, or MPI, or third party applications that can
    > facilitate luanching of such 100 programs and finally automatically collect
    > back the 100 results from these 100 runs?
    >
    > I am thinking this is too hard to do well by me myself manually. I need some
    > kind of automation to manage this whole thing.
    >
    > Please shed some lights on me! Thanks a lot!



    Hi,try this way:
    oh,I'm not an English native speaker,so please ignore the spelling
    errors if they occur
    1,you must find a proper way to split your problem into small pieces
    (it has nothing to do with MPI. This is a key step and it's just a math
    problem)
    2, rewrite your matlab programm, make it can work with some parameters
    to solve a part of your problem...(the parameters it received decide
    which part it should to compute) I mean, after this step ,you should be
    able to solve the problem by repeated calling the programme,each time
    with a different parameter.
    3. Compile your matlab programme into .EXE file. There are several ways
    to do that. Maybe LCC can work.
    4.Write a wrapper tool for your programme.The wrapper should use MPI to
    interact with each other,and make it call your .EXE programme with
    parameters it received from the center control node.
    5. Dispatch your wrappers and .EXE programmes

    P.S MPI itself can dispatch and collect back results perfectly.No
    third party application is needed.Google MPI_Broadcast and MPI_Wait
    for more.


  6. Re: help me solve a problem in converting a non-parallel programinto parallel program...

    PiggestPig wrote:
    > "Mike д
    > "
    >> Hi all,
    >>
    >> I need your help.
    >>
    >> I have a Matlab program, which uses the Matlab symbolic toolbox and does
    >> exclusively symbolic computations. It calls Maple's library internally. My
    >> program is currently non-parallel. I need to run it on a super computer. The
    >> super computer does not have Matlab and it is Linux based and supports MPI.
    >>
    >> I didn't figure out how to convert non-parallel program into parallel
    >> exectuables on Linux with MPI.
    >>
    >> Since I failed to get a nice solution of the above problem, now I am willing
    >> to make a comprise, I am going to make it coarse-grained distrubuted
    >> computing:
    >>
    >> (1) It is easy to compile Matlab non-parallel program into non-parallel
    >> exectuables;
    >> (2) I am willing to launch 100 independent copies fo these exectuables on
    >> 100 CPUs;
    >>
    >> Is there a mechanism in Linux, or MPI, or third party applications that can
    >> facilitate luanching of such 100 programs and finally automatically collect
    >> back the 100 results from these 100 runs?
    >>
    >> I am thinking this is too hard to do well by me myself manually. I need some
    >> kind of automation to manage this whole thing.
    >>
    >> Please shed some lights on me! Thanks a lot!

    >
    >
    > Hi,try this way:
    > oh,I'm not an English native speaker,so please ignore the spelling
    > errors if they occur
    > 1,you must find a proper way to split your problem into small pieces
    > (it has nothing to do with MPI. This is a key step and it's just a math
    > problem)
    > 2, rewrite your matlab programm, make it can work with some parameters
    > to solve a part of your problem...(the parameters it received decide
    > which part it should to compute) I mean, after this step ,you should be
    > able to solve the problem by repeated calling the programme,each time
    > with a different parameter.
    > 3. Compile your matlab programme into .EXE file. There are several ways
    > to do that. Maybe LCC can work.
    > 4.Write a wrapper tool for your programme.The wrapper should use MPI to
    > interact with each other,and make it call your .EXE programme with
    > parameters it received from the center control node.
    > 5. Dispatch your wrappers and .EXE programmes
    >
    > P.S MPI itself can dispatch and collect back results perfectly.No
    > third party application is needed.Google MPI_Broadcast and MPI_Wait
    > for more.
    >

    Of course, .EXE programs do not run in Linux.

    --
    .~. Jean-David Beyer Registered Linux User 85642.
    /V\ PGP-Key: 9A2FC99A Registered Machine 241939.
    /( )\ Shrewsbury, New Jersey http://counter.li.org
    ^^-^^ 08:45:01 up 78 days, 11:18, 3 users, load average: 4.36, 4.36, 4.32

  7. Re: help me solve a problem in converting a non-parallel program into parallel program...

    On Sat, 6 Jan 2007 21:33:27 -0800, "Mike" wrote:

    >I have a Matlab program, which uses the Matlab symbolic toolbox and does
    >exclusively symbolic computations. It calls Maple's library internally. My
    >program is currently non-parallel. I need to run it on a super computer. The
    >super computer does not have Matlab and it is Linux based and supports MPI.
    >
    >I didn't figure out how to convert non-parallel program into parallel
    >exectuables on Linux with MPI.


    >Is there a mechanism in Linux, or MPI, or third party applications that can
    >facilitate luanching of such 100 programs and finally automatically collect
    >back the 100 results from these 100 runs?
    >
    >I am thinking this is too hard to do well by me myself manually. I need some
    >kind of automation to manage this whole thing.
    >
    >Please shed some lights on me! Thanks a lot!


    You are probably looking for IPC, InterProcessCommunication.

    Check out Chapter 5 in
    http://www.advancedlinuxprogramming.com/alp-folder

    Personally I would do it with Perl, and Perl's IPC modules,
    like IPC::Open3
    Perl makes it easy to setup socket connections, the IPC, etc.
    and combine them into a single program.

    One of the things to consider is whether you can collect results
    all at the end-of-run in a results-file, or whether you need to collect
    the results in real-time as they are returned. Possibly you need an even
    more complex system, where results are returned in realtime to the
    collecting system, decisions are made, and new commands are sent out
    to the clients, so they can alter their computations.

    In other words, it may be a bit more complex than shell scripts can
    cleanly handle, and you might want to look into Perl, Python, etc.

    I would ask this question at http://perlmonks.org for some advice
    from a wide variety of systems programmers, it's free, and probably
    would set you on the right track.




    --
    I'm not really a human, but I play one on earth.
    http://zentara.net/japh.html

  8. Re: help me solve a problem in converting a non-parallel programinto parallel program...

    Mike wrote:
    > "Logan Shaw" wrote in message
    > news:45a090dc$0$5175$4c368faf@roadrunner.com...
    >> Mike wrote:


    >>> Is there a mechanism in Linux, or MPI, or third party applications that
    >>> can
    >>> facilitate luanching of such 100 programs


    >> Yes, it's called a loop, and the shell can do it. As in:
    >>
    >> for i in 1 2 3 4 5 6 7 8 9
    >> do
    >> ./myprogram $i &
    >> done


    > Do you mean I just need to launch the 100 programs using a loop and then the
    > supercomputer will automatically dispatch the jobs to 100 CPUs?


    Good point. Obviously, it will depend on the specifics of the computer.

    But in most cases, it probably will be good enough. There is no guarantee
    that the job will get a fixed assignment to one of the 100 CPUs, but it
    very well may not matter if a job moves between one CPU and another.
    The scheduler makes this decision for you.

    Some (most?) schedulers in multiprocessor systems have a "processor
    affinity" feature where they note which processor a job most recently
    ran on and prefer to schedule it on that processor again in the future.
    That does not create a fixed binding, but it does reduce the (already
    fairly small) overhead of switching CPUs a bit.

    The real answer to this question depends on what type of multiprocessor
    architecture that system has.

    Is it:

    (1) an SMP machine (SMP stands for symmetric multiprocessor and basically
    means a machine where all processors are identical and all share the
    same physical memory; with SMP, there is very little penalty for
    switching around between processors)?
    (2) a NUMA (NUMA is like SMP except that some processors have local memory
    they can access faster than others; this is harder to program for
    since it's very important which processor is running what, but it
    can get higher performance)?
    (3) actually not truly a single multiprocessor machine but instead a
    cluster of completely separate machines (each with their own
    process, memory, and instance of the operating system) connected
    together by a network, i.e. a Beowulf cluster?
    (4) something else?

    The right thing to do will depend on the type of machine. In fact,
    if it is a cluster of machines, then launching all the jobs locally
    with a simple loop is probably precisely the *wrong* thing to do.
    I looked up MPI, and it seems there are MPI implementations that can
    support different architectures, so the fact that you mentioned MPI
    doesn't really seem to imply what kind of architecture you have.

    So, I suggest you should talk to the support staff for the computer
    you will be using. They probably have a web page that you can use
    to read up about how to do all this, and they are surely used to
    questions from people in exactly your position, i.e. people who've
    never run anything on a 100-processor system before, which would
    be the majority of the people in the world. :-)

    - Logan

  9. Re: help me solve a problem in converting a non-parallel program into parallel program...


    "Mike" wrote in message
    news:enq0n6$af8$1@news.Stanford.EDU...
    > Hi all,
    >
    > I need your help.
    >
    > I have a Matlab program, which uses the Matlab symbolic toolbox and does
    > exclusively symbolic computations. It calls Maple's library internally. My
    > program is currently non-parallel. I need to run it on a super computer.
    > The
    > super computer does not have Matlab and it is Linux based and supports
    > MPI.
    >
    > I didn't figure out how to convert non-parallel program into parallel
    > exectuables on Linux with MPI.
    >
    > Since I failed to get a nice solution of the above problem, now I am
    > willing
    > to make a comprise, I am going to make it coarse-grained distrubuted
    > computing:
    >
    > (1) It is easy to compile Matlab non-parallel program into non-parallel
    > exectuables;
    > (2) I am willing to launch 100 independent copies fo these exectuables on
    > 100 CPUs;
    >
    > Is there a mechanism in Linux, or MPI, or third party applications that
    > can
    > facilitate luanching of such 100 programs and finally automatically
    > collect
    > back the 100 results from these 100 runs?
    >
    > I am thinking this is too hard to do well by me myself manually. I need
    > some
    > kind of automation to manage this whole thing.
    >
    > Please shed some lights on me! Thanks a lot!
    >

    Check my werbsite for the MPI tutorial link to learn about MPI itself.
    Basically there are only 6 functions you need to know.

    http://www.personal.leeds.ac.uk/~bgy1mm


    Under MPI, each program is identical, but they send messages to each other,
    so the data and the execution statye can be different on each node.
    Generally, however, all nodes will communicate at the same point.

    So your algorithm is

    lauch program with all problems stored on node 0
    divide problem into 100 chunks, send data to all nodes.
    calculate chunk on each node
    send solutions back to node 0
    print out result.

    step 2 may be easy or difficult, depending on your problem.

    The problem is that the code is in Matlab whilst MPI provides C and Fortran.
    The easiest solution, conceptually, is to translate the code from Matlab to
    C. I am sure there will be some tool to enable you to this without rewriting
    everything manually, however it is likely to be difficult to use.



  10. Re: help me solve a problem in converting a non-parallel program into parallel program...

    On Sun, 07 Jan 2007 06:33:27 +0100, Mike wrote:

    > Hi all,
    >
    > I need your help.
    >
    > I have a Matlab program, which uses the Matlab symbolic toolbox and does
    > exclusively symbolic computations.


    Did you consider to use "Matlab Distributed Computing Toolbox 3.0"?

    --
    This signature intentionally left almost blank.

  11. Re: help me solve a problem in converting a non-parallel program into parallel program...


    "Malcolm McLean" wrote in message
    news:v-CdnQd0-KhxsTzYnZ2dnUVZ8tqqnZ2d@bt.com...
    >
    > "Mike" wrote in message
    > news:enq0n6$af8$1@news.Stanford.EDU...
    >> Hi all,
    >>
    >> I need your help.
    >>
    >> I have a Matlab program, which uses the Matlab symbolic toolbox and does
    >> exclusively symbolic computations. It calls Maple's library internally.
    >> My
    >> program is currently non-parallel. I need to run it on a super computer.
    >> The
    >> super computer does not have Matlab and it is Linux based and supports
    >> MPI.
    >>
    >> I didn't figure out how to convert non-parallel program into parallel
    >> exectuables on Linux with MPI.
    >>
    >> Since I failed to get a nice solution of the above problem, now I am
    >> willing
    >> to make a comprise, I am going to make it coarse-grained distrubuted
    >> computing:
    >>
    >> (1) It is easy to compile Matlab non-parallel program into non-parallel
    >> exectuables;
    >> (2) I am willing to launch 100 independent copies fo these exectuables on
    >> 100 CPUs;
    >>
    >> Is there a mechanism in Linux, or MPI, or third party applications that
    >> can
    >> facilitate luanching of such 100 programs and finally automatically
    >> collect
    >> back the 100 results from these 100 runs?
    >>
    >> I am thinking this is too hard to do well by me myself manually. I need
    >> some
    >> kind of automation to manage this whole thing.
    >>
    >> Please shed some lights on me! Thanks a lot!
    >>

    > Check my werbsite for the MPI tutorial link to learn about MPI itself.
    > Basically there are only 6 functions you need to know.
    >
    > http://www.personal.leeds.ac.uk/~bgy1mm
    >



    I like your suggestion - learning 6 functions is good. But in your tutorial
    you said MPI doesnot work with C++?

    >
    > Under MPI, each program is identical, but they send messages to each
    > other, so the data and the execution statye can be different on each node.
    > Generally, however, all nodes will communicate at the same point.
    >
    > So your algorithm is
    >
    > lauch program with all problems stored on node 0
    > divide problem into 100 chunks, send data to all nodes.
    > calculate chunk on each node
    > send solutions back to node 0
    > print out result.
    >
    > step 2 may be easy or difficult, depending on your problem.
    >


    I don't have to divide the program. Matlab allows me to compile a
    non-parallel Matlab program into non-parallel c program and executables.

    So I need to dispatch 100 copy of such executables and hopefully they will
    run on 100 cpus simultaneously.



  12. Re: help me solve a problem in converting a non-parallel program into parallel program...


    "Georg Bisseling" wrote in message
    newsp.tlsxs8h4z4pjzz@donner.fini...
    > On Sun, 07 Jan 2007 06:33:27 +0100, Mike wrote:
    >
    >> Hi all,
    >>
    >> I need your help.
    >>
    >> I have a Matlab program, which uses the Matlab symbolic toolbox and does
    >> exclusively symbolic computations.

    >
    > Did you consider to use "Matlab Distributed Computing Toolbox 3.0"?
    >


    Did you see I said the super-computer does not have Matlab installed?



  13. Re: help me solve a problem in converting a non-parallel program into parallel program...

    On Sun, 07 Jan 2007 22:03:44 +0100, Mike wrote:

    >
    > "Georg Bisseling" wrote in message
    > newsp.tlsxs8h4z4pjzz@donner.fini...
    >> On Sun, 07 Jan 2007 06:33:27 +0100, Mike wrote:
    >>
    >>> Hi all,
    >>>
    >>> I need your help.
    >>>
    >>> I have a Matlab program, which uses the Matlab symbolic toolbox and does
    >>> exclusively symbolic computations.

    >>
    >> Did you consider to use "Matlab Distributed Computing Toolbox 3.0"?
    >>

    >
    > Did you see I said the super-computer does not have Matlab installed?


    That could be changed, couldn't it? Maybe you can bring your own Matlab
    in libraries?

    I interpret your "answer" as "No, I did not yet consider using it.".

    If you see no way to use the parallelization features that come with
    the programming environment of your choice then it can get very hard.

    So let me ask me another question:

    At the level of control that you currently have in your program, could you
    split the problem in about 100 nearly equaly time consuming parts?
    If the answer is yes then you could really let run 100 instances of your
    program via sh/rsh/ssh (whatever suits the given super computer).

    If the answer is no, then the question becomes: can you convert the program
    to get more control? Answering this question is very tough without knowing
    the details. If you do not provide the details, nobody can help.
    Did you try to consult some newsgroups that are centered around Matlab like
    comp.soft-sys.matlab or cern.matlab?

    --
    This signature intentionally left almost blank.

  14. Re: help me solve a problem in converting a non-parallel program into parallel program...


    "Mike" wrote
    >> lauch program with all problems stored on node 0
    >> divide problem into 100 chunks, send data to all nodes.
    >> calculate chunk on each node
    >> send solutions back to node 0
    >> print out result.
    >>
    >> step 2 may be easy or difficult, depending on your problem.
    >>

    >
    > I don't have to divide the program. Matlab allows me to compile a
    > non-parallel Matlab program into non-parallel c program and executables.
    >
    > So I need to dispatch 100 copy of such executables and hopefully they will
    > run on 100 cpus simultaneously.

    You've got to divide the problem, not the program.
    If matlab will spit out a C program that will take as a parameter 1/100 th
    of your problem and return 1/100th of the answer, then you've got it made.
    Simply compile the C under MPI, then in your main function write a little
    bit of code to send 1/100th of the problem to each node, collect the
    answers, and probaly you'll need a bit of code to collate them.

    Eg you want to calcuate an average of 10000 numbers in parallel.

    matlab spits out

    double sum(double *x, int N)
    {
    int i;
    doubel answer = 0;
    for(i=0;i answer += x[i];
    return answer;
    }

    pur MPI program goes something like

    #include "mpi.h"

    int main(int argc, char **argv)
    {
    double array[10000];
    int rank;
    MPI_Init(argc, argv);
    MPI_Rank(MPI_COMM_WORLD, &rank);
    readinarray(array);
    if(rank == 0)
    {
    for(i=1;i<100;i++)
    /* haven't got the parameters quite right */
    MPI_Send(array + i * 100, MPI_DOUBLE, 100, i, MPI_COMM_WORLD):
    }
    else
    {
    MPI_Recv(array, MPI_DOUBLE,100, 0, MPI_COMM_WORLD);
    }
    /* calcuate a partial answer */
    answer = sum(array, 100);
    if(rank != 0)
    MPI_Send(&answer, MPI_DUBLE, 1, 0, MPI_COMM_WORLD):
    else
    {
    double total = answer;
    double partial;
    for(i=1;i<100;i++)
    {
    MPI_Recv(&partial, MPI_DOUBLE, 1, i, MPI_COMM_WORLD);
    total += partial;
    }
    printf("average is %g\n", total/10000);
    }

    MPI_Finalize();
    }

    This is only untested code, but it gives you the general idea. An average is
    extremely easy to break up. Other functions might be a lot harder.



  15. Re: help me solve a problem in converting a non-parallel program into parallel program...

    Mike wrote:

    > I don't have to divide the program. Matlab allows me to compile a
    > non-parallel Matlab program into non-parallel c program and
    > executables.
    >


    Note that (a) Matlab is available for linux and (b) GNU Octave
    is available for linux, if your code can run under octave (okay probably
    not true in your case given maple usage, but hey), it can
    help skirt the licensing issues that can crop up running
    closed-source stuff like matlab on large clusters.

    > So I need to dispatch 100 copy of such executables and hopefully they
    > will run on 100 cpus simultaneously.


    There are a variety of systems that both real supercomputers and
    ordinary linux clusters run that manage such things (as previously
    hinted by other posters).

    *** You'll just have to *ask the local systems guy* what they're running
    (or read the local documentation), if anything. What you want to do
    sounds like a typical use case for a mid-range linux batch cluster to
    munch through, though - Don't be too proud to just ask the guys running
    the system you have access to, rather than random folk on usenet, they
    probably know the system in question far better than anyone out here!

    [Following list is non-exhaustive]

    (i) batch systems:

    Many, perhaps most (at least in academic/research environments),
    supercomputers and linux clusters run batch job processing software.
    Such software includes "Torque", "PBS", "Sun Grid Engine". In this
    model, one writes a job script, submits it, it's put in a queue, and
    scheduled for execution (by a batch scheduler like "maui"). Batch
    processing allows fair sharing of a limited computational resource
    while providing guaranteed run times of jobs and exclusive locks on
    nodes.

    It feels a bit 1970s I guess, but unless users tolerate highly variable
    run times due to users' processes slowing eachother down (and most
    _don't_ at the high end), batch processing remains the choice.

    In your case, you might submit 100 jobs. The batch+scheduler system,
    usually implemented as a collection of daemon processes, looks after
    spawning your processes on available nodes (or reserving nodes for your
    use later if need be according to some priority scheme), tracking
    stdio, staging in input data and staging back out outputs, and so on,
    according to your specifications laid out in the job submission
    script(s) for your jobs. (you can typically pass parameters or at least
    env vars into the script, thus reusing the same basic script for 1000s
    of similar but nonidentical runs)

    [Note 1] Alternatively you might structure your run as a single 100-proc
    parallel job, perhaps with the help of an MPI implementation or a ready
    made intra-batch-job task farming script.

    Here are the instructions for a pretty typical batch facility:
    http://www.ichec.ie/user_documentation
    (I'm not affiliated with them, I run a smaller 256-proc system elsewhere
    in Ireland).


    (ii) Process migrating linux clusters:

    In these clusters, which typically involve a patched kernel, you just
    spawn everything on the head node, and the system farms out (with
    varying degrees of efficiency and post-migration performance issues -
    check out "kerrighed" for a particularly interesting one) to worker
    nodes. These *may* _also_ be batch systems, but typically you see them
    where a cluster is used by a single department or research group, where
    squabbling over "hogging" of resources is less likely. The naive "kick
    stuff off in a loop in shell script with &" may well work on such
    clusters.

    (To people in the field "beowulf" suggests a particular kind of process
    migrating linux cluster, though laymen tend to use the word
    in a much looser sense, thanks in no small part to trolls on /. ...)


    (iii) bunch o' machines linux clusters:

    In these clusters, there's no persistent management software or tight
    integration, and users can just kick off stuff on whatever node for
    however long whenever they like with ssh - But systems like
    mpich2's "mpd" persistent daemons are often used to manage parallel
    jobs more efficiently. This works fine so long as your users all like
    and care about eachother and pay attention to eachother's needs.
    As if.




  16. Re: help me solve a problem in converting a non-parallel program into parallel program...


    "Malcolm McLean" wrote in message
    news:OOmdnR_K0LLS8DzYnZ2dnUVZ8s6gnZ2d@bt.com...
    >
    >
    > Eg you want to calcuate an average of 10000 numbers in parallel.
    >
    > matlab spits out
    >
    > double sum(double *x, int N)
    > {
    > int i;
    > doubel answer = 0;
    > for(i=0;i > answer += x[i];
    > return answer;
    > }
    >
    > pur MPI program goes something like
    >
    > #include "mpi.h"
    >
    > int main(int argc, char **argv)
    > {
    > double array[10000];
    > int rank;
    > MPI_Init(argc, argv);
    > MPI_Rank(MPI_COMM_WORLD, &rank);
    > readinarray(array);
    > if(rank == 0)
    > {
    > for(i=1;i<100;i++)
    > /* haven't got the parameters quite right */
    > MPI_Send(array + i * 100, MPI_DOUBLE, 100, i, MPI_COMM_WORLD):
    > }
    > else
    > {
    > MPI_Recv(array, MPI_DOUBLE,100, 0, MPI_COMM_WORLD);
    > }
    > /* calcuate a partial answer */
    > answer = sum(array, 100);
    > if(rank != 0)
    > MPI_Send(&answer, MPI_DUBLE, 1, 0, MPI_COMM_WORLD):
    > else
    > {
    > double total = answer;
    > double partial;
    > for(i=1;i<100;i++)
    > {
    > MPI_Recv(&partial, MPI_DOUBLE, 1, i, MPI_COMM_WORLD);
    > total += partial;
    > }
    > printf("average is %g\n", total/10000);
    > }
    >
    > MPI_Finalize();
    > }
    >
    > This is only untested code, but it gives you the general idea. An average
    > is extremely easy to break up. Other functions might be a lot harder.
    >


    Shame on me. I tried your tutorial examples, but there is no mpicc, or
    mpirun on the debugging computer that I am using. (I don't have access to
    the super computer yet and I would like to debug everything locally
    first...)

    What's the best mpi implementation? I am not a Linux admin, will I be able
    to install an MPI and test debugging my program?



+ Reply to Thread