p_online - how to control CPU state from user space ? - Linux

This is a discussion on p_online - how to control CPU state from user space ? - Linux ; Hi there. First, I'm not an aficionado of this group, so please redirect me to the right place if not here. I'm trying to bind a user thread to one CPU on an SMP machine. I've seen the thread can ...

+ Reply to Thread
Results 1 to 17 of 17

Thread: p_online - how to control CPU state from user space ?

  1. p_online - how to control CPU state from user space ?

    Hi there.

    First, I'm not an aficionado of this group, so please redirect me to the
    right place if not here.

    I'm trying to bind a user thread to one CPU on an SMP machine. I've seen
    the thread can be bound using the affinity calls, but then, the manpages
    do not say how I can prevent other threads from trying to use this CPU.
    And if I can bind the thread and the CPU, I've stil got a problem with
    interruptions. I'm missing the call to force interrupts to be dispatched
    to other CPUs and not interfere with the one I've bound.

    I'm migrating from Solaris which provides a p_online call.

    Any idea how I should do that ?

    TIA

    rrr

  2. Re: p_online - how to control CPU state from user space ?

    Hello,

    > First, I'm not an aficionado of this group, so please redirect me to the
    > right place if not here.
    >
    > I'm trying to bind a user thread to one CPU on an SMP machine. I've seen
    > the thread can be bound using the affinity calls, but then, the manpages
    > do not say how I can prevent other threads from trying to use this CPU.
    > And if I can bind the thread and the CPU, I've stil got a problem with
    > interruptions. I'm missing the call to force interrupts to be dispatched
    > to other CPUs and not interfere with the one I've bound.
    >
    > I'm migrating from Solaris which provides a p_online call.
    >
    > Any idea how I should do that ?


    AFAIK, there is no equivalent in Linux for /p_online()/. That does not
    mean that it's not possible to do what you're aiming at; unfortunately
    I have no idea how without touching at the kernel.

    Cross-posted to c.o.l.d.s, as people there might help you further.

    Cheers,
    Loic.


  3. Re: p_online - how to control CPU state from user space ?

    loic-dev@gmx.net forwarded in
    news:1168247175.090139.106110@q40g2000cwq.googlegr oups.com:
    >> I'm trying to bind a user thread to one CPU on an SMP machine. I've
    >> seen the thread can be bound using the affinity calls, but then, the
    >> manpages do not say how I can prevent other threads from trying to
    >> use this CPU.


    You can bind your thread to a specified set of CPUs with
    sched_setaffinity.

    Since the first argument to sched_setaffinity is a process ID (which on
    Linux is actually the kernel thread ID), there's nothing to prevent you
    from gathering a list of all the currently running ((*)user) threads and
    using sched_setaffinity on each one. Also, since affinity is inherited
    by child processes, once you've done this to the init process and
    anything else already running, it should "stick" unless there is a
    process actively trying to subvert your policy. (Preventing that --
    i.e. disabling the ability for arbitrary processes to reset their
    affinities -- would require significant kernel code changes.)

    (*) There are several kernel threads that are created at startup on each
    CPU. Not 100% sure what the effects would be, but I'm guessing it would
    be extremely unwise to change their affinities.

    >> And if I can bind the thread and the CPU, I've stil got a problem
    >> with interruptions. I'm missing the call to force interrupts to be
    >> dispatched to other CPUs and not interfere with the one I've bound.


    Each interrupt can be forced to run on a set of CPUs by writing
    /proc/irq//smp_affinity

    GH

  4. Re: p_online - how to control CPU state from user space ?

    In comp.os.linux.development.system Gil Hamilton wrote:

    | Since the first argument to sched_setaffinity is a process ID (which on
    | Linux is actually the kernel thread ID), there's nothing to prevent you
    | from gathering a list of all the currently running ((*)user) threads and
    | using sched_setaffinity on each one. Also, since affinity is inherited
    | by child processes, once you've done this to the init process and
    | anything else already running, it should "stick" unless there is a
    | process actively trying to subvert your policy. (Preventing that --
    | i.e. disabling the ability for arbitrary processes to reset their
    | affinities -- would require significant kernel code changes.)

    Any process can do this to any process? Even non-root?

    It really should be a privileged operation. Most particularly, non-root
    users should not be able to affect any different user. Also, it should
    be restricted which CPUs a non-root user can add or remove. A per-CPU
    setting global to the system and only settable by root would specify if
    a non-root user is allowed to add that CPU or remove that CPU from the
    process affinity. Additionally, CPUs should be reservable for users or
    groups.


    | (*) There are several kernel threads that are created at startup on each
    | CPU. Not 100% sure what the effects would be, but I'm guessing it would
    | be extremely unwise to change their affinities.

    Hopefully non-root users can't do this.


    |>> And if I can bind the thread and the CPU, I've stil got a problem
    |>> with interruptions. I'm missing the call to force interrupts to be
    |>> dispatched to other CPUs and not interfere with the one I've bound.
    |
    | Each interrupt can be forced to run on a set of CPUs by writing
    | /proc/irq//smp_affinity

    Presumably for x86.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-01-08-1018@ipal.net |
    |------------------------------------/-------------------------------------|

  5. Re: p_online - how to control CPU state from user space ?

    phil-news-nospam@ipal.net wrote in news:entr88015t8@news1.newsguy.com:
    > In comp.os.linux.development.system Gil Hamilton
    > wrote:
    >| Since the first argument to sched_setaffinity is a process ID (which
    >| on Linux is actually the kernel thread ID), there's nothing to
    >| prevent you from gathering a list of all the currently running
    >| ((*)user) threads and using sched_setaffinity on each one.


    > Any process can do this to any process? Even non-root?


    No, of course not. It's a privileged operation to set the affinity of
    threads you don't own. Read the man pages. I was just pointing the
    direction, not providing a detailed specification.


    >|>> And if I can bind the thread and the CPU, I've stil got a problem
    >|>> with interruptions. I'm missing the call to force interrupts to be
    >|>> dispatched to other CPUs and not interfere with the one I've bound.
    >|
    >| Each interrupt can be forced to run on a set of CPUs by writing
    >| /proc/irq//smp_affinity
    >
    > Presumably for x86.


    Technically, this _is_ architecture-specific code, but it appears to be
    implemented on all of the more common architectures.

    GH

  6. Re: p_online - how to control CPU state from user space ?

    On Mon, 08 Jan 2007 13:48:52 +0100, Gil Hamilton wrote:

    > loic-dev@gmx.net forwarded in
    > news:1168247175.090139.106110@q40g2000cwq.googlegr oups.com:
    >>> I'm trying to bind a user thread to one CPU on an SMP machine. I've
    >>> seen the thread can be bound using the affinity calls, but then, the
    >>> manpages do not say how I can prevent other threads from trying to
    >>> use this CPU.

    >
    > You can bind your thread to a specified set of CPUs with
    > sched_setaffinity.
    >
    > Since the first argument to sched_setaffinity is a process ID (which on
    > Linux is actually the kernel thread ID), there's nothing to prevent you
    > from gathering a list of all the currently running ((*)user) threads and
    > using sched_setaffinity on each one. Also, since affinity is inherited
    > by child processes, once you've done this to the init process and
    > anything else already running, it should "stick" unless there is a
    > process actively trying to subvert your policy. (Preventing that --
    > i.e. disabling the ability for arbitrary processes to reset their
    > affinities -- would require significant kernel code changes.)
    >
    > (*) There are several kernel threads that are created at startup on each
    > CPU. Not 100% sure what the effects would be, but I'm guessing it would
    > be extremely unwise to change their affinities.
    >
    >>> And if I can bind the thread and the CPU, I've stil got a problem
    >>> with interruptions. I'm missing the call to force interrupts to be
    >>> dispatched to other CPUs and not interfere with the one I've bound.

    >
    > Each interrupt can be forced to run on a set of CPUs by writing
    > /proc/irq//smp_affinity
    >
    > GH


    Thanks for your answers.
    I had missed the proc/irq affinity feature.
    I suppose I can walk through the proc fs to get a list of all tid's.
    Unmaking the job however looks a bit harder.

    Is there a paper somewhere describing these topics ?

    And by the way, if pid's are real kernel tid's, is there a kind of
    similar relation between nptl thread_t's and tid's. Just in case I mean to
    use the pthread_setaffinity instead of sched_setaffinity. (Looks like
    gettid is no more available except as a syscall withing glibc-2.5).

    Thanx again.

    rrr

  7. Re: p_online - how to control CPU state from user space ?

    moi wrote in
    news:45a3e9aa$0$293$426a74cc@news.free.fr:
    > On Mon, 08 Jan 2007 13:48:52 +0100, Gil Hamilton wrote:
    >> sched_setaffinity.

    > I suppose I can walk through the proc fs to get a list of all tid's.
    > Unmaking the job however looks a bit harder.


    Indeed it would be harder. What's the "big picture" you're trying to
    achieve with this?


    > Is there a paper somewhere describing these topics ?


    I doubt there's anything that puts it all together. There is the Posix
    thread interface and then there is the Linux system call interface.
    glibc provides the glue, but it is quite system-specific. And here,
    you're trying to do something not covered by the Posix interface so by
    definition you're into the realm of the system-specific.


    > And by the way, if pid's are real kernel tid's, is there a kind of
    > similar relation between nptl thread_t's and tid's. Just in case I
    > mean to use the pthread_setaffinity instead of sched_setaffinity.
    > (Looks like gettid is no more available except as a syscall withing
    > glibc-2.5).


    Obviously there is an underlying relationship between the thread_t and
    the TID, but Posix provides no way of obtaining the TID as far as I
    know.

    GH

  8. Re: p_online - how to control CPU state from user space ?

    On Tue, 09 Jan 2007 20:59:52 +0100, Gil Hamilton wrote:

    > moi wrote in
    > news:45a3e9aa$0$293$426a74cc@news.free.fr:
    >> On Mon, 08 Jan 2007 13:48:52 +0100, Gil Hamilton wrote:
    >>> sched_setaffinity.

    >> I suppose I can walk through the proc fs to get a list of all tid's.
    >> Unmaking the job however looks a bit harder.

    >
    > Indeed it would be harder. What's the "big picture" you're trying to
    > achieve with this?
    >

    <...>

    The purpose here is to be able to launch a real time task, with my own
    simplified scheduler and never make it wait. I used this technique on
    Solaris and it gave quite good results (about 40 us max latencies). IOs
    were all mmaped to a PCI device on a dedicated bus and communication with
    other processes were achieved through shm's. I tried to write a POSIX only
    version of this program, playing with priorities and mlocking the whole
    memory image, but as soon as the process actually goes to sleep, it's
    impossible to manage the time it will take to wake up. This goes for Linux
    and Solaris just the same, (except I've had a look at the Linux sources).
    It seems Linux drivers are getting better everyday but old pieces of code
    which totally disable interrupts for a lap of time or acquire some kind of
    global lock are stil not uncommon. One solution is to keep away from the
    kernel, keeping greedily one CPU for myself, performing active waits in
    case I've got nothing to do. Well, now it seems this is not so simple.

    Executing hard real time tasks on a multi-user workstation running a
    standard OS, and of course, without being too obviously noticed, is an
    interesting challenge.

    rrr

  9. Re: p_online - how to control CPU state from user space ?

    On a sunny day (09 Jan 2007 23:43:02 GMT) it happened moi
    wrote in <45a42886$0$314$426a74cc@news.free.fr>:

    >On Tue, 09 Jan 2007 20:59:52 +0100, Gil Hamilton wrote:
    >
    >> moi wrote in
    >> news:45a3e9aa$0$293$426a74cc@news.free.fr:
    >>> On Mon, 08 Jan 2007 13:48:52 +0100, Gil Hamilton wrote:
    >>>> sched_setaffinity.
    >>> I suppose I can walk through the proc fs to get a list of all tid's.
    >>> Unmaking the job however looks a bit harder.

    >>
    >> Indeed it would be harder. What's the "big picture" you're trying to
    >> achieve with this?
    >>

    ><...>
    >
    >The purpose here is to be able to launch a real time task, with my own
    >simplified scheduler and never make it wait. I used this technique on
    >Solaris and it gave quite good results (about 40 us max latencies)


    Are you aware of Novell real time Linux?
    http://www.novell.com/products/realtime/overview.html
    It is specified at <30us.

  10. Re: p_online - how to control CPU state from user space ?

    Hello,

    > First, I'm not an aficionado of this group, so please redirect me to the
    > right place if not here.
    >
    > I'm trying to bind a user thread to one CPU on an SMP machine. I've seen
    > the thread can be bound using the affinity calls, but then, the manpages
    > do not say how I can prevent other threads from trying to use this CPU.
    > And if I can bind the thread and the CPU, I've stil got a problem with
    > interruptions. I'm missing the call to force interrupts to be dispatched
    > to other CPUs and not interfere with the one I've bound.
    >
    > I'm migrating from Solaris which provides a p_online call.
    >
    > Any idea how I should do that ?


    AFAIK, there is no equivalent in Linux for /p_online()/. That does not
    mean that it's not possible to do what you're aiming at; unfortunately
    I have no idea how without touching at the kernel.

    Cross-posted to c.o.l.d.s, as people there might help you further.

    Cheers,
    Loic.


  11. Re: p_online - how to control CPU state from user space ?

    loic-dev@gmx.net forwarded in
    news:1168247175.090139.106110@q40g2000cwq.googlegr oups.com:
    >> I'm trying to bind a user thread to one CPU on an SMP machine. I've
    >> seen the thread can be bound using the affinity calls, but then, the
    >> manpages do not say how I can prevent other threads from trying to
    >> use this CPU.


    You can bind your thread to a specified set of CPUs with
    sched_setaffinity.

    Since the first argument to sched_setaffinity is a process ID (which on
    Linux is actually the kernel thread ID), there's nothing to prevent you
    from gathering a list of all the currently running ((*)user) threads and
    using sched_setaffinity on each one. Also, since affinity is inherited
    by child processes, once you've done this to the init process and
    anything else already running, it should "stick" unless there is a
    process actively trying to subvert your policy. (Preventing that --
    i.e. disabling the ability for arbitrary processes to reset their
    affinities -- would require significant kernel code changes.)

    (*) There are several kernel threads that are created at startup on each
    CPU. Not 100% sure what the effects would be, but I'm guessing it would
    be extremely unwise to change their affinities.

    >> And if I can bind the thread and the CPU, I've stil got a problem
    >> with interruptions. I'm missing the call to force interrupts to be
    >> dispatched to other CPUs and not interfere with the one I've bound.


    Each interrupt can be forced to run on a set of CPUs by writing
    /proc/irq//smp_affinity

    GH

  12. Re: p_online - how to control CPU state from user space ?

    In comp.os.linux.development.system Gil Hamilton wrote:

    | Since the first argument to sched_setaffinity is a process ID (which on
    | Linux is actually the kernel thread ID), there's nothing to prevent you
    | from gathering a list of all the currently running ((*)user) threads and
    | using sched_setaffinity on each one. Also, since affinity is inherited
    | by child processes, once you've done this to the init process and
    | anything else already running, it should "stick" unless there is a
    | process actively trying to subvert your policy. (Preventing that --
    | i.e. disabling the ability for arbitrary processes to reset their
    | affinities -- would require significant kernel code changes.)

    Any process can do this to any process? Even non-root?

    It really should be a privileged operation. Most particularly, non-root
    users should not be able to affect any different user. Also, it should
    be restricted which CPUs a non-root user can add or remove. A per-CPU
    setting global to the system and only settable by root would specify if
    a non-root user is allowed to add that CPU or remove that CPU from the
    process affinity. Additionally, CPUs should be reservable for users or
    groups.


    | (*) There are several kernel threads that are created at startup on each
    | CPU. Not 100% sure what the effects would be, but I'm guessing it would
    | be extremely unwise to change their affinities.

    Hopefully non-root users can't do this.


    |>> And if I can bind the thread and the CPU, I've stil got a problem
    |>> with interruptions. I'm missing the call to force interrupts to be
    |>> dispatched to other CPUs and not interfere with the one I've bound.
    |
    | Each interrupt can be forced to run on a set of CPUs by writing
    | /proc/irq//smp_affinity

    Presumably for x86.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-01-08-1018@ipal.net |
    |------------------------------------/-------------------------------------|

  13. Re: p_online - how to control CPU state from user space ?

    phil-news-nospam@ipal.net wrote in news:entr88015t8@news1.newsguy.com:
    > In comp.os.linux.development.system Gil Hamilton
    > wrote:
    >| Since the first argument to sched_setaffinity is a process ID (which
    >| on Linux is actually the kernel thread ID), there's nothing to
    >| prevent you from gathering a list of all the currently running
    >| ((*)user) threads and using sched_setaffinity on each one.


    > Any process can do this to any process? Even non-root?


    No, of course not. It's a privileged operation to set the affinity of
    threads you don't own. Read the man pages. I was just pointing the
    direction, not providing a detailed specification.


    >|>> And if I can bind the thread and the CPU, I've stil got a problem
    >|>> with interruptions. I'm missing the call to force interrupts to be
    >|>> dispatched to other CPUs and not interfere with the one I've bound.
    >|
    >| Each interrupt can be forced to run on a set of CPUs by writing
    >| /proc/irq//smp_affinity
    >
    > Presumably for x86.


    Technically, this _is_ architecture-specific code, but it appears to be
    implemented on all of the more common architectures.

    GH

  14. Re: p_online - how to control CPU state from user space ?

    On Mon, 08 Jan 2007 13:48:52 +0100, Gil Hamilton wrote:

    > loic-dev@gmx.net forwarded in
    > news:1168247175.090139.106110@q40g2000cwq.googlegr oups.com:
    >>> I'm trying to bind a user thread to one CPU on an SMP machine. I've
    >>> seen the thread can be bound using the affinity calls, but then, the
    >>> manpages do not say how I can prevent other threads from trying to
    >>> use this CPU.

    >
    > You can bind your thread to a specified set of CPUs with
    > sched_setaffinity.
    >
    > Since the first argument to sched_setaffinity is a process ID (which on
    > Linux is actually the kernel thread ID), there's nothing to prevent you
    > from gathering a list of all the currently running ((*)user) threads and
    > using sched_setaffinity on each one. Also, since affinity is inherited
    > by child processes, once you've done this to the init process and
    > anything else already running, it should "stick" unless there is a
    > process actively trying to subvert your policy. (Preventing that --
    > i.e. disabling the ability for arbitrary processes to reset their
    > affinities -- would require significant kernel code changes.)
    >
    > (*) There are several kernel threads that are created at startup on each
    > CPU. Not 100% sure what the effects would be, but I'm guessing it would
    > be extremely unwise to change their affinities.
    >
    >>> And if I can bind the thread and the CPU, I've stil got a problem
    >>> with interruptions. I'm missing the call to force interrupts to be
    >>> dispatched to other CPUs and not interfere with the one I've bound.

    >
    > Each interrupt can be forced to run on a set of CPUs by writing
    > /proc/irq//smp_affinity
    >
    > GH


    Thanks for your answers.
    I had missed the proc/irq affinity feature.
    I suppose I can walk through the proc fs to get a list of all tid's.
    Unmaking the job however looks a bit harder.

    Is there a paper somewhere describing these topics ?

    And by the way, if pid's are real kernel tid's, is there a kind of
    similar relation between nptl thread_t's and tid's. Just in case I mean to
    use the pthread_setaffinity instead of sched_setaffinity. (Looks like
    gettid is no more available except as a syscall withing glibc-2.5).

    Thanx again.

    rrr

  15. Re: p_online - how to control CPU state from user space ?

    moi wrote in
    news:45a3e9aa$0$293$426a74cc@news.free.fr:
    > On Mon, 08 Jan 2007 13:48:52 +0100, Gil Hamilton wrote:
    >> sched_setaffinity.

    > I suppose I can walk through the proc fs to get a list of all tid's.
    > Unmaking the job however looks a bit harder.


    Indeed it would be harder. What's the "big picture" you're trying to
    achieve with this?


    > Is there a paper somewhere describing these topics ?


    I doubt there's anything that puts it all together. There is the Posix
    thread interface and then there is the Linux system call interface.
    glibc provides the glue, but it is quite system-specific. And here,
    you're trying to do something not covered by the Posix interface so by
    definition you're into the realm of the system-specific.


    > And by the way, if pid's are real kernel tid's, is there a kind of
    > similar relation between nptl thread_t's and tid's. Just in case I
    > mean to use the pthread_setaffinity instead of sched_setaffinity.
    > (Looks like gettid is no more available except as a syscall withing
    > glibc-2.5).


    Obviously there is an underlying relationship between the thread_t and
    the TID, but Posix provides no way of obtaining the TID as far as I
    know.

    GH

  16. Re: p_online - how to control CPU state from user space ?

    On Tue, 09 Jan 2007 20:59:52 +0100, Gil Hamilton wrote:

    > moi wrote in
    > news:45a3e9aa$0$293$426a74cc@news.free.fr:
    >> On Mon, 08 Jan 2007 13:48:52 +0100, Gil Hamilton wrote:
    >>> sched_setaffinity.

    >> I suppose I can walk through the proc fs to get a list of all tid's.
    >> Unmaking the job however looks a bit harder.

    >
    > Indeed it would be harder. What's the "big picture" you're trying to
    > achieve with this?
    >

    <...>

    The purpose here is to be able to launch a real time task, with my own
    simplified scheduler and never make it wait. I used this technique on
    Solaris and it gave quite good results (about 40 us max latencies). IOs
    were all mmaped to a PCI device on a dedicated bus and communication with
    other processes were achieved through shm's. I tried to write a POSIX only
    version of this program, playing with priorities and mlocking the whole
    memory image, but as soon as the process actually goes to sleep, it's
    impossible to manage the time it will take to wake up. This goes for Linux
    and Solaris just the same, (except I've had a look at the Linux sources).
    It seems Linux drivers are getting better everyday but old pieces of code
    which totally disable interrupts for a lap of time or acquire some kind of
    global lock are stil not uncommon. One solution is to keep away from the
    kernel, keeping greedily one CPU for myself, performing active waits in
    case I've got nothing to do. Well, now it seems this is not so simple.

    Executing hard real time tasks on a multi-user workstation running a
    standard OS, and of course, without being too obviously noticed, is an
    interesting challenge.

    rrr

  17. Re: p_online - how to control CPU state from user space ?

    On a sunny day (09 Jan 2007 23:43:02 GMT) it happened moi
    wrote in <45a42886$0$314$426a74cc@news.free.fr>:

    >On Tue, 09 Jan 2007 20:59:52 +0100, Gil Hamilton wrote:
    >
    >> moi wrote in
    >> news:45a3e9aa$0$293$426a74cc@news.free.fr:
    >>> On Mon, 08 Jan 2007 13:48:52 +0100, Gil Hamilton wrote:
    >>>> sched_setaffinity.
    >>> I suppose I can walk through the proc fs to get a list of all tid's.
    >>> Unmaking the job however looks a bit harder.

    >>
    >> Indeed it would be harder. What's the "big picture" you're trying to
    >> achieve with this?
    >>

    ><...>
    >
    >The purpose here is to be able to launch a real time task, with my own
    >simplified scheduler and never make it wait. I used this technique on
    >Solaris and it gave quite good results (about 40 us max latencies)


    Are you aware of Novell real time Linux?
    http://www.novell.com/products/realtime/overview.html
    It is specified at <30us.

+ Reply to Thread