[9fans] Quick question on stopping a process that waits for IO - Plan9

This is a discussion on [9fans] Quick question on stopping a process that waits for IO - Plan9 ; Guys, when somebody tries to stop a process that is waiting for the IO the process doesn't get transferred to a Stopped state immediately but only when the scheduler sees it for the first time. This leads to a process ...

+ Reply to Thread
Results 1 to 15 of 15

Thread: [9fans] Quick question on stopping a process that waits for IO

  1. [9fans] Quick question on stopping a process that waits for IO

    Guys,

    when somebody tries to stop a process that is waiting for the IO the
    process
    doesn't get transferred to a Stopped state immediately but only when
    the scheduler sees it for the first time. This leads to a process
    writing to
    a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient.

    Is there any way I can poke the target process so that it gets attention
    from the scheduler an can be put in a Stopped state?

    Thanks,
    Roman.


  2. Re: [9fans] Quick question on stopping a process that waits for IO

    On Sun, 2008-11-02 at 21:55 -0800, Roman Shaposhnik wrote:
    > Is there any way I can poke the target process so that it gets attention
    > from the scheduler an can be put in a Stopped state?


    I know, I know we all don't like those guys who talk to themselves
    on mailing lists replying to their own emails, but since there were
    no takers here's what I cooked up so far:
    echo stop > /proc/n/ctl &
    echo ping > /proc/n/note
    cat /proc/n/note > /dev/null

    That of course doesn't quite work either since once I resume 'n'
    by issuing
    echo start > /proc/n/ctl
    the process fails with:
    error reading : interrupted

    So the question remains -- what is the proper way of putting a process
    that waits for an IO into a Stopped state?

    Sorry that I keep bothering the list, but I can't really think
    of anything here :-(

    Thanks,
    Roman.



  3. Re: [9fans] Quick question on stopping a process that waits for IO

    On Sun, Nov 2, 2008 at 9:55 PM, Roman Shaposhnik wrote:
    > Guys,
    >
    > when somebody tries to stop a process that is waiting for the IO the process
    > doesn't get transferred to a Stopped state immediately but only when
    > the scheduler sees it for the first time. This leads to a process writing to
    > a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient.
    >
    > Is there any way I can poke the target process so that it gets attention
    > from the scheduler an can be put in a Stopped state?
    >


    Hi Roman, what's the bigger picture here? I am trying to understand
    .... sorry if I'm missing the obvious.

    ron


  4. Re: [9fans] Quick question on stopping a process that waits for IO

    >> Is there any way I can poke the target process so that it gets attention
    >> from the scheduler an can be put in a Stopped state?

    >
    > I know, I know we all don't like those guys who talk to themselves
    > on mailing lists replying to their own emails, but since there were
    > no takers here's what I cooked up so far:
    > echo stop > /proc/n/ctl &
    > echo ping > /proc/n/note
    > cat /proc/n/note > /dev/null
    >
    > That of course doesn't quite work either since once I resume 'n'
    > by issuing
    > echo start > /proc/n/ctl
    > the process fails with:
    > error reading : interrupted


    this is an uncaught error in port/proc.c:/^sleep. many interruptable
    calls to sleep are wrapped by
    while(waserror())
    ;
    sleep(...);
    poperror();

    but this is a catch-22. if the i/o resumed and didn't send a note,
    it would not be possible to interrupt processes with del with rio's
    current convention.

    on the other hand, sending a note is the only way to get to procctl
    unless one is doing syscall tracing.

    oddly procctl says
    /*
    * called splhi() by notify(). See comment in notify for the
    * reasoning.
    */
    but there doesn't seem to be a comment in any version of notify
    i can find.

    >
    > So the question remains -- what is the proper way of putting a process
    > that waits for an IO into a Stopped state?


    i don't think it's possible without changing the kernel.
    but it's a good question, why does it work this way?
    obviously one doesn't want to stop a process and kernel work in
    the middle of i/o. there may be locks involved, etc.
    but one could easily check for Proc_stopme on syscall
    exit.

    does anyone remember the thinking that went into this?

    - erik



  5. Re: [9fans] Quick question on stopping a process that waits for IO

    On Mon, 2008-11-03 at 17:01 -0800, ron minnich wrote:
    > On Sun, Nov 2, 2008 at 9:55 PM, Roman Shaposhnik wrote:
    > > when somebody tries to stop a process that is waiting for the IO the process
    > > doesn't get transferred to a Stopped state immediately but only when
    > > the scheduler sees it for the first time. This leads to a process writing to
    > > a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient.
    > >
    > > Is there any way I can poke the target process so that it gets attention
    > > from the scheduler an can be put in a Stopped state?
    > >

    >
    > Hi Roman, what's the bigger picture here? I am trying to understand
    > ... sorry if I'm missing the obvious.


    I'm glad you've asked ;-) In fact, there's a bigger context and it is
    around managing processes run by cpu(1) from the terminal host. I was
    planning on writing an email on that subject to this list over the
    weekend but I need to amass some level of intelligence in that area
    first.

    For now, though, the easiest way to illustrate my problem would be to
    imagine a situation where you want to write some data into a Data
    segment of the process that is waiting for the IO. Now, waiting for
    the IO is almost as good as state as Stopped (the process is stuck in
    kernel waiting to be put back on the runqueue) so one would expect the
    transition between these 2 states to be almost trivial. Literally, all
    I'm asking is -- "dear kernel, please don't advance this process even
    if you otherwise can". All I need is a frozen state so that I can
    manipulate some data. Yet, I can't figure out how to do that:
    term% ps | grep 310
    glenda 310 0:00 0:00 24K Pread cat

    term% cat /proc/310/seg*
    Stack 0f000000 10000000 0
    Text R 00001000 00006000 0
    Data 00006000 00007000 0
    Bss 00007000 00007000 0

    term% dd -if /dev/zero -bs 1 -count 1 -of /proc/310/mem -seek 6100
    write: bad process or channel control request
    1+0 records in
    0+0 records out

    Of course, the process is Spleeping, so we need to stop it:

    term% echo stop > /proc/310/ctl

    This hangs, so I'll provide some input to 'cat' so that it becomes
    Stopped and I can do:
    term% dd -if /dev/zero -bs 1 -count 1 -of /proc/310/mem -seek 6100
    1+0 records in
    1+0 records out

    But! I'm really stuck if I can't give that input to something
    that is Pread.

    How are debugging/analysis tools supposed to deal with a situation
    like that? I would imagine this to be a pretty common situation
    for them, right?

    Thanks,
    Roman.



  6. Re: [9fans] Quick question on stopping a process that waits for IO

    > I'm asking is -- "dear kernel, please don't advance this process even
    > if you otherwise can". All I need is a frozen state so that I can


    not so easy on a multiprocessor. (unless you turn all but one
    processor off.)

    - erik



  7. Re: [9fans] Quick question on stopping a process that waits for IO

    > I'm glad you've asked ;-) In fact, there's a bigger context and it is
    > around managing processes run by cpu(1) from the terminal host. I was
    > planning on writing an email on that subject to this list over the
    > weekend but I need to amass some level of intelligence in that area
    > first.
    >
    > For now, though, the easiest way to illustrate my problem would be to
    > imagine a situation where you want to write some data into a Data
    > segment of the process that is waiting for the IO. Now, waiting for
    > the IO is almost as good as state as Stopped (the process is stuck in
    > kernel waiting to be put back on the runqueue) so one would expect the
    > transition between these 2 states to be almost trivial. Literally, all
    > I'm asking is -- "dear kernel, please don't advance this process even
    > if you otherwise can". All I need is a frozen state so that I can
    > manipulate some data. Yet, I can't figure out how to do that:


    i don't think the kernel has this level of control.

    let's suppose that we have a process that gets a stop message
    that's doing i/o. let's suppose that it's doing io to a particularly
    cranky device with lots of neat locks that really hates getting
    interrupted. i don't know, something esoteric — say, ata. are you
    really so sure that you can do what you want to this process
    when ata io is going on?

    i don't think this is safe unless all copies back to user space are
    done by some kernel function that does all the proper checking.
    and you have some way of unlocking safely from any part of the
    i/o code.

    i sure would like to hear from someone who knows more about
    what went into the design of notes. i don't think i've thought of
    all the scary dark corners.

    - erik



  8. Re: [9fans] Quick question on stopping a process that waits for IO

    On Nov 4, 2008, at 8:01 PM, erik quanstrom wrote:

    >> I'm asking is -- "dear kernel, please don't advance this process even
    >> if you otherwise can". All I need is a frozen state so that I can

    >
    > not so easy on a multiprocessor. (unless you turn all but one
    > processor off.)


    Hm. May be its getting late, but I can't quite see why that would
    be the case (or may be I didn't quite communicate the intent
    in my plea to the kernel ;-))

    The target process is *already* waiting for the IO stuck inside the
    kernel. It is not on a runqueue, not it is considered to be places
    there.
    All of the user-level data structures are frozen and
    not changing. All I'm asking is that if, and only if, the process
    gets attention from the scheduler (for whatever reason) it doesn't
    get placed on a runque.

    I don't quite see how MP would matter in this case.

    Thanks,
    Roman.


  9. Re: [9fans] Quick question on stopping a process that waits for IO

    On Nov 4, 2008, at 8:00 PM, erik quanstrom wrote:
    > i don't think the kernel has this level of control.
    >
    > let's suppose that we have a process that gets a stop message
    > that's doing i/o. let's suppose that it's doing io to a particularly
    > cranky device with lots of neat locks that really hates getting
    > interrupted. i don't know, something esoteric say, ata. are you
    > really so sure that you can do what you want to this process
    > when ata io is going on?


    Well, may be not. But I would like my right to shoot myself in the
    foot not to be revoked ;-)

    > i sure would like to hear from someone who knows more about
    > what went into the design of notes. i don't think i've thought of
    > all the scary dark corners.



    Same here.

    Now, There are two issues on the table right now: the first
    one has everything to do with notes, and I really would like to
    better understand that area myself. Hence I'm all ears.

    The other issue (how to manipulate the running process) has
    less to do with notes per se. As I said, I do appreciate the reasoning
    that a process has to be in a frozen state in order for writes
    on /proc/n/mem to succeed. By a frozen state I still mean the
    kind of a state that would prevent a scheduler to place it
    on a runqueue. Stopped is one such state, but may be it
    shouldn't be the only one.

    Would it be completely unreasonable for devproc.c to also
    allow writes for the processes which are in process of being
    stopped?

    Thanks,
    Roman.


  10. Re: [9fans] Quick question on stopping a process that waits for IO

    >>> I'm asking is -- "dear kernel, please don't advance this process even
    >>> if you otherwise can". All I need is a frozen state so that I can

    >>
    >> not so easy on a multiprocessor. (unless you turn all but one
    >> processor off.)

    >
    > Hm. May be its getting late, but I can't quite see why that would
    > be the case (or may be I didn't quite communicate the intent
    > in my plea to the kernel ;-))
    >
    > The target process is *already* waiting for the IO stuck inside the
    > kernel. It is not on a runqueue, not it is considered to be places
    > there.


    since procwrite doesn't acquire anything other than the debug lock,
    how do you know? the proc could start up again before you notice.

    you could start grabbing locks, stealing stuff off queues or adding
    if(up->procctl) statements here and there, but i have a strong feeling
    thair be dragons in that direction.

    the place to catch processes is on syscall entry and/or exit. stop
    messages are only checked on syscall entry.

    i have a feeling that you want something other than notes for
    your problem. cpu arranges that del works.

    - erik



  11. Re: [9fans] Quick question on stopping a process that waits for IO

    On Sun, Nov 02, 2008 at 09:55:16PM -0800, Roman Shaposhnik wrote:
    > Guys,
    >
    > when somebody tries to stop a process that is waiting for the IO the
    > process
    > doesn't get transferred to a Stopped state immediately but only when
    > the scheduler sees it for the first time. This leads to a process
    > writing to
    > a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient.
    >
    > Is there any way I can poke the target process so that it gets attention
    > from the scheduler an can be put in a Stopped state?
    >
    > Thanks,
    > Roman.


    Thinking aloud, and looking mostly at the port/ and pc/ code:

    port/devproc.c:/^procstopwait will be called when we write "stop" into
    /proc/n/ctl. Modulo details, this function will:
    qlock p->debug
    mark us as p's debugger
    set p->procctl to Proc_stopme
    drop p->debug
    wait until p->state == Stopped by using procstopped()
    return

    If a process is stopped on I/O, that wait is going to take "a while" (until
    I/O completes). Eventually, however, it will return into
    pc/trap.c:/^syscall and will eventually run the code:

    > if(scallnr!=RFORK && (up->procctl || up->nnote)){
    > splhi();
    > notify(ureg);
    > }


    whereupon we expect [1] that it sees that up->procctl is nonzero and the
    process will put itself to sleep inside notify(), awake the waiter above,
    and everything will be fine.

    So it seems like it wouldn't be very hard to define a "soonstop" command
    which considered processes blocked on I/O "as good as stopped" since there
    are two outcomes:
    After we set up->procctl, the target process returned to userland and will
    sleep on the next trap they take or syscall they make. In this case, we
    should set ourselves as p's debugger and wait as we have a bounded wait:
    exactly one time quantum at most.

    After we set up->procctl, the process is still blocked (or blocked again)
    on I/O. They will not make it back to userland after this I/O completes.
    In this case, we do not need to register ourselves as p's debugger and
    may immediately return.

    Subject to appropriate fencing (if required, I'm not sure; sim. [1]) it
    seems that we can distinguish these cases in the requesting process.

    User memory and registers (probably just the return register, tho') would be
    subject to change by the kernel during the period where the process is
    asleep, but this is not too dissimilar from the state of the world when
    shared memory is in flight.

    Would this suffice? Did I miss something obvious?
    --nwf;

    [1] It may be necessary to insert a memory fence after the setting of
    p->procctl or before the read of up->procctl, since several places
    do not acquire up->debug before reading up->procctl. Can somebody
    comment?

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEARECAAYFAkkSgzwACgkQTeQabvr9Tc8+UwCbB8lShL5SrC 4rHY67I6lAu4C2
    AhQAn2pwm/kvsfCW47h03njeslGPzMQO
    =aAz2
    -----END PGP SIGNATURE-----


  12. Re: [9fans] Quick question on stopping a process that waits for IO

    On Nov 5, 2008, at 9:40 PM, Nathaniel W Filardo wrote:
    > Would this suffice?


    It sounds like exactly the kind of thing I was talking about.

    > Did I miss something obvious?


    And this would be a million dollar question here. I don't
    think you did (although Eric constantly warns us of
    dragons), but on the other hand I have very little
    experience with the kernel itself.

    Thanks,
    Roman.


  13. Re: [9fans] Quick question on stopping a process that waits for IO

    On Nov 5, 2008, at 4:55 AM, erik quanstrom wrote:
    >>>> I'm asking is -- "dear kernel, please don't advance this process
    >>>> even
    >>>> if you otherwise can". All I need is a frozen state so that I can
    >>>
    >>> not so easy on a multiprocessor. (unless you turn all but one
    >>> processor off.)

    >>
    >> Hm. May be its getting late, but I can't quite see why that would
    >> be the case (or may be I didn't quite communicate the intent
    >> in my plea to the kernel ;-))
    >>
    >> The target process is *already* waiting for the IO stuck inside the
    >> kernel. It is not on a runqueue, not it is considered to be places
    >> there.

    >
    > since procwrite doesn't acquire anything other than the debug lock,
    > how do you know? the proc could start up again before you notice.


    How? If there's a stop message already written to /proc/n/ctl. Once
    that is done, the process is guaranteed to be in 2 states and those
    states only: continue waiting for the I/O, being actually Stopped.
    Both of the don't let the scheduler take it to the runqueue.

    > the place to catch processes is on syscall entry and/or exit. stop
    > messages are only checked on syscall entry.
    >
    > i have a feeling that you want something other than notes for
    > your problem.


    If by the problem you mean stopping a process doing I/O than,
    I would gladly accept any solution be it notes or otherwise.

    > cpu arranges that del works.



    Well, it doesn't on Stopped processes.

    Thanks,
    Roman.


  14. Re: [9fans] Quick question on stopping a process that waits for IO

    On Nov 4, 2008, at 4:34 AM, erik quanstrom wrote:
    >> So the question remains -- what is the proper way of putting a
    >> process
    >> that waits for an IO into a Stopped state?

    >
    > i don't think it's possible without changing the kernel.
    > but it's a good question, why does it work this way?
    > obviously one doesn't want to stop a process and kernel work in
    > the middle of i/o. there may be locks involved, etc.
    > but one could easily check for Proc_stopme on syscall
    > exit.
    >
    > does anyone remember the thinking that went into this?


    I was waiting for somebody to chime in, but I guess no takers :-(

    Well, one last thing to report before we put this thread to rest
    would be the fact that acid(1) seems to be suffering from the
    same issue. If the process is doing I/O I can't modify the variables.
    Not a huge limitation, but a surprising one nonetheless if one
    draws from experience of working with conventional debuggers.

    Thanks,
    Roman.


  15. Re: [9fans] Quick question on stopping a process that waits for IO

    On Thu, Nov 06, 2008 at 09:18:47PM -0800, Roman Shaposhnik wrote:
    > On Nov 5, 2008, at 9:40 PM, Nathaniel W Filardo wrote:
    >> Would this suffice?

    >
    > It sounds like exactly the kind of thing I was talking about.


    OK. To reiterate what I said earlier, these kinds of "soonstop"-ed
    processes may still make some amount of progress in the kernel before
    becoming stopped, but they will indeed stop at their next scheduling
    opportunity, and ideally well before (when they go to return to userland).

    One could imagine a parallel "soonnote" which considered the note delivered
    even if the process was asleep in I/O. This might be the more appropriate
    thing for sending kill requests from the shell? Having not looked at the
    shell, this might be a bad idea; I'm not sure.

    >> Did I miss something obvious?

    >
    > And this would be a million dollar question here. I don't
    > think you did (although Eric constantly warns us of
    > dragons), but on the other hand I have very little
    > experience with the kernel itself.


    I hope somebody comments on the fencing that is or isn't needed here. Since
    we have parts of the kernel peeking witout locks at ->procctl, I worry about
    races, and wonder how, or if, the current design avoids them.

    ( Incidentally, if you're curious about in-kernel development and want to
    look at a strange system, it is exactly the species of dragons we have
    encountered here -- where the kernel has hijacked a userland thread to do
    driver operations, and is potentially sleeping with driver resources held or
    in local store -- that drove the design of the Coyotos kernel to being a
    strictly transactional system. AFAIR there it is always possible to tell a
    thread to kill itself after its current transaction or, if asleep or in
    commit phase, to force it out of stall queues and then kill it. )

    --nwf;

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEARECAAYFAkkT2kkACgkQTeQabvr9Tc+bKQCePBSEXHs4hv D6GXp+fdb3fQyI
    5GYAoIeQj67JAqDuwZhBPG0L9qpvbTIc
    =iHqP
    -----END PGP SIGNATURE-----


+ Reply to Thread