doubts about disk scheduling - Linux

This is a discussion on doubts about disk scheduling - Linux ; > I suspect that Windows still doesn't know how to lower the priority > of a background process, or doesn't know how to preempt it for > responsiveness. It knew this since NT 3.5 at least. "Background" means - NOT ...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 21 to 33 of 33

Thread: doubts about disk scheduling

  1. Re: doubts about disk scheduling

    > I suspect that Windows still doesn't know how to lower the priority
    > of a background process, or doesn't know how to preempt it for
    > responsiveness.


    It knew this since NT 3.5 at least. "Background" means - NOT the process who
    owns the foreground window.

    --
    Maxim Shatskih, Windows DDK MVP
    StorageCraft Corporation
    maxim@storagecraft.com
    http://www.storagecraft.com


  2. Re: doubts about disk scheduling

    "Ciaran Keating" writes:
    > On Wed, 13 Dec 2006 01:47:39 +1100, Rainer Weikusat
    > wrote:
    >
    >> Except that the problem the original poster was phantasizing about is
    >> imaginary (if not an outright fabrication).

    >
    > Over here in alt.os.development, fantasizing and using your
    > imagination is allowed.


    A discussion which refers to facts that are not facts is pointless.
    Especially, if circumstance give raise to the suspicion that the real
    motivation behind the post and especially, the crosspost, could have
    been to post something in favor of some feature of a recently released
    commercial operating system (like that it reportedly cannot be used
    with a main memory size <= 2G :->>).

    >> The only reason why an

    >
    > And we like to keep an open mind. I mean, we positively enjoy it.


    It is fine of you are not interested in discussing OS technology
    (strongly suggested by removing the content of my statement). But
    don't pretend that you are, then. Write novels.

    >>> That may be true, if reducing disk IO is the goal.

    >>
    >> And because the disk is by far the slowest heavily used peripheral,
    >> increasing disk througput is the goal

    >
    > Well, actually, the original question was: "I have looked on disk
    > scheduling algorithms [...] and the main thing that striked me is that
    > most of the algorithms [...] don't take into consideration the
    > "priority of the process". [...]


    First, see above. Second: The priority of the process is non-relevant
    for write requests, because write requests are supposed to be
    processed asynchronously by the kernel, ie the process copies the
    data into an in-kernel memory buffer (simplified) and is done with the
    request. It is possible to sabotage this through
    cargo-cult-programming, like pointless calls to fsync, but that is not
    an I/O priorization problem. Third, the priority does not matter for
    read requests because most reads are supposed to come from the disk
    cache in RAM.

    That is the situation as it should be. This, of course, assumes that
    the user is capable of not overloading the machine he is using to the
    point that it barely crawls along and to then go shopping for a faster
    CPU. Any 'desktop' power luser will, of course, just do that. Enter
    priority based I/O-scheduling, which has been implemented in Linux 2.6
    since March 2005 (see kernel documentation).

    BTW, I am using a laptop with 192M of RAM and a dreadfully slow IDE
    disk, but for some strange reason, I can use it interactively while
    doing large builds

    Without using a different algorithm than the basic one outlined above.

    F'up2 colds

  3. Re: doubts about disk scheduling

    > been to post something in favor of some feature of a recently released
    > commercial operating system (like that it reportedly cannot be used
    > with a main memory size <= 2G :->>).


    Wrong. 1GB is enough, the working set of the empty OS is ~512MB.

    --
    Maxim Shatskih, Windows DDK MVP
    StorageCraft Corporation
    maxim@storagecraft.com
    http://www.storagecraft.com


  4. Re: doubts about disk scheduling

    "Maxim S. Shatskih" writes:

    [...]

    > the working set of the empty OS is ~512MB.


    The term you were looking for would be something like 'memory
    footprint' ...


  5. Re: doubts about disk scheduling

    Rainer Weikusat wrote:
    > [...] Especially, if circumstance give raise to the suspicion that
    > the real motivation behind the post and especially, the crosspost,
    > could have been to post something in favor of some feature of a
    > recently released commercial operating system [...]


    who cares which operating system has which features? the original post
    asked about i/o scheduling algorithms - this is NOT related to any
    operating system.


    if you want discuss wether linux or vista is better, head over to the
    heise.de-forums. thank you.



    > [...] Second: The priority of the process is non-relevant for write
    > requests, because write requests are supposed to be processed
    > asynchronously by the kernel, ie the process copies the data into an
    > in-kernel memory buffer (simplified) and is done with the request.
    > It is possible to sabotage this through cargo-cult-programming, like
    > pointless calls to fsync, but that is not an I/O priorization
    > problem. Third, the priority does not matter for read requests
    > because most reads are supposed to come from the disk cache in RAM.


    that means i/o scheduling itself is pointless, because all requests are
    serviced from/to ram?

    common...



    > [...] Enter priority based I/O-scheduling, which has been implemented
    > in Linux 2.6 since March 2005 (see kernel documentation).


    hehey... if you can post a link to the documentation of the
    implementation or explain it here, you finally posted a relevant answer
    to the original post.



    > And because the disk is by far the slowest heavily used peripheral,
    > increasing disk througput is the goal because the latency is always
    > going to suck.


    why not try to improve responsiveness by reducing i/o latency and
    troughput for foreground processes?



    regards,
    simon


    --

    http://valhalla.iru.ch/

    ----== Posted via Newsfeeds.Com - Unlimited-Unrestricted-Secure Usenet News==----
    http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
    ----= East and West-Coast Server Farms - Total Privacy via Encryption =----

  6. Re: doubts about disk scheduling

    Simon Felix writes:
    > Rainer Weikusat wrote:


    [...]

    >> [...] Second: The priority of the process is non-relevant for write
    >> requests, because write requests are supposed to be processed
    >> asynchronously by the kernel, ie the process copies the data into an
    >> in-kernel memory buffer (simplified) and is done with the request.
    >> It is possible to sabotage this through cargo-cult-programming, like
    >> pointless calls to fsync, but that is not an I/O priorization
    >> problem. Third, the priority does not matter for read requests
    >> because most reads are supposed to come from the disk cache in RAM.

    >
    > that means i/o scheduling itself is pointless, because all requests are
    > serviced from/to ram?


    That means what I have written.

  7. Re: doubts about disk scheduling

    On Wed, 13 Dec 2006 20:28:39 +1100, Rainer Weikusat
    wrote:

    > "Ciaran Keating" writes:


    >> Over here in alt.os.development, fantasizing and using your
    >> imagination is allowed.

    >
    > A discussion which refers to facts that are not facts is pointless.


    Well, before we get into an argument ;-) let's stop. I'm coming from the
    alt.os.development perspective, and I interpreted the original post as
    asking whether it would be possible to design an OS that did xyz. It seems
    you're coming from one of the Linux groups' perspective, and perhaps you
    interpreted it as asking whether xyz could be done on Linux (or perhaps on
    any existing OS.) Have I got that right?


    > The priority of the process is non-relevant
    > for write requests, because write requests are supposed to be
    > processed asynchronously by the kernel,


    This is what I mean. In the green fields where I come from, there is no
    "supposed to". Is this the basis of our disagreement?


    > F'up2 colds


    ?


    Cheers,
    Ciaran

    --
    Ciaran Keating
    Amadán Technologies Pty Ltd

  8. Re: doubts about disk scheduling

    In article <87tzzzamdr.fsf@farside.sncag.com>,
    Rainer Weikusat wrote:

    >> the working set of the empty OS is ~512MB.


    >The term you were looking for would be something like 'memory
    >footprint' ...


    Or should that be "elephantine memory footprint"?




  9. Re: doubts about disk scheduling

    xu_feng_xu@yahoo.com wrote:
    > I would appreciate any help on the following questions.
    >
    > I have looked on disk scheduling algorithms
    >
    > and the main thing that striked me is that most of the algorithms that
    > i have read in the textbooks (some are explained in the previous )
    > don't take into consideration the "priority of the process". Being the

    ....
    > their importance. any comment?


    I have some comments to make which are fairly long but I'll try to keep
    them readable. Basically that the process priority is not a good
    instrument for managing disk scheduling.

    Taking a step back from just looking at the disk, at any given time
    there is a quantity of 'work' for the computer system to do. To
    accomplish this work will require resources of various kinds - CPU,
    disk, display, user-input etc. It is a key job of the OS to adequately
    schedule these resources.

    It appears that there is a critical distinction between /real-time/
    work and the rest. As I understand it real time is NOT generally
    supported under PC OSs (even if they have a scheduler class called
    "real time"). To handle real-time work requires the OS to give
    /guarantees/ to the tasks which need them. These could be conditional
    in the sense that the OS will provide X resources per time interval as
    long as the task does not try to consume more than Y. A good example is
    CPU time - the task may request to be invoked with a certain frequency
    and to be allowed a certain amount of time to execute on each occasion.
    The OS agrees to provide this if the task does not try to exceed its
    agreed execution time. If it does the scheduler is right to preempt it
    and flag the task as in error. (The action taken on this error is
    another discussion.)

    Of course, CPU guarantees can be useless unless they are backed up by
    guarantees for other resources too. This, I think, is where many OSes
    fail - including all Windows versions - though someone who knows more
    about they could maybe correct me on that. Symptoms may be that video
    becomes jittery when another load is added to the system.

    Once the real-time guarantees have been taken away there will be
    certain resources left for other work. Note, for this to be effective
    the contracts will be for certain /minimum/ guarantees of service to
    the real-time tasks - e.g. will need to read /at least/ X bytes of file
    Y per time interval (and other issues out of scope here). There is no
    reason why resources allocated to but not used by real-time work cannot
    be used for other work as long as the OS is able to keep the guarantees
    it made to the real-time tasks.

    Once the real-time issues are out of the way how do the rest of the
    resources get allocated to other tasks? This is where your disk
    scheduling issues come in. I guess there are many answers. To take an
    analogy from the world of networking where voice and data packets
    contend for a slow link, the voice packets may be allocated certain
    guaranteed bandwidth (the real-time model) but the rest are not (these
    days) allocated a fixed priority scheme. Rather the rest may be divided
    into classes - 1) business critical transactional traffic, 2) business
    critical bulk traffic, 3) inter-server syncronisation, 4) backups, 5)
    everything else. The categories I've mentioned are not important here
    and are just for illustration: note that there are several categories
    and each has /different scheduling needs/.

    Once traffic has been categorised then the routers can make intelligent
    decisions about how to schedule these packets through the low-bandwidth
    link. How it does that depends not on a simple priority scheme but on
    the needs of the applications producing and consuming the traffic - the
    transactional traffic needs fast response but not much bandwidth,
    business critical bulk traffic may need high reliability but not
    necessarily low turn-round times etc.

    I guess I am suggesting that the same principles apply to OS resource
    scheduling. Whatever the needs of the applications (or, the users
    interacting with the apps) should govern how they they are scheduled,
    not a fixed priority scheme. In fact, the process priority may be
    irrelevant to the needed disk priority. Some apps need lots of CPU
    time, some need lots of disk activity - either sequential or random or
    both - some need to finish soon and others can finish later.

    OK, given the above model how can it be used?

    One guide to the efficiency of a computer system is how well it manages
    to keep its resources used. (It is debatable whether this is a good
    measure - in fact the topic is far broader - but will serve to make the
    point.) The system will become less efficient as higher and higher
    demands are placed on its resources. Take the CPU for instance. It
    takes a very small amount of time to change from handling one task to
    handling another. If it has up to a few hundred tasks on the go at one
    time the time spent switching between them may not be too high and it
    may be able to get round them all quick enough to be able to give each
    a good enough slice of the CPU's time. If it had tens of thousands of
    tasks to schedule at once the overheads and the time taken to get round
    them all would make each one appear very slow.

    It is similar but even more critical with disk. Having too many tasks
    all expecting to access parts of a disk at the same time could lead to
    them all killing the performance of each other.

    For such reasons and others some systems provide a job scheduler. Its
    task is to keep pieces of requested work in a queue where they wait
    until the job scheduler thinks that it is OK to start executing another
    one. Further, a system may swap-out an active process for a while if
    the system's overall workload becomes too high. Of course, the job
    scheduler can make better decisions about incoming work if it has an
    indication of what resources jobs will demand most.

    Very simplistically: Where a task needs to be interactive its active
    set should be kept in memory; where a task is I/O bound and
    non-interactive it should maximise throughput.

    So re. your query about whether a high priority task should get higher
    priority disk access I think my answer is a qualified No.

    A final thought: a scheduler can be used wherever there is a limited
    resource to control. Therefore there may be a separate scheduler for
    each disk spindle or for each disk channel, in addition to the general
    memory/cache/disk control. I'm not sure if I am misapplying a term here
    but where a process manages a resource I think of it as a 'manager'
    rather than a scheduler. Scheduling would be just one part of what it
    does.


  10. Re: doubts about disk scheduling

    James Harris wrote:
    >
    > It appears that there is a critical distinction between /real-time/
    > work and the rest. As I understand it real time is NOT generally
    > supported under PC OSs (even if they have a scheduler class called
    > "real time").


    Those OSes offer "soft real time". It is suitable for the desktop where,
    for example, video capture would require "real time" (i.e, very fast,
    nearly instantaneous) response from the system.

    > To handle real-time work requires the OS to give
    > /guarantees/ to the tasks which need them.


    That sort of guarantee is commonly called "hard real time" and found in
    embedded control systems, not PCs and servers. Hard real time system
    have an upper bound on response time that is guaranteed not to be exceeded.

  11. Re: doubts about disk scheduling


    John-Paul Stewart wrote:
    > James Harris wrote:

    ....

    > Those OSes offer "soft real time". It is suitable for the desktop where,
    > for example, video capture would require "real time" (i.e, very fast,
    > nearly instantaneous) response from the system.


    Generally I agree with the comments you made but I should say I think
    of real-time video "capture" as needing predictable bursts of CPU and
    other resources rather than having to happen instantaneously. Of
    course, receipt of data from the video stream would need to keep up but
    the rest - processing and writing to disk - could happen a little
    later. Data coming in from a network card presents similar issues but
    less predictable and more diverse ones.


  12. Re: doubts about disk scheduling

    John-Paul Stewart writes:
    > James Harris wrote:
    >> It appears that there is a critical distinction between /real-time/
    >> work and the rest. As I understand it real time is NOT generally
    >> supported under PC OSs (even if they have a scheduler class called
    >> "real time").

    >
    > Those OSes offer "soft real time". It is suitable for the desktop
    > where, for example, video capture would require "real time" (i.e, very
    > fast, nearly instantaneous) response from the system.


    It does not mean 'very fast'. It means that the video stream consists
    of 'blobs' of data that come at regular intervals and need to be
    processed when they come. It is called 'soft' real time because
    nothing particularly dramatic is going to happen if you occasionally
    miss such a 'deadline' (except maybe the captured stream contains
    distortions), especially, if the time lag is small. User interfaces,
    especially GUIs, generally have 'soft realtime requirements'
    (something should happen when the user does something).

    Opposed to this, a 'hard real-time system' must not miss a deadline,
    because otherwise, some serious malfunction is going to happen (like
    'nuclear power plant explodes').

    If a task uses realtime scheduling, that usually means that it is not
    subject to 'normal' dynamic priority recalculation based on its CPU
    usage, but has been assigned a static priority, which is higher than
    that of any 'dynamically scheduled' task. If it becomes runnable, it
    will preempt any running task of a lower priority and it will either
    run until it is itself preempted by a higher priority task or until it
    voluntarily relinquishes the CPU (SCHED_FIFO), or that it runs until
    some timeslice has expired and it its then preempted if a task of the same
    priority is runnable (SCHED_RR).

  13. Re: doubts about disk scheduling

    Rainer Weikusat wrote:
    [about real-time systems]
    > It does not mean 'very fast'. It means that the video stream consists
    > of 'blobs' of data that come at regular intervals and need to be
    > processed when they come. It is called 'soft' real time because
    > nothing particularly dramatic is going to happen if you occasionally
    > miss such a 'deadline' (except maybe the captured stream contains
    > distortions), especially, if the time lag is small. User interfaces,
    > especially GUIs, generally have 'soft realtime requirements'
    > (something should happen when the user does something).
    >
    > Opposed to this, a 'hard real-time system' must not miss a deadline,
    > because otherwise, some serious malfunction is going to happen (like
    > 'nuclear power plant explodes').
    >
    > If a task uses realtime scheduling, that usually means that it is not
    > subject to 'normal' dynamic priority recalculation based on its CPU
    > usage, but has been assigned a static priority, which is higher than
    > that of any 'dynamically scheduled' task. If it becomes runnable, it
    > will preempt any running task of a lower priority and it will either
    > run until it is itself preempted by a higher priority task or until it
    > voluntarily relinquishes the CPU (SCHED_FIFO), or that it runs until
    > some timeslice has expired and it its then preempted if a task of the same
    > priority is runnable (SCHED_RR).


    Great summary. Now, the CPU is not the only resource that might need to
    be properly scheduled in order to meet a deadline. This means both
    controlling preemption to or from the critical process and choosing
    the right process when a resource is voluntarily released.

    Disk I/O is not generally appropriate in the hard real-time process
    since it's hard to put a reasonable upper bound on the time it can take;
    real drives have nasty habits like becoming unresponsive for a while
    when they decide it's time to recalibrate their seek tables, and
    "transparently" retrying, correcting, or relocating sectors. Video
    streaming requires a special mode declaring that you'd rather get an
    error promptly than corrected data late.

    If a process has somewhat relaxed real-time requirements so disk I/O
    is appropriate, then it still needs to be protected from an excessive
    delay due to a long queue, say from a cache purge. I've seen systems
    in which the asynchronous writes are throttled just to avoid this
    problem, although that then reduces the effectiveness of the elevator
    algorithm. It would be better to schedule the real-time I/O ahead of
    the writes, so that the real-time process can make its deadline, even
    though this reduces the efficiency and throughput of the disk.

    Often enough, priority is the only indication the OS has that a process
    has some real-time requirement. Always scheduling I/O by priority ahead
    of disk position is somewhat extreme but does help in cases such as
    already mentioned in this thread, where a background build bogs down
    foreground work. Of course, memory ownership also has to be prioritized
    properly.

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2