A random thought about swap - Minix

This is a discussion on A random thought about swap - Minix ; Hi everybody, I have seen several requests on this newsgroup for swap implementation. We are so accustomed having swap partition that an operating system without it sounds like incomplete and crapy; at least for me at first hearing. But ... ...

+ Reply to Thread
Results 1 to 11 of 11

Thread: A random thought about swap

  1. A random thought about swap

    Hi everybody,

    I have seen several requests on this newsgroup for swap implementation.

    We are so accustomed having swap partition that an operating system
    without it sounds like incomplete and crapy; at least for me at first
    hearing.

    But ... a thing that strikes me on my Linux boxes with one Giga of RAM (
    which is actually not so much) is that it (almost) never swaps. Even
    under relative important load with a KDE session + Openoffice running +
    Mozilla running + Eclipse; well according to free(1) it doesn't swap.

    We know that the two mainstream free Desktop environments (KDE and
    GNOME) were underoptimized in regards of memory consumption. The
    developers have been/are working to improve that, and successfully.
    Another example: the uClibc shows it is possible to have a full fledged
    GNU/Linux-compatible libc with relatively low memory footprint.

    So by optimizing softwares we may get more funtionalities while not
    exploding RAM needs.

    With the lower price of RAM and bigger size in sight, the situation
    comes to: "You need more memory ? Just buy it".

    It may very well OS'es don't really need swap any more.

    I am wrong ?

    Sylvain

  2. Re: A random thought about swap

    There will always be a case when "swap" is needed: the system needs
    more RAM than is physically available. This is too vague and arbitrary
    of a condition to rule it out no matter what the technology is.
    Personal computers used to only have a few K of memory. Following your
    line of thinking, it would have been reasonable then to say, "with 16
    MEGABYTES, we'd never run out of memory."

    Swap, of course, is not the only solution to the memory problem, but
    it's a pretty good compromise. Running applications that aren't being
    used can be stored to swap without fundamentally altering their
    behavior. We could create special APIs so that programs can be
    unloaded fully without swapping to free memory when there is a demand,
    but then there would be more overhead in loading the software again
    when it's needed perhaps a moment later.

    Also, imagine that you needed the memory only temporarily. Whatever
    process that needed it would have to be denied the memory (without
    swap), so it would either have to sit and wait for an indefinite amount
    of time or fail and exit. It would have been more practical to just
    let it run real quick so that all tasks could be completed in a timely
    manner rather than some failing completely while others ran faster.

    I don't have the stamina to list every conceivable advantage, but you
    get the picture. I'm not trying to say that swap is "necessary" or
    even desirable in every case, but it certainly hasn't been obsoleted,
    and won't be at any predictable time in the future.

    Lemur


  3. Re: Re: A random thought about swap

    Lemur wrote :
    > There will always be a case when "swap" is needed: the system needs
    > more RAM than is physically available.


    Ressources are _always_ limited.

    Consider a system whith 1 Giga physical RAM + 1 Giga swap partition. It
    means a total virtual RAM of 2Gb. If you ever need more, you hit the limit.

    On the Linux systems I manage that have a similar configuration ( it
    amounts to two desktops and 2 servers ) swapping occur very rarely if
    ever ( I have never observed it ).

    Now. Suppose you have a box with 1 Gb and no way to extend the RAM using
    swapping. If you ever need more ressource ( which I doubt for most
    applications today ) the correct way to solve the problem would be to
    add more RAM, isn't it ? Then you put 2 Gb in your box and you get the
    same limit as above. But a lot faster.

    According to my experience 2 Gb is sufficient for the system to never be
    of memory shortage.

    Ok. I grant that not every hardware can be upgraded so.

    > Personal computers used to only have a few K of memory. Following your
    > line of thinking, it would have been reasonable then to say, "with 16
    > MEGABYTES, we'd never run out of memory."


    In these good old time also microkernels such as minix3 were deemed too
    slow to be of any practical interest

    I have recently developed an embedded system with a Linux kernel, a
    filesystem in ramdisk (sigh), a SOAP server, a Web server - and colleges
    that program drivers as if they were programming for MS Windows - that
    fits comfortably in 16 Megabytes of RAM. In fact there is some extra room.

    I know some people who would call bloatware this already small OS, and
    maybe they would have it right.

    My point is: for a newly developed OS - targetting among other embedded
    applications - on which there is much work yet to do - implementing swap
    should have a rather low priority; since on any recent computer there is
    plenty enough room of RAM to make it run without trouble.

  4. Re: A random thought about swap

    On Jan 26, 2:50 pm, SniX wrote:
    > My point is: for a newly developed OS - targetting among other embedded
    > applications - on which there is much work yet to do - implementing swap
    > should have a rather low priority; since on any recent computer there is
    > plenty enough room of RAM to make it run without trouble.


    I think you are right beyond question. Considering the practical
    constraints of most embedded systems, memory usage should be limited to
    whatever is readily accessible.

    Embedded systems have the luxury of conforming to a more well known
    design (eg limited resources and narrow task requirements) than other,
    more general systems. Swap was a convenient solution on my *desktop*
    when I could not afford enough RAM--being a poor college student does
    that to you. Swap is a useful tool when necessary, and it's hard to
    predict whether or not it's needed unless you have a specific target
    system in mind.

    Also, it occurred to me that you might be referring to swap partitions
    in particular, which are not a necessary part of swap. For example,
    even in Linux you can have swap files on your file system; this
    solution is more flexible and could be evoked only when absolutely
    necessary.

    > I know some people who would call bloatware this already small OS, and
    > maybe they would have it right.


    Yes, but luckily my _incredibly_ fast and powerful 586 machine with a
    whopping 40 megabytes of RAM is more than capable of handling a mammoth
    system like minix.


  5. Re: A random thought about swap



    On 27 Jan., 08:10, "Lemur" wrote:

    > constraints of most embedded systems, memory usage should be limited to
    > whatever is readily accessible.


    And where would that swap space reside anyway?

    Guido


  6. Re: A random thought about swap

    On Jan 27, 3:52 am, __guid...@web.de wrote:
    > And where would that swap space reside anyway?


    That's probably the best reason why memory usage should be limited,
    but there are some embedded devices that have access of some kind of
    expanded storage, whether they are DVR's with hard drives or MP3
    players with flash memory. Even though these devices could perform
    swap, it doesn't make any sense for either of them to do that because
    of how they function.

    Lemur


  7. Re: Re: A random thought about swap

    On Fri, 26 Jan 2007, SniX wrote:

    > Now. Suppose you have a box with 1 Gb and no way to extend the RAM using
    > swapping. If you ever need more ressource ( which I doubt for most
    > applications today )

    Image editing? Editing a 10 megapixel image ->
    10 megabyte per color -> 30meg, plus 10 meg for a mask and another 30 for
    each layer you get into the 100's of meg very quickly. Have a few you're
    working on or a really complex one and you've above a gig.

    16 channels of mutlitrack audio at 24bit x 96kHz costs about 3/4 gig for a
    3 minute song. Audio may lend itself to leave it on the disk/write it back
    out methods better so this might not be such a good example, but it's a
    nice application for an embedded system.

    Video editing?

    And these are the type of apps being used to sell systems now...

    3ch

  8. Re: A random thought about swap

    Hi everybody,

    just for fun the following snippet illustrates how a process can extends
    its memory beyond the available RAM, without OS managed swapping,
    provided the mmap() system call.

    #include
    #include
    #include
    #include
    #include
    #include
    #include
    #include
    #include

    void * mmap_malloc(unsigned long size)
    {
    void * ptr = malloc(size);
    /* not surprisingly on my computer with only 1 Gb the above call will
    fails in this example */
    if(ptr)
    return ptr;

    /* open a swap file - name should be randomized */
    int fd = open("/tmp/toto.swap",O_RDWR | O_CREAT,S_IRUSR|S_IWUSR);
    if (-1==fd)
    return NULL;
    /* on Linux, mmap() doesn't allow to make the size of the file grow, so
    let get what we need */
    if ( -1 == ftruncate(fd,size) )
    {
    close(fd);
    return NULL;
    }
    /* now map the file in memory */
    ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED,
    fd, 0);

    if(!ptr)
    {
    close(fd);
    return NULL;
    }
    return ptr;
    }

    int main(void)
    {
    /* let request more memory than the computer has RAM */
    void * ptr = mmap_malloc(2000000000);
    if (!ptr)
    {
    perror("mmap_malloc");
    exit(1);
    }

    /* We've got our virtual memory ! */

    /* let write something uninterresting in it */
    {
    int i;
    for(i=0;i *(((char*)ptr)+i)=0xFF;
    }

    return 0;
    }

    This code is obviously quite rough; something more elaborated could be
    implemented by the libc, to make it transparent to the caller.

    Please note that I haven't tested it on minix itself but on Linux.

    AFAIK mmap() is not available in minix 3, or is it ?

    Sylvain

  9. Re: A random thought about swap

    On Tue, 30 Jan 2007, __guido__@web.de wrote:

    > Exactly.
    > If an application needs to juggle huge amounts of data, how do people
    > get the idea the kernel will do a better job of it than the
    > application programmer, who knows the access patterns?


    If you follow that idea then almost every service offerd by the OS could
    be done better in the application program. Does that mean the ideal is
    MSDOS?

    And every program that might need to swap has to have that code in it.
    Since there is no swap OS all those copies of swap code must stay in
    memory [ya can't swap out the swap code!]. Sounds like bloat to me.

    If it benifits you app to keep it's own swap file, do it! If the benifit
    is small it may not be worth it and using a standard method that happens
    to be offered by the OS might not be that bad. Disk space is "too cheap to
    meter" these days too, even more so that memory.

    > If you are short on RAM, you should have bought a 10% weaker CPU and
    > got the RAM you need from the money saved.

    ? Often the "weaker" CPU won't handle as much memory. And how do you know
    when you buy a machine how much memory you might need some day? Or speed?
    What if after buying the weaker large memory machine I get handed a
    processor bound task? Sometimes it's all guess & compromise.

    3ch


  10. Re: A random thought about swap

    On Jan 30, 9:11 am, __guid...@web.de wrote:

    > swap partitions used to be faster than swap files on Linux for
    > almost a decade. But this is no longer true for recent 2.6
    > kernels AFAIK. The layer of indirection for swap files has
    > been eliminated.


    What you say may be true, I really haven't been following Linux kernel
    developments. But the only way I can understand this working is if
    file space is pre-allocated with contiguous blocks, so the swap space
    can be treated as just a chunk of raw disk space -- effectively no
    different from a partition, except the space allocation is not in the
    partition table but somewhere else, and the file system understands
    the space is not available for ordinary files.

    I believe the best way to do swapping is to use a separate physical
    disk, that is, a separate spindle. If the swap space is a partition on
    a disk that does not have any file systems used by the currently-
    running operating system, then seeks will be independent on the swap
    disk and on the disk containing the file systems in use. Whether the
    space is allocated as the whole disk, a partition on the disk, or a
    pre-allocated large file is not as important as whether the data is
    read and written by an independent r/w head.

    - Al


  11. Re: A random thought about swap

    On 31 Jan., 15:29, "Al Woodhull" wrote:

    >[...] -- effectively no
    > different from a partition, except the space allocation is not in the
    > partition table but somewhere else, and the file system understands
    > the space is not available for ordinary files.


    Considering that multi-boot setups aren't exactly rare with Linux,
    that's not so insignificant a difference. Even if you allow Linux to
    take over the disk, size of swap partition isn't I don't want to
    decide
    at install time, especially since it can not be increased later
    without
    major hassle. And you need be an expert to make an iformed decision.

    The size of the swap file can be changed any time without reboot.
    There is a swapd program available to do this. But I don't see the
    point
    with modern disk's abundant sizes, Just make it twice as big as your
    RAM
    or something and forget about it. Maybe change as you add RAM.

    >
    > I believe the best way to do swapping is to use a separate physical
    > disk, that is, a separate spindle. If the swap space is a partition on
    > a disk that does not have any file systems used by the currently-
    > running operating system, then seeks will be independent on the swap
    > disk and on the disk containing the file systems in use. Whether the
    > space is allocated as the whole disk, a partition on the disk, or a
    > pre-allocated large file is not as important as whether the data is
    > read and written by an independent r/w head.


    Sure. But going to great length to justify swap.
    I don't say swap isn't usefull in some situations.
    But I doubt those apply to the purposes MINIX3 is built.
    Again, if I can choose, I'd rather double my RAM.

    A swap implementation will be available for MINIX3 sometime,
    whatever we are discussing here, no doubt.
    As long as the kernel design isn't compromised for it and it will
    be an optional service, that shouldn't bother anyone.
    I know the Linux kernel underwent massive changes, when this
    feature was added, to cater for it. That's the idea I don't like.

    Guido


+ Reply to Thread