Memory architecture and internals? - Minix

This is a discussion on Memory architecture and internals? - Minix ; Hi, I'm curious on Minix 3's memory architecture and internals on x86 specifically, but also in general. Is there a resource for this? I gather that there's no virtual memory-- primitive flat access memory architecture, one monolythic memory space. Is ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: Memory architecture and internals?

  1. Memory architecture and internals?

    Hi, I'm curious on Minix 3's memory architecture and internals on x86
    specifically, but also in general. Is there a resource for this?

    I gather that there's no virtual memory-- primitive flat access memory
    architecture, one monolythic memory space. Is this accurate? How far
    will virtual memory eventually go? Will everything outside the kernel
    have its own VMA?

    I'm curious as to the difficulty of implementing a memory manager in
    Minix 3 to supply and control virtual memory. I don't understand that
    level of the hardware specifially yet; I know about TLB, cache
    flushing, and multi-level page tables in the same way a person knows
    about valves and pistons and crank shafts in a car and what they do
    yet can't build an engine due to lack of practical knowledge. This is
    a specific area of interest to me.

  2. Re: Memory architecture and internals?

    > I gather that there's no virtual memory-- primitive flat access memory
    > architecture, one monolythic memory space. Is this accurate? How far
    > will virtual memory eventually go? Will everything outside the kernel
    > have its own VMA?
    >
    > I'm curious as to the difficulty of implementing a memory manager in
    > Minix 3 to supply and control virtual memory. I don't understand that
    > level of the hardware specifially yet; I know about TLB, cache
    > flushing, and multi-level page tables in the same way a person knows
    > about valves and pistons and crank shafts in a car and what they do
    > yet can't build an engine due to lack of practical knowledge. This is
    > a specific area of interest to me.


    There is no virtual memory, but that does not mean that there is a
    single flat memory space. Minix uses segments to control access to
    meory. Each process has its own segments and only the kernel is capable
    of accessing any physical addresses (well... in theory, since the root
    can open the memory device to access physical memory as a read/write
    file).

    The virtual memory implementation is not yet there, and you should
    specifically ask the person working on it if you want to find out more.
    From what I heard, I think it will use paging to allow more flexibility
    and efficiency in allocating physical memory. It will not implement
    swapping.

    For more information on how virtual memory is done on Intel, consult
    the Intel Architecture Software Developers Manual:
    http://www.intel.com/design/pentiumi...als/243190.htm (vol 1)
    http://www.intel.com/design/pentiumi...als/243191.htm (vol 2)
    http://www.intel.com/design/pentiumi...als/243192.htm (vol 3)
    (this version is old, but has all the interesting stuff and is less
    complex than recent versions)

    For more information on the implementation of Minix, get "Operating
    Systems Design and Implementation" by Andy Tanenbaum:
    http://www.pearsonhighered.com/acade...388,00%2Ben-US
    S_01DBC.html

    --
    With kind regards,
    Erik van der Kouwe

  3. Re: Memory architecture and internals?

    On Jun 10, 2:32*am, "Erik van der Kouwe" few.vu.nl>
    wrote:
    > There is no virtual memory, but that does not mean that there is a
    > single flat memory space. Minix uses segments to control access to
    > meory. Each process has its own segments and only the kernel is capable
    > of accessing any physical addresses (well... in theory, since the root
    > can open the memory device to access physical memory as a read/write
    > file).
    >


    Okay, so it's not a great model, but it attempts to isolate memory
    access. I'm not sure how well this works, not really so familiar with
    32-bit protected mode segments. Just the basics.

    > The virtual memory implementation is not yet there, and you should
    > specifically ask the person working on it if you want to find out more.
    > From what I heard, I think it will use paging to allow more flexibility
    > and efficiency in allocating physical memory. It will not implement
    > swapping.
    >


    I'd rather see a swap service and have VM communicate with it. Which
    is probably a good idea anyway.

  4. Re: Memory architecture and internals?

    > > There is no virtual memory, but that does not mean that there is a
    > > single flat memory space. Minix uses segments to control access to
    > > meory. Each process has its own segments and only the kernel is
    > > capable of accessing any physical addresses (well... in theory,
    > > since the root can open the memory device to access physical memory
    > > as a read/write file).
    > >

    >
    > Okay, so it's not a great model, but it attempts to isolate memory
    > access. I'm not sure how well this works, not really so familiar with
    > 32-bit protected mode segments. Just the basics.


    I don't see how that is not a great model. It not only attempts but
    also succeeds in isolating memory access, and it succeeds in avoiding
    vulnerabilities using code modification without needing NX.

    It is not strange for the root to have full access to hardware; AFAIK
    it is like this in other Unices as well.

    Paging can be done on top of segmenting without the applications even
    noticing, so the model is quite future-proof.

    > > The virtual memory implementation is not yet there, and you should
    > > specifically ask the person working on it if you want to find out
    > > more. From what I heard, I think it will use paging to allow more
    > > flexibility and efficiency in allocating physical memory. It will
    > > not implement swapping.
    > >

    >
    > I'd rather see a swap service and have VM communicate with it. Which
    > is probably a good idea anyway.


    I suppose everyone would prefer to also have swapping, but for now VM
    would be great by itself. I think the reasoning for not including
    swapping is that it substantially increases complexity while not
    yielding important benefits. Moreover it is not needed for education
    and embedded systems, which are intended to be Minix main ways of use.

    The point of not yielding important benefits may need some elaboration:
    RAM is growing in size and dropping in price very quickly. New
    computers will generally have at least 1GB now, but often more.
    Applications on Minix are not getting more memory hungry at the same
    pace as, for example, new Windows versions and their applications do.
    When VM allocates memory only when needed, it removes the waste of
    memory that is currently taking place. When this is done, Minix apps
    will probably not suffer from memory shortages for a long time and
    hence do not need swapping.

    --
    With kind regards,
    Erik van der Kouwe

  5. Re: Memory architecture and internals?

    On Jun 14, 3:59*am, "Erik van der Kouwe" few.vu.nl>
    wrote:

    > I don't see how that is not a great model. It not only attempts but
    > also succeeds in isolating memory access, and it succeeds in avoiding
    > vulnerabilities using code modification without needing NX.


    as I understand, every process has full access to the same memory
    space, with segments set up so that that access is --- rather than rw
    or --x; this means the full length of VMA (or in this case, physical
    memory?) is shared by all applications; something needing 2G of VMA
    can't exist with something else needing 2G of VMA, and likely will
    have to deal with memory fragmentation anyway. Consideration for a
    general purpose system that might run i.e. a database.

    >
    > It is not strange for the root to have full access to hardware; AFAIK
    > it is like this in other Unices as well.
    >


    Nods, /dev/mem

    > Paging can be done on top of segmenting without the applications even
    > noticing, so the model is quite future-proof.
    >


    Yes, if it's one shared virtual memory area.

    > > I'd rather see a swap service and have VM communicate with it. *Which
    > > is probably a good idea anyway.

    >
    > I suppose everyone would prefer to also have swapping, but for now VM
    > would be great by itself. I think the reasoning for not including
    > swapping is that it substantially increases complexity while not
    > yielding important benefits. Moreover it is not needed for education
    > and embedded systems, which are intended to be Minix main ways of use.
    >


    Of course. I'm saying, I'd rather see the "swapping" infrastructure
    separated from general VM, rather than tightly integrated. This
    allows for things like memory compression and swap caching to have
    their own implementation layer, without having a complex body of code
    to try to pry apart and refactor before you can add such a thing or
    any other new idea you have (add compressed memory to the Linux kernel
    and you'll get it).

    > The point of not yielding important benefits may need some elaboration:
    > RAM is growing in size and dropping in price very quickly. New
    > computers will generally have at least 1GB now, but often more.
    > Applications on Minix are not getting more memory hungry at the same
    > pace as, for example, new Windows versions and their applications do.
    > When VM allocates memory only when needed, it removes the waste of
    > memory that is currently taking place. When this is done, Minix apps
    > will probably not suffer from memory shortages for a long time and
    > hence do not need swapping.
    >


    I would like to see Minix develop as a full blown desktop OS, and I
    believe there is an advantage to making a microkernel into a general
    purpose OS; it's many reusable parts, there's no "slimming it down for
    a special purpose" because every extension for desktop use is made in
    isolated parts. I don't see why there should be an intentional
    artificial limit for Minix 3's functionality; if students or outsiders
    want to bring it up to be a GPOS, then why not?

    There's no reason, for example, that Minix shouldn't have realtime
    response, good for desktop use (especially gaming!), super computer
    grids, and micro-controller embedded systems like wifi/bluetooth
    chipsets playing with a radio chip; minix as a full blown desktop OS
    is effectively minix for wifi firmware, minus a couple control
    services, plus a bunch of desktop-oriented services like VM and swap
    etc. Because of the simplicity of just picking out the parts you need
    and extending the system in any way necessary, I see no reason a fully
    developed microkernel isn't an excellent choice for a general purpose
    OS.

    > --
    > With kind regards,
    > Erik van der Kouwe



  6. Re: Memory architecture and internals?

    > > I don't see how that is not a great model. It not only attempts but
    > > also succeeds in isolating memory access, and it succeeds in
    > > avoiding vulnerabilities using code modification without needing NX.

    >
    > as I understand, every process has full access to the same memory
    > space, with segments set up so that that access is --- rather than rw
    > or --x; this means the full length of VMA (or in this case, physical
    > memory?) is shared by all applications; something needing 2G of VMA
    > can't exist with something else needing 2G of VMA, and likely will
    > have to deal with memory fragmentation anyway. Consideration for a
    > general purpose system that might run i.e. a database.


    On IA32, memory can only be accessed through segments and processes can
    only use those segments set up for them.

    Since Minix does not allow segments to overlap and shares only
    read-only code segments between processes, there is no way one process
    can read from or write to the data segment of another process other
    than by ways controlled by the OS (that is, devices like /dev/mem and
    debugging functions).

    You seem to think that segment bounds are not checked, but they are. A
    simple test reveals this:

    #include

    int main(void)
    {
    printf("%d\n", *(int *) 0x30000);
    return 0;
    }

    This will segfault when compiled and run since the default space for
    dynamic data and stack is only 0x20000 bytes.

    > > Paging can be done on top of segmenting without the applications
    > > even noticing, so the model is quite future-proof.

    >
    > Yes, if it's one shared virtual memory area.


    I do not see why that would be needed.

    > I would like to see Minix develop as a full blown desktop OS, and I
    > believe there is an advantage to making a microkernel into a general
    > purpose OS; it's many reusable parts, there's no "slimming it down for
    > a special purpose" because every extension for desktop use is made in
    > isolated parts. I don't see why there should be an intentional
    > artificial limit for Minix 3's functionality; if students or outsiders
    > want to bring it up to be a GPOS, then why not?
    >
    > There's no reason, for example, that Minix shouldn't have realtime
    > response, good for desktop use (especially gaming!), super computer
    > grids, and micro-controller embedded systems like wifi/bluetooth
    > chipsets playing with a radio chip; minix as a full blown desktop OS
    > is effectively minix for wifi firmware, minus a couple control
    > services, plus a bunch of desktop-oriented services like VM and swap
    > etc. Because of the simplicity of just picking out the parts you need
    > and extending the system in any way necessary, I see no reason a fully
    > developed microkernel isn't an excellent choice for a general purpose
    > OS.


    It sure might be, but this does not fit well with Minix' goals. If you
    are serious about this you would be best off forking the project, since
    the VU staff which controls it closely guards Minix against unwanted
    bulk and complexity. Maybe you could call it Maxix?

    --
    With kind regards,
    Erik van der Kouwe

  7. Re: Memory architecture and internals?

    On Jun 15, 1:40*pm, "Erik van der Kouwe" few.vu.nl>
    wrote:
    >
    > On IA32, memory can only be accessed through segments and processes can
    > only use those segments set up for them.
    >
    > Since Minix does not allow segments to overlap and shares only
    > read-only code segments between processes, there is no way one process
    > can read from or write to the data segment of another process other
    > than by ways controlled by the OS (that is, devices like /dev/mem and
    > debugging functions).


    Yes. My concern here is that all processes share the same VMA, and so
    if one process uses a chunk of memory then that necessarily precludes
    another process from allocating memory in those addresses. A
    developer of the Build engine once said, "You can't port Build to a 32-
    bit OS because it needs a contiguous chunk of memory 8 megs long, and
    that just won't be available in one straight block in a multi-tasking
    OS because of memory fragmentation." He didn't understand how virtual
    memory worked, but in this case such an assertion might hold correct
    in random but probable cases.

    >
    > > > Paging can be done on top of segmenting without the applications
    > > > even noticing, so the model is quite future-proof.

    >
    > > Yes, if it's one shared virtual memory area.

    >
    > I do not see why that would be needed.
    >


    because if it's referencing non-virtual memory (i.e. physical memory),
    you can't pull the page table mapping out and get a fault on attempt
    to access unmapped address.

    >
    > It sure might be, but this does not fit well with Minix' goals. If you
    > are serious about this you would be best off forking the project, since
    > the VU staff which controls it closely guards Minix against unwanted
    > bulk and complexity. Maybe you could call it Maxix?
    >


    This seems like a good idea. I need to find some programmer friends
    who would like to though... large one-man projects FAIL. HARD.
    Doubly so since, as academically interested as I am, I haven't the
    knowledge nor experience to write an OS (I mean, I could do some,
    but ... damn).


    > --
    > With kind regards,
    > Erik van der Kouwe- Hide quoted text -
    >
    > - Show quoted text -



+ Reply to Thread