DosAllocMem and OBJ_ANY - OS2

This is a discussion on DosAllocMem and OBJ_ANY - OS2 ; Hallo, > > Thanks, Lars. Sorry for my DD illiteracy, but for me, what you > discuss looks like "memory allocated by DD and passed to a process". > Should not the "possible breakage" scenario be the opposite: "memory > ...

+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3
Results 41 to 54 of 54

Thread: DosAllocMem and OBJ_ANY

  1. Re: DosAllocMem and OBJ_ANY

    Hallo,
    >
    > Thanks, Lars. Sorry for my DD illiteracy, but for me, what you
    > discuss looks like "memory allocated by DD and passed to a process".
    > Should not the "possible breakage" scenario be the opposite: "memory
    > allocated by process and passed to DD" ?


    It depends on what the DD does and how it operates. A DD might provide an
    API that allocates memory on behalf of an application and passes the linear
    address to the application with full addressability by the application (and
    the DD). But you are right, for sound drivers, display drivers etc. it is
    the application that allocates the memory and passes it to the DD.
    On the other hand, you can call an IOCTL (SCREENDD_GETLINEARACCESS) to get
    returned the linear frame buffer address. If you call that IOCTL, the DD
    will make the frame buffer addressable out of the context of the process
    calling that IOCTL. It would use either VMGlobalToProcess or PhysToUVirt to
    make that frame buffer addressable by the calling process. In practice that
    IOCTL might be restricted to the VGA aperture 0xA0000-0xBFFFF because
    screen01.sys/screen02.sys are VERY old, in theory it could allow you to
    address the whole linear frame buffer (maybe 128 Megs in size or more) and
    ,in case of VMGlobalToProcess, addressable with high memory addresses (to
    not consume space in low memory).


    For that case you state you have these options, even if you use high memory
    addresses:
    1.) You can use (AllocGDTSelector/)LinToGDTSelector(/FreeGDTSelector)
    DeviceHelper routines to get a GDT selector and consequently a 16:16 pointer
    from a linear address passed into the device driver. The drawback is that
    you are restricted to 64 kBytes per GDT selector which really is a limiting
    factor as GDT selectors are a scarce resource,global to the whole system and
    there are only (65536 / 8 -1) = 8191 GDT selectors available overall (don't
    forget that a whole bunch of them are occupied by API call gates, device
    driver/IFS code and data segments etc.)

    2.) The much better alternative is to write assembler routines that make use
    of 48-bit far pointer addressing (16:32 pointers). They will load the DS
    FLAT selector into the segment register (the FLAT selector descriptor
    describes a memory range starting at linear address 0 with a limit of
    0xFFFFFFFF) and then use the full 32-bit offset (= the passed linear
    address). That is possible and I think it's often done in newer device
    drivers using a mixed 16-bit/32-bit programming model. There is no need to
    allocate any new GDT selector and the buffer can be arbitrarily large.
    Older drivers will of course not support that as they won't expect really
    large buffers (remember Gates: "no PC will ever need more than 640 kBytes"
    ?)
    Instead they will employ 1.) and they would usually have a pretty tight size
    limit.

    Either way, you should be able to pass high memory linear addresses to the
    driver but 1.) is very limiting if you have large buffers.


    > Or are you discussing memory-pre-digestion by DD to translate memory
    > addresses as-usable-by-ring3 into something usable-by-ring-0?


    No. DDs can always address memory that is addressable from ring3. No
    additional steps are necessary.

    Lars



  2. Re: DosAllocMem and OBJ_ANY

    >> Or are you discussing memory-pre-digestion by DD to translate memory
    >> addresses as-usable-by-ring3 into something usable-by-ring-0?

    >
    > No. DDs can always address memory that is addressable from ring3. No
    > additional steps are necessary.


    Oops, that is not correct. A DD needs to use VMProcessToGlobal to map
    process memory to global address space.
    Note: Global address space is everything beyond what is specified with
    VIRTUALADDRESSLIMIT config.sys setting, that is, everything above
    per-process and shared high memory address space.

    Lars



  3. Re: DosAllocMem and OBJ_ANY

    > low memory if the video BIOS code is pure 16-bit. If the user buffer is in
    > high memory, the caller will have to use DosAliasMem to get a memory alias
    > in low memory.


    Yes, but what makes you think the caller has to be the application?

    The whole functionality and the interface looks more like something IBM needed
    mainly for internal purposes. I could imagine that the critical areas within
    the API (e.g. GPI) call DosAliasMem by themselves if necessary. Maybe they forgot
    to adapt some APIs (the ones that show problems). This would make much more sense
    than moving the task to the application level. I'd be astonished if IBM would
    break the rules for their well designed API so badly.

    The technical background discussed by you together with the DosAliasMem description
    requires for my point of view a very deep knowledge about the system architecture.
    I don't think that this is appropriate for a higher level API which usually hides
    these details. An average developer should not have to deal with these details and
    usually cannot deal with them because of either lack of knowledge or just because
    he/she cannot know how the API is internally implemented.
    Look at the parameters of DosAliasMem, based on what should an application developer
    decide which parameters to chose? It is not possible without knowing the API
    implementation and their requirements.
    For driver level development this might be a bit different as here the knowledge is
    probably required anyway but when drivers call APIs they also cannot look into a
    crystal ball to know what the API internally requires.

    Summarizing the above I think it makes the most sense that IBM targeted DosAliasMem
    for drivers (or direct driver callers) to allow them to map high memory for their
    purpose because they know when and how this would be required while the normal
    application cannot know this.

  4. Re: DosAllocMem and OBJ_ANY

    Hallo,

    > The whole functionality and the interface looks more like something IBM
    > needed
    > mainly for internal purposes. I could imagine that the critical areas
    > within
    > the API (e.g. GPI) call DosAliasMem by themselves if necessary. Maybe they
    > forgot
    > to adapt some APIs (the ones that show problems). This would make much
    > more sense
    > than moving the task to the application level. I'd be astonished if IBM
    > would
    > break the rules for their well designed API so badly.


    Don't forget that the GPI is much older than high memory.High memory is not
    really well designed into the OS at all. That's what makes it so awkward,
    sad as it is ...

    > Look at the parameters of DosAliasMem, based on what should an application
    > developer
    > decide which parameters to chose? It is not possible without knowing the
    > API
    > implementation and their requirements.


    My questions were just for technical interest. You don't have to know in
    detail how it works. You just have to know that DosAliasMem will give you an
    alias in low memory region for memory allocated in high memory region.

    > For driver level development this might be a bit different as here the
    > knowledge is
    > probably required anyway but when drivers call APIs they also cannot look
    > into a
    > crystal ball to know what the API internally requires.
    >
    > Summarizing the above I think it makes the most sense that IBM targeted
    > DosAliasMem
    > for drivers (or direct driver callers) to allow them to map high memory
    > for their
    > purpose because they know when and how this would be required while the
    > normal
    > application cannot know this.


    DosAliasMem can only be called by Ring-3 code, that is, application level
    code. Not by device drivers.

    Lars



  5. Re: DosAllocMem and OBJ_ANY

    > My questions were just for technical interest. You don't have to know in
    > detail how it works. You just have to know that DosAliasMem will give you an
    > alias in low memory region for memory allocated in high memory region.


    But one has to know which parameters have to be chosen. And this is not
    really obvious based on the documentation. It is not even documented for
    what the function is needed, no link to DosAllocMem or other hints.

    If this is really thought to be used by applications the way we think,
    most implementations would end up as follows (I mean almost always!):

    Every call to DosAllocMem with OBJ_ANY will be followed by a DosAliasMem
    if the buffer is passed to an API call. In fact they could have built the
    mechanism then directly into DosAllocMem. In a simple application it might
    be possible to follow up where the allocated buffer finally is used but in
    a more complex application this is almost impossible to predict, especially
    if C++ or something similar is used to hide implementation details.
    For the cases you specifically know by design that the buffer will never be
    passed to a system API, a suppression flag for DosAllocMem would have been
    much more usable. The default would be to always create the alias which would
    automatically put the application on the safe side.

    Example:
    Assume there is an object that loads a bitmap file and provides it later
    on for instance to algorithms. Well, here the alias would not be required.
    BUT, the bitmap data could also be used for drawing. In this case the alias
    would be required to be on the safe side. Because the bitmap file handler
    cannot know for what the buffer is finally used, it would always need to
    alias the buffer.
    The other way round would be thinkable as well, I mean that the user of the
    buffer (the algorithms or the drawing code) creates the alias. There are 2
    problems. 1st there seems to be no unalias functionality, the alias is only
    freed if the buffer is freed. So the buffer would be at least aliased once
    (assuming multiple DosAliasMem reuses the selector for multiple calls).
    2nd is that the buffer address must fulfill certain alignment requirements.
    But the buffer user has no control of this. It would need to imply that
    DosAllocMem has been used and not malloc from the compiler runtime library.
    You don't want this!

    Ahhhhhhhha..., and maybe scenario 2 describes the reason why the alias cannot
    be created within the system API when needed. The API would have the same problem.
    The buffer passed to the API does not always have to be aligned as required by
    DosAliasMem.

    Anyway, in both scenarios you end up having an alias, so why is it not implied
    in DosAllocMem when OBJ_ANY is used? Is the waste of resources really so big
    to rectify the significantly increased complexity?


  6. Re: DosAllocMem and OBJ_ANY

    In <47bcc73f$0$23013$9b4e6d93@newsspool1.arcor-online.net>, on 02/21/2008
    at 01:34 AM, Heiko Nitzsche said:

    Hi,

    >Anyway, in both scenarios you end up having an alias, so why is it not
    >implied in DosAllocMem when OBJ_ANY is used? Is the waste of resources
    >really so big to rectify the significantly increased complexity?


    All of what you say is true, but you have to remember the probable context
    in which IBM created the API. They had a large base of installed code
    that they wanted to avoid breaking at almost any cost. At the same time
    they had a need for the functionality DosAliasMem provided. The result is
    an API that has limitiations but that implements what they needed. I
    really don't know when the API was implemeneted, but TTBOMK it was private
    until the Addendum was published.

    I tend to read between the lines. Since the docs show an example of using
    DosAliasMem to implement buffers that are read-only to the givee and
    read-write to the owner, this is the reason the API exists. The API was
    implemented in a way that affected a minimal amount of existing kernel
    code. The fact that the alias needed to be in low memory and use an LDT
    descriptor was an unavoidable side effect to the implementation.

    Steven

    --
    --------------------------------------------------------------------------------------------
    Steven Levine MR2/ICE 3.00 beta 11pre #10183
    eCS/Warp/DIY/14.103a_W4 www.scoug.com irc.ca.webbnet.info #scoug (Wed 7pm PST)
    --------------------------------------------------------------------------------------------


  7. Re: DosAllocMem and OBJ_ANY

    [A complimentary Cc of this posting was sent to
    Heiko Nitzsche
    ], who wrote in article <47bca203$0$377$9b4e6d93@newsspool2.arcor-online.net>:

    > The whole functionality and the interface looks more like something
    > IBM needed mainly for internal purposes. I could imagine that the
    > critical areas within the API (e.g. GPI) call DosAliasMem by
    > themselves if necessary. Maybe they forgot to adapt some APIs (the
    > ones that show problems). This would make much more sense than
    > moving the task to the application level. I'd be astonished if IBM
    > would break the rules for their well designed API so badly.


    Your suspicions are correct. There is about 60 OS/2 APIs which are
    known to be safe to use with high memory. There are wrappers about a
    dozen other APIs. The remaining 6000 APIs are NOT (known to be) safe
    to use with high memory.

    It is this simple: high memory as implemented on OS/2 is not intended
    for general use.

    Hope this helps,
    Ilya

  8. Re: DosAllocMem and OBJ_ANY

    [A complimentary Cc of this posting was NOT [per weedlist] sent to
    Lars Erdmann
    ], who wrote in article <47bc982e$0$374$9b4e6d93@newsspool2.arcor-online.net>:
    > > Or are you discussing memory-pre-digestion by DD to translate memory
    > > addresses as-usable-by-ring3 into something usable-by-ring-0?

    >
    > No. DDs can always address memory that is addressable from ring3. No
    > additional steps are necessary.


    As I said, my understanding of DD architecture is very limited, but
    STILL I think you must be wrong. ;-)

    Different processes have different virtual memory, so if DD wants to
    read/write a buffer supplied by the application, and it needs to do it
    in interrupt time, it must somehow massage the address, right?

    The virtual addressing in use of interrupt time is just the addressing
    of the "currently running thread", right?

    Thanks,
    Ilya

  9. Re: DosAllocMem and OBJ_ANY

    On Thu, 21 Feb 2008 06:52:28 UTC, Ilya Zakharevich
    wrote:

    > Different processes have different virtual memory, so if DD wants to
    > read/write a buffer supplied by the application, and it needs to do it
    > in interrupt time, it must somehow massage the address, right?
    >
    > The virtual addressing in use of interrupt time is just the addressing
    > of the "currently running thread", right?


    If a driver needs access to applicaton memory at interrupt time (or to
    be more correct: outside the application task's context), it needs to
    map
    this memory into the system area using VMProcessToGlobal. The result is
    an alias that is accessable independend of the current context. The
    system
    area is located above VIRTUALADDRESSLIMIT and that is why we get
    problems
    when people are setting VIRTUALADDRESSLIMIT too high.

    BTW, many drivers don't need this addressability during interrupt,
    because modern DMA-driven hardware will require physical addresses
    either in form of a contiguous block or an array of page lists.



    --
    Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
    http://www.s-t.de
    Please remove all characters left of the "R" in my email address


  10. Re: DosAllocMem and OBJ_ANY

    On a pleasant day while strolling in
    comp.os.os2.programmer.misc, a person by the name of
    Ruediger Ihle exclaimed:
    > On Mon, 18 Feb 2008 19:20:31 UTC, no_spam@outgoing.net (Jeffrey Smick)
    > wrote:
    >
    > > DosAliasMem....
    > > Does not exist in help! You spend time reading .h files, I guess!

    >
    > It's documented in \book\addenum.inf



    Amusing that the remaining OS/2 developers could not
    figure out how to update the (rather complex) toolkit
    help files

    > DosSubAllocMem is evil anyway ;-).


    But the only way to suballocate shared memory ?


    --
    aaronl at consultant dot com
    For every expert, there is an equal and
    opposite expert. - Arthur C. Clarke

  11. Re: DosAllocMem and OBJ_ANY

    [A complimentary Cc of this posting was sent to
    Aaron Lawrence
    ], who wrote in article :
    > > DosSubAllocMem is evil anyway ;-).


    > But the only way to suballocate shared memory ?


    In ring 3, it is trivial to do. IIRC, EMX may even have the already
    shipped API to suballocate any pool...

    Hope this helps,
    Ilya

  12. Re: DosAllocMem and OBJ_ANY

    > I tend to read between the lines. Since the docs show an example of using
    > DosAliasMem to implement buffers that are read-only to the givee and
    > read-write to the owner, this is the reason the API exists. The API was
    > implemented in a way that affected a minimal amount of existing kernel
    > code. The fact that the alias needed to be in low memory and use an LDT
    > descriptor was an unavoidable side effect to the implementation.


    And meanhwile I think I figured out why the alias cannot be done by
    default in DosAllocMem. Obviously you can only map up to 512MB in sum
    for the whole process, not per block. I tried to create an alias for two
    370MB blocks, first one was OK but for the second I got OUT_OF_MEMORY.
    So this mechanism is only usable for smaller chunks used temporary.
    And this explains why the API is separated.

    My scenario is a bit different. I allocate a huge block (512MB and up),
    put a bitmap in and draw a rectangle from it for each WM_PAINT message.
    With some address calculations it might be possible to fulfill the alignment
    requirements to map a partial linear area of the block but unfortunately
    this does not help for bitmaps. An additional render buffer would be
    required and some code that extracts the drawing region data to it.

    As this somehow contradicts the idea to reduce memory usage and also hits
    performance (goal is quick interactive responsive), I'll first stick with
    drawing from highmem and monitor whether really problems show up and then
    go the other way. With SNAP, the latest Matrox drivers and Panorama there
    seems to be no problem, otherwise this would have been an issue for Firefox
    and SeaMonkey already. I think almost all users use one of the drivers above.
    I'll try with plain VGA driver but I think nobody will use it for more than
    installation and especially not for working with graphics.

  13. Re: DosAllocMem and OBJ_ANY

    Hallo Rudi,

    > If a driver needs access to applicaton memory at interrupt time (or to
    > be more correct: outside the application task's context), it needs to
    > map
    > this memory into the system area using VMProcessToGlobal. The result is
    > an alias that is accessable independend of the current context.


    Does that also mean that if I only need addressability at task time (for
    example if I can handle all access to a user buffer provided in an IOCTL
    from within the strat routine before returning from it) that I DO NOT need
    to call VMProcessToGlobal ? Does the Ring-0 DS FLAT selector also work with
    "process space" addresses ?

    More concrete:
    FAT32.IFS should really use 48-bit far addresses to access the cache. That
    would allow much larger cache sizes than 2 Megs as no allocation of GDT
    selectors would be necessary.
    "Today", FAT32.IFS allocates cache with VMAlloc using VMDHA_FIXED |
    VMDHA_CONTIG |VMDHA_USEHIGHMEM flags (is fixed and contiguous really
    necessary ? I doubt it.) but NOT with the VMDHA_PROCESS flag. That is
    already allocated in system arena address space, therefore addressable out
    of any process context, correct ?
    Can you briefly repeat the implications of the VMDHA_USEHIGHMEM flag ? I
    remember something about EARLYMEMINIT ...

    Lars



  14. Re: DosAllocMem and OBJ_ANY

    On Sun, 2 Mar 2008 03:36:27 UTC, "Lars Erdmann"
    wrote:

    Hi Lars,

    > Does that also mean that if I only need addressability at task time (for
    > example if I can handle all access to a user buffer provided in an IOCTL
    > from within the strat routine before returning from it) that I DO NOT need
    > to call VMProcessToGlobal ? Does the Ring-0 DS FLAT selector also work with
    > "process space" addresses ?


    Yes.


    > "Today", FAT32.IFS allocates cache with VMAlloc using VMDHA_FIXED |
    > VMDHA_CONTIG |VMDHA_USEHIGHMEM flags (is fixed and contiguous really
    > necessary ? I doubt it.)


    Depends on what type of DMA operations are to be expected to/from that
    area. The "fixed" attribute is probably O.K. to prevent the FAT32 cache
    from beeing paged out.


    > but NOT with the VMDHA_PROCESS flag. That is already allocated in
    > system arena address space, therefore addressable out of any process
    > context, correct ?


    Yes. Use DOS32FLATSEL.


    > Can you briefly repeat the implications of the VMDHA_USEHIGHMEM flag ?
    > I remember something about EARLYMEMINIT ...


    You remember correctly. There appear to be two memory pools. One below
    16M and one above. When a driver allocates memory during INIT or
    INIT_COMPLETE and VMDHA_USEHIGHMEM is not set, the request is fullfilled
    by taking it from the "below 16M" pool. This made good sense at times,
    where slave DMA was state of the art technique. Today its gets more and
    more problematic, because this pool is becoming a scarce ressource. This
    can be avoided by specifying VMDHA_USEHIGHMEM. In this case, the
    allocation
    is taken from the "above 16M" pool.

    So far so good.

    But unfortunately the internal workings of EARLYMEMINIT create an
    unexpected side effect. The setting basically "pumps up" the
    "below 16M" pool to full size of installed memory by reducing the
    "above 16M" pool to zero. The means, that allocations will fail,
    if EARLYMEMINIT is active and VMDHA_USEHIGHMEM is set.

    The obvious workaround is to do the allocation first with
    VMDHA_USEHIGHMEM set and if that fails retry it without.




    --
    Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
    http://www.s-t.de
    Please remove all characters left of the "R" in my email address


+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3