DosAllocMem and OBJ_ANY - OS2

This is a discussion on DosAllocMem and OBJ_ANY - OS2 ; I'm currently changing most of my programs to use High Memory by using OBJ_ANY for DosAllocMem. So far on eCS this works fine and I can allocate more than 512MB just fine. I could not find any hint in OS2TK ...

+ Reply to Thread
Page 1 of 3 1 2 3 LastLast
Results 1 to 20 of 54

Thread: DosAllocMem and OBJ_ANY

  1. DosAllocMem and OBJ_ANY

    I'm currently changing most of my programs to use High Memory by
    using OBJ_ANY for DosAllocMem. So far on eCS this works fine and
    I can allocate more than 512MB just fine.

    I could not find any hint in OS2TK about backward compatibility.
    Will DosAllocMem on older OS ignore OBJ_ANY and allocate from low
    memory pool as fallback instead (pre Warp 4.5x and Warp 3)?

    If it is not backward compatible, what is the recommended way
    to detect the support?

  2. Re: DosAllocMem and OBJ_ANY

    On Sun, 10 Feb 2008 19:36:18 UTC, Heiko Nitzsche
    wrote:

    > Will DosAllocMem on older OS ignore OBJ_ANY and allocate from low
    > memory pool as fallback instead (pre Warp 4.5x and Warp 3)?


    No.


    > If it is not backward compatible, what is the recommended way
    > to detect the support?


    Retry the allocation without OBJ_ANY.



    --
    Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
    http://www.s-t.de
    Please remove all characters left of the "R" in my email address


  3. Re: DosAllocMem and OBJ_ANY

    >> If it is not backward compatible, what is the recommended way
    >> to detect the support?

    >
    > Retry the allocation without OBJ_ANY.


    Yeah, trial and error is what I expected .

    Do you know if there are restrictions for using HMS memory objects in
    PM or GPI? I especially need this for bitmap creation and GpiDrawBits.

    So far it seems to work, but something more definitive would be highly
    appreciated.

  4. Re: DosAllocMem and OBJ_ANY

    Heiko Nitzsche ha scritto:

    > I'm currently changing most of my programs to use High Memory by
    > using OBJ_ANY for DosAllocMem. So far on eCS this works fine and
    > I can allocate more than 512MB just fine.
    >
    > I could not find any hint in OS2TK about backward compatibility.
    > Will DosAllocMem on older OS ignore OBJ_ANY and allocate from low
    > memory pool as fallback instead (pre Warp 4.5x and Warp 3)?
    >
    > If it is not backward compatible, what is the recommended way
    > to detect the support?


    I used the following code on program initialization to check the availability
    of high memory:

    // check if high memory objects are supported and maximum available memory
    g.rc = DosQuerySysInfo(QSV_VIRTUALADDRESSLIMIT, QSV_VIRTUALADDRESSLIMIT,
    &cbAvailMem, sizeof(ULONG));
    if (g.rc == NO_ERROR)
    {
    g.mode |= FSJ_MEMHIGHOBJ;
    g.rc = DosQuerySysInfo(QSV_MAXHPRMEM, QSV_MAXHPRMEM,
    &cbAvailMem, sizeof(ULONG));
    }
    else
    {
    g.rc = DosQuerySysInfo(QSV_MAXPRMEM, QSV_MAXPRMEM,
    &cbAvailMem, sizeof(ULONG));
    }
    // if cannot get the size of available memory show an error message
    if (g.rc) return handleError(FALSE, FALSE, SZERR_MAXPRMEM);


    --
    bye
    Alessandro

  5. Re: DosAllocMem and OBJ_ANY

    Hi,

    Heiko Nitzsche schrieb:
    > If it is not backward compatible, what is the recommended way
    > to detect the support?


    allocate a small memory object in high memory at the application startup
    and set a global static field depending on the success to OBJ_ANY or
    zero. You can or this field to the flags of DosAllocMem whereever you like.


    Marcel

  6. Re: DosAllocMem and OBJ_ANY

    On Sun, 10 Feb 2008 19:56:47 UTC, Heiko Nitzsche wrote:

    > Do you know if there are restrictions for using HMS memory objects in
    > PM or GPI? I especially need this for bitmap creation and GpiDrawBits.


    GpiDrawBits works when passed high memory stuff, we use that in Mozilla,
    too. If you want to know which functions do not work, take a look at the
    wrapper header in kLibc. (Hmm, just wanted to point you to the online
    repository, but that site doesn't answer at the moment. Tt's shipped as
    os2safe.h with GCC 3.3.5 CSD3.)
    --
    Greetings, | My From: address is valid as is the version without "spam"
    Peter. | I try to find real messages among the spam once a week

  7. Re: DosAllocMem and OBJ_ANY

    >> Do you know if there are restrictions for using HMS memory objects in
    >> PM or GPI? I especially need this for bitmap creation and GpiDrawBits.

    >
    > GpiDrawBits works when passed high memory stuff, we use that in Mozilla,


    Excellent. I assume PM and GPI are pure 32bit and thus should have
    no problem dealing with high memory objects. Dos calls seem to be
    more critical (e.g. I think Vio stuff is still mostly 16bit).

    > too. If you want to know which functions do not work, take a look at the
    > wrapper header in kLibc. (Hmm, just wanted to point you to the online
    > repository, but that site doesn't answer at the moment. Tt's shipped as
    > os2safe.h with GCC 3.3.5 CSD3.)


    I found os2safe.h (have GCC installed) but how is it to be interpreted?

    Are the Safe* functions wrappers for non-safe API functions or do they
    declare them as safe?

    #define DosForceDelete SafeDosForceDelete
    #define DosQueryHType SafeDosQueryHType
    #define DosCreatePipe SafeDosCreatePipe
    #define DosQueryNPHState SafeDosQueryNPHState
    #define DosWaitNPipe SafeDosWaitNPipe
    #define DosSetFilePtr SafeDosSetFilePtr
    #define DosSetFilePtrL SafeDosSetFilePtrL
    #define DosDupHandle SafeDosDupHandle
    #define DosOpenL SafeDosOpenL
    #define DosQueryFHState SafeDosQueryFHState
    #define DosSetDateTime SafeDosSetDateTime
    #define DosStartSession SafeDosStartSession
    #define DosQueryAppType SafeDosQueryAppType
    #define DosWinUpper SafeDosWinUpper

    Assuming the Safe* functions are wrappers to make these API functions
    safe, can I assume that everything else is safe for high memory objects?
    It looks strange that especially DosOpenL is not high mem safe as
    it was introduced quite late. Maybe it is reusing old DosOpen code
    or, wait...ahh its probably due to 16bit IFS interface?

  8. Re: DosAllocMem and OBJ_ANY

    [A complimentary Cc of this posting was NOT [per weedlist] sent to
    Peter Weilbacher
    ], who wrote in article :
    > GpiDrawBits works when passed high memory stuff, we use that in Mozilla,
    > too.


    IIRC what the developer of PMView reported, some Gpi APIs (BitBlit?)
    do not work with high memory. (They work with SOME video drivers,
    though...)

    > If you want to know which functions do not work, take a look at the
    > wrapper header in kLibc.


    There are a small handful of functions which work. All the rest do not.

    AFAIU, the "wrapped" stuff is for APIs people managed to trace
    problems to. The using the unwrapped unsupported stuff is just
    playing russian rulette - OK for some applications, not OK for some others.

    Hoep this helps,
    Ilya

  9. Re: DosAllocMem and OBJ_ANY

    1.) The OS/2 Programming Guide and Reference Addendum lists APIs
    supporting high memory. I don't know if the list is complete.
    2.) Who would care about DosOpenL ? You don't pass large buffers to that
    API so what's the point ?
    3.) There is a way out of the misery: You can use DosAliasMem to get an
    alias in low memory for a high memory object. Then your memory is
    accessible via LDT selectors (and the compatibility region thunking
    algorithm applies).
    You can then safely pass that alias to the API function not supporting
    high memory. Just make sure you DO NOT use SEL_USE32 with DosAliasMem
    (because that would create a 32-bit LDT selector instead of 16-bit LDT
    selectors where 32-bit LDT selectors do not work with 16-bit code). Also
    be aware that LDT selectors are a scarce resource (because all of them
    are normally used to map low memory to 16-bit far pointers in a way that
    the compatibility region thunking algorithm applies). You should free
    the alias as soon as possible.
    If you no longer need the alias address, you use DosFreeMem on the alias
    to free the alias (that WILL NOT free the memory, only the alias).

    Lars

    Heiko Nitzsche schrieb:
    >>> Do you know if there are restrictions for using HMS memory objects in
    >>> PM or GPI? I especially need this for bitmap creation and GpiDrawBits.

    >>
    >> GpiDrawBits works when passed high memory stuff, we use that in Mozilla,

    >
    > Excellent. I assume PM and GPI are pure 32bit and thus should have
    > no problem dealing with high memory objects. Dos calls seem to be
    > more critical (e.g. I think Vio stuff is still mostly 16bit).
    >
    >> too. If you want to know which functions do not work, take a look at the
    >> wrapper header in kLibc. (Hmm, just wanted to point you to the online
    >> repository, but that site doesn't answer at the moment. Tt's shipped
    >> as os2safe.h with GCC 3.3.5 CSD3.)

    >
    > I found os2safe.h (have GCC installed) but how is it to be interpreted?
    >
    > Are the Safe* functions wrappers for non-safe API functions or do they
    > declare them as safe?
    >
    > #define DosForceDelete SafeDosForceDelete
    > #define DosQueryHType SafeDosQueryHType
    > #define DosCreatePipe SafeDosCreatePipe
    > #define DosQueryNPHState SafeDosQueryNPHState
    > #define DosWaitNPipe SafeDosWaitNPipe
    > #define DosSetFilePtr SafeDosSetFilePtr
    > #define DosSetFilePtrL SafeDosSetFilePtrL
    > #define DosDupHandle SafeDosDupHandle
    > #define DosOpenL SafeDosOpenL
    > #define DosQueryFHState SafeDosQueryFHState
    > #define DosSetDateTime SafeDosSetDateTime
    > #define DosStartSession SafeDosStartSession
    > #define DosQueryAppType SafeDosQueryAppType
    > #define DosWinUpper SafeDosWinUpper
    >
    > Assuming the Safe* functions are wrappers to make these API functions
    > safe, can I assume that everything else is safe for high memory objects?
    > It looks strange that especially DosOpenL is not high mem safe as
    > it was introduced quite late. Maybe it is reusing old DosOpen code
    > or, wait...ahh its probably due to 16bit IFS interface?


  10. Re: DosAllocMem and OBJ_ANY

    Thanks for the hints!

    > 1.) The OS/2 Programming Guide and Reference Addendum lists APIs
    > supporting high memory. I don't know if the list is complete.


    Yes, I know this list. But all on it refers to Dos, nothing mentioned
    about graphics or multimedia APIs which are the parts that would especially
    require this support. To my experience IBM very often thought deeply
    about application coverage of their products. So I have difficulties
    to accept that they would have overseen high memory support coverage
    for such kind of memory hungry applications on a server OS.

    A Vio buffer of some KB or a file buffer of some hundred KB, I think
    we can skip this as reason for introducing HMS support.

    I see so far the following critical areas:
    - network performance for many clients (many, many buffers for caching)
    - large office applications
    - high resolution graphics
    - high resolution video

    Lets have a look at graphics. Try working with a 5000x4000x32bpp bitmap.
    Just decoding it, for instance from JPEG, will result in a final size of
    about 80MB (add more memory for the decoding). Then it is just loaded
    but you haven't done anything with it.
    Digital SLRs will soon reach this image size and I suspect they will go up
    to 48bpp in not to far future because of HDR (High Dynamic Range) support.
    Many image manipulation algorithms require a lot of additional memory.
    Trying to scale the above image by factor 2 with a resampling scaler will
    fail because not enough memory can be allocated from the low pool.
    If algorithms should scale well on SMP, even more memory would be required
    to reduce synchronization locks between multiple worker threads trying to
    access shared buffers.
    If GPI would not support drawing an area of the bitmap from a large high
    memory buffer, an additional drawing buffer in low memory (+processing power
    for extracting the relevant rectangle) or an alias would be required.
    This would make it complicated and somehow contradicts the initial goal.

    Graphics is one area but video is critical as well. We will soon have
    high resolution TV around the world and it's really memory intensive
    to work with such recorded videos!

    If the core OS APIs cannot provide support for such memory intensive
    applications, the OS would die quite soon. Having a system with 3-4GB
    of RAM would not help if it cannot be efficiently used. I think this
    was the reason for IBM to introduce HMS support. If they really forgot
    the graphics engine (I still doubt), HMS would have a very limited use.

    As Peter told us, GpiDrawBits seems to work fine. I also have not
    discovered any issues blitting areas from huge buffers (600MB). Also
    GpiCreateBitmap showed no obvious problems so far. Well you cannot
    create such a huge GPI bitmap (probably GPI allocates memory from
    low memory for internal purposes or the graphics driver does not
    support such huge bitmaps), but this is OK for most cases.

    I think most of the graphics APIs are fully 32bit (for performance
    reasons) and thus there is some hope

    > 2.) Who would care about DosOpenL ? You don't pass large buffers to that
    > API so what's the point ?


    As long as DosWrite and DosRead support large buffers, there is no problem.
    It just makes it error prone as you have to think twice before using every
    single API function. As DosOpenL is new and was introduced in the same time
    frame as HMS, it would have been wise to write it compatible to HMS.

    > 3.) There is a way out of the misery: You can use DosAliasMem to get an
    > alias in low memory for a high memory object. Then your memory is
    > accessible via LDT selectors (and the compatibility region thunking
    > algorithm applies).
    > You can then safely pass that alias to the API function not supporting
    > high memory. Just make sure you DO NOT use SEL_USE32 with DosAliasMem
    > (because that would create a 32-bit LDT selector instead of 16-bit LDT
    > selectors where 32-bit LDT selectors do not work with 16-bit code). Also
    > be aware that LDT selectors are a scarce resource (because all of them
    > are normally used to map low memory to 16-bit far pointers in a way that
    > the compatibility region thunking algorithm applies). You should free
    > the alias as soon as possible.


    Interesting, I haven't heared about it so far.
    But trying to reduce use of one scarce resource by using another scarce
    resources is probably not the way to go for many applications. But it would
    definitely help for my scenario (temporary use during drawing/creating a bitmap).
    So far I see no issue with direct HMS usage, but it's good to know a fallback

    Thanks,
    Heiko

  11. Re: DosAllocMem and OBJ_ANY

    [A complimentary Cc of this posting was sent to
    Heiko Nitzsche
    ], who wrote in article <47b37e36$0$23016$9b4e6d93@newsspool1.arcor-online.net>:
    > Lets have a look at graphics. Try working with a 5000x4000x32bpp bitmap.
    > Just decoding it, for instance from JPEG, will result in a final size of
    > about 80MB (add more memory for the decoding). Then it is just loaded
    > but you haven't done anything with it.
    > Digital SLRs will soon reach this image size


    Already there.

    > and I suspect they will go up
    > to 48bpp in not to far future because of HDR (High Dynamic Range) support.


    The current State-of-Art is 42bpp (after demosaicing). Of course, to
    manipulate this data (after gamma conversion), you need at least 48bpp.

    > Graphics is one area but video is critical as well. We will soon have
    > high resolution TV around the world and it's really memory intensive
    > to work with such recorded videos!


    Currently, without time-lapsing, it is hard to get above about 3K x 2K
    video (you can rent 10MP video camera, but I think there is only a
    handful in the pool).

    > If the core OS APIs cannot provide support for such memory intensive
    > applications, the OS would die quite soon.


    LOL! OS/2 is dead for almost a decade already. If you need anything
    up-to-date, you switch off it.

    > > 2.) Who would care about DosOpenL ? You don't pass large buffers to that
    > > API so what's the point ?


    One still passes a buffer (a file name). IIRC, it is not highmem-supported.

    Hope this helps,
    Ilya

  12. Re: DosAllocMem and OBJ_ANY

    >>> 2.) Who would care about DosOpenL ? You don't pass large buffers to that
    >>> API so what's the point ?

    >
    > One still passes a buffer (a file name). IIRC, it is not highmem-supported.


    Come on. Copying a filename (< 260 bytes) into a buffer in low memory or
    on the stack is not a problem. If yes, use DosAliasMem.

    Lars

  13. Re: DosAllocMem and OBJ_ANY

    Heiko Nitzsche writes:

    > Yes, I know this list. But all on it refers to Dos, nothing mentioned
    > about graphics or multimedia APIs which are the parts that would especially
    > require this support.


    I have enabled high memory support in my audio editing software. I have
    succeeded in loading a .WAV file over 800 MB in size. Use of multimedia
    APIs is limited to just what I need to play an audio clip, but it has no
    trouble playing the 800 MB clip.


  14. Re: DosAllocMem and OBJ_ANY

    On a pleasant day while strolling in
    comp.os.os2.programmer.misc, a person by the name of
    Heiko Nitzsche exclaimed:
    > require this support. To my experience IBM very often thought deeply
    > about application coverage of their products. So I have difficulties
    > to accept that they would have overseen high memory support coverage
    > for such kind of memory hungry applications on a server OS.


    The only kind they probably cared about would be
    database, Java and other server applications, where
    those pesky big customers keep using it and want to use
    modern hardware.

    OS/2 GUI? That's Java right?

    --
    aaronl at consultant dot com
    For every expert, there is an equal and
    opposite expert. - Arthur C. Clarke

  15. Re: DosAllocMem and OBJ_ANY

    [A complimentary Cc of this posting was NOT [per weedlist] sent to
    Lars Erdmann
    ], who wrote in article <47b3e31a$0$378$9b4e6d93@newsspool2.arcor-online.net>:
    > >>> 2.) Who would care about DosOpenL ? You don't pass large buffers to that
    > >>> API so what's the point ?

    > >
    > > One still passes a buffer (a file name). IIRC, it is not highmem-supported.


    > Come on. Copying a filename (< 260 bytes) into a buffer in low memory or
    > on the stack is not a problem.


    Of course it is not a problem, IF one REMEMBERS that there is a
    problem when one does not do it. ;-)

    Yours,
    Ilya



  16. Re: DosAllocMem and OBJ_ANY

    Thanks for your help and hints!

    I finally settled with the following implementation for
    allocations larger than 4KB. I hope ERROR_INVALID_PARAMETER
    will catch the missing highmem support on old systems.
    Unfortunately I cannot test it on such a system.

    void * gbmmem_malloc(size_t size)
    {
    void * pBuffer = NULL;

    /* initially try to allocate from High Memory Pool */
    static ULONG highmemAlloc = OBJ_ANY;

    /* allocate from high memory first */
    APIRET apiRet = DosAllocMem(&pBuffer, (ULONG)size,
    PAG_READ | PAG_WRITE | PAG_COMMIT | highmemAlloc);
    if (apiRet == ERROR_INVALID_PARAMETER)
    {
    /* try again without highmem request as fallback for older systems */
    apiRet = DosAllocMem(&pBuffer, (ULONG)size,
    PAG_READ | PAG_WRITE | PAG_COMMIT);
    if (apiRet != NO_ERROR)
    {
    return NULL;
    }
    /* Allocation from high memory pool failed but low memory was OK.
    * Switching off highmem allocation retries for further requests.
    */
    highmemAlloc = 0;
    }
    if (apiRet != NO_ERROR)
    {
    return NULL;
    }
    return pBuffer;
    }

  17. Re: DosAllocMem and OBJ_ANY

    Heiko Nitzsche schrieb:
    > Thanks for your help and hints!
    >
    > I finally settled with the following implementation for
    > allocations larger than 4KB. I hope ERROR_INVALID_PARAMETER
    > will catch the missing highmem support on old systems.
    > Unfortunately I cannot test it on such a system.


    Yes you can. On your system, set VIRTUALADDRESSLIMIT=512 (in
    config.sys), then you have the old behaviour (without high memory).

    Lars

  18. Re: DosAllocMem and OBJ_ANY

    On Thu, 14 Feb 2008 23:40:12 +0100, Heiko Nitzsche
    wrote:

    > if (apiRet == ERROR_INVALID_PARAMETER)
    > {
    > /* try again without highmem request as fallback for older systems */
    > apiRet = DosAllocMem(&pBuffer, (ULONG)size,
    > PAG_READ | PAG_WRITE | PAG_COMMIT);
    > if (apiRet != NO_ERROR)
    > {
    > return NULL;
    > }
    > /* Allocation from high memory pool failed but low memory was OK.
    > * Switching off highmem allocation retries for further requests.
    > */
    > highmemAlloc = 0;
    > }
    > if (apiRet != NO_ERROR)
    > {
    > return NULL;
    > }
    > return pBuffer;


    Optimizing the above bit to:

    if (apiRet == ERROR_INVALID_PARAMETER)
    {
    /* Allocation from high memory pool failed but low memory was OK.
    * Switching off highmem allocation retries for further requests.
    */
    highmemAlloc = 0;
    /* try again without highmem request as fallback for older systems */
    apiRet = DosAllocMem(&pBuffer, (ULONG)size,
    PAG_READ | PAG_WRITE | PAG_COMMIT);
    }

    return (apiRet == NO_ERROR ? pBuffer : NULL);

  19. Re: DosAllocMem and OBJ_ANY

    Heiko Nitzsche wrote:
    > Lets have a look at graphics. Try working with a 5000x4000x32bpp bitmap.
    > ...


    [snipped for brevity, but read in full]

    > As Peter told us, GpiDrawBits seems to work fine. I also have not
    > discovered any issues blitting areas from huge buffers (600MB). Also
    > GpiCreateBitmap showed no obvious problems so far. Well you cannot
    > create such a huge GPI bitmap (probably GPI allocates memory from
    > low memory for internal purposes or the graphics driver does not
    > support such huge bitmaps), but this is OK for most cases.
    >
    > I think most of the graphics APIs are fully 32bit (for performance
    > reasons) and thus there is some hope


    To my recollection, there is a portion of Gpi's implementation that can
    occur in the video device driver itself (as well as some portion of
    DIVE). In this case, it is possible to wind up in 16 bit code. We're
    lucky that GRADD drivers like SNAP are so pervasive (and actually
    necessary at this stage) that this has become less of a concern, but it
    still could cause some issues with legacy drivers.

    --
    [Reverse the parts of the e-mail address to reply.]

  20. Re: DosAllocMem and OBJ_ANY

    tholen@antispam.ham wrote:
    > Heiko Nitzsche writes:
    >
    >>Yes, I know this list. But all on it refers to Dos, nothing mentioned
    >>about graphics or multimedia APIs which are the parts that would especially
    >>require this support.

    >
    > I have enabled high memory support in my audio editing software. I have
    > succeeded in loading a .WAV file over 800 MB in size. Use of multimedia
    > APIs is limited to just what I need to play an audio clip, but it has no
    > trouble playing the 800 MB clip.


    Are you using UniAud or another 32-bit audio driver? Your results are
    positive, but may be unintentionally misleading in this case. I use
    Sander's 32-bit SBLive driver, so I can't augment your findings
    unfortunately.

    --
    [Reverse the parts of the e-mail address to reply.]

+ Reply to Thread
Page 1 of 3 1 2 3 LastLast