On Sun, 10 Feb 2008 19:36:18 UTC, Heiko Nitzsche
wrote:

> Will DosAllocMem on older OS ignore OBJ_ANY and allocate from low
> memory pool as fallback instead (pre Warp 4.5x and Warp 3)?

No.


> If it is not backward compatible, what is the recommended way
> to detect the support?

Retry the allocation without OBJ_ANY.



--
Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
http://www.s-t.de
Please remove all characters left of the "R" in my email address

>> If it is not backward compatible, what is the recommended way
>> to detect the support?
>
> Retry the allocation without OBJ_ANY.

Yeah, trial and error is what I expected .

Do you know if there are restrictions for using HMS memory objects in
PM or GPI? I especially need this for bitmap creation and GpiDrawBits.

So far it seems to work, but something more definitive would be highly
appreciated.

Heiko Nitzsche ha scritto:

> I'm currently changing most of my programs to use High Memory by
> using OBJ_ANY for DosAllocMem. So far on eCS this works fine and
> I can allocate more than 512MB just fine.
>
> I could not find any hint in OS2TK about backward compatibility.
> Will DosAllocMem on older OS ignore OBJ_ANY and allocate from low
> memory pool as fallback instead (pre Warp 4.5x and Warp 3)?
>
> If it is not backward compatible, what is the recommended way
> to detect the support?

I used the following code on program initialization to check the availability
of high memory:

// check if high memory objects are supported and maximum available memory
g.rc = DosQuerySysInfo(QSV_VIRTUALADDRESSLIMIT, QSV_VIRTUALADDRESSLIMIT,
&cbAvailMem, sizeof(ULONG));
if (g.rc == NO_ERROR)
{
g.mode |= FSJ_MEMHIGHOBJ;
g.rc = DosQuerySysInfo(QSV_MAXHPRMEM, QSV_MAXHPRMEM,
&cbAvailMem, sizeof(ULONG));
}
else
{
g.rc = DosQuerySysInfo(QSV_MAXPRMEM, QSV_MAXPRMEM,
&cbAvailMem, sizeof(ULONG));
}
// if cannot get the size of available memory show an error message
if (g.rc) return handleError(FALSE, FALSE, SZERR_MAXPRMEM);


--
bye
Alessandro

Hi,

Heiko Nitzsche schrieb:
> If it is not backward compatible, what is the recommended way
> to detect the support?

allocate a small memory object in high memory at the application startup
and set a global static field depending on the success to OBJ_ANY or
zero. You can or this field to the flags of DosAllocMem whereever you like.


Marcel

On Sun, 10 Feb 2008 19:56:47 UTC, Heiko Nitzsche wrote:

> Do you know if there are restrictions for using HMS memory objects in
> PM or GPI? I especially need this for bitmap creation and GpiDrawBits.

GpiDrawBits works when passed high memory stuff, we use that in Mozilla,
too. If you want to know which functions do not work, take a look at the
wrapper header in kLibc. (Hmm, just wanted to point you to the online
repository, but that site doesn't answer at the moment. Tt's shipped as
os2safe.h with GCC 3.3.5 CSD3.)
--
Greetings, | My From: address is valid as is the version without "spam"
Peter. | I try to find real messages among the spam once a week

>> Do you know if there are restrictions for using HMS memory objects in
>> PM or GPI? I especially need this for bitmap creation and GpiDrawBits.
>
> GpiDrawBits works when passed high memory stuff, we use that in Mozilla,

Excellent. I assume PM and GPI are pure 32bit and thus should have
no problem dealing with high memory objects. Dos calls seem to be
more critical (e.g. I think Vio stuff is still mostly 16bit).

> too. If you want to know which functions do not work, take a look at the
> wrapper header in kLibc. (Hmm, just wanted to point you to the online
> repository, but that site doesn't answer at the moment. Tt's shipped as
> os2safe.h with GCC 3.3.5 CSD3.)

I found os2safe.h (have GCC installed) but how is it to be interpreted?

Are the Safe* functions wrappers for non-safe API functions or do they
declare them as safe?

#define DosForceDelete SafeDosForceDelete
#define DosQueryHType SafeDosQueryHType
#define DosCreatePipe SafeDosCreatePipe
#define DosQueryNPHState SafeDosQueryNPHState
#define DosWaitNPipe SafeDosWaitNPipe
#define DosSetFilePtr SafeDosSetFilePtr
#define DosSetFilePtrL SafeDosSetFilePtrL
#define DosDupHandle SafeDosDupHandle
#define DosOpenL SafeDosOpenL
#define DosQueryFHState SafeDosQueryFHState
#define DosSetDateTime SafeDosSetDateTime
#define DosStartSession SafeDosStartSession
#define DosQueryAppType SafeDosQueryAppType
#define DosWinUpper SafeDosWinUpper

Assuming the Safe* functions are wrappers to make these API functions
safe, can I assume that everything else is safe for high memory objects?
It looks strange that especially DosOpenL is not high mem safe as
it was introduced quite late. Maybe it is reusing old DosOpen code
or, wait...ahh its probably due to 16bit IFS interface?

[A complimentary Cc of this posting was NOT [per weedlist] sent to
Peter Weilbacher
], who wrote in article :
> GpiDrawBits works when passed high memory stuff, we use that in Mozilla,
> too.

IIRC what the developer of PMView reported, some Gpi APIs (BitBlit?)
do not work with high memory. (They work with SOME video drivers,
though...)

> If you want to know which functions do not work, take a look at the
> wrapper header in kLibc.

There are a small handful of functions which work. All the rest do not.

AFAIU, the "wrapped" stuff is for APIs people managed to trace
problems to. The using the unwrapped unsupported stuff is just
playing russian rulette - OK for some applications, not OK for some others.

Hoep this helps,
Ilya

1.) The OS/2 Programming Guide and Reference Addendum lists APIs
supporting high memory. I don't know if the list is complete.
2.) Who would care about DosOpenL ? You don't pass large buffers to that
API so what's the point ?
3.) There is a way out of the misery: You can use DosAliasMem to get an
alias in low memory for a high memory object. Then your memory is
accessible via LDT selectors (and the compatibility region thunking
algorithm applies).
You can then safely pass that alias to the API function not supporting
high memory. Just make sure you DO NOT use SEL_USE32 with DosAliasMem
(because that would create a 32-bit LDT selector instead of 16-bit LDT
selectors where 32-bit LDT selectors do not work with 16-bit code). Also
be aware that LDT selectors are a scarce resource (because all of them
are normally used to map low memory to 16-bit far pointers in a way that
the compatibility region thunking algorithm applies). You should free
the alias as soon as possible.
If you no longer need the alias address, you use DosFreeMem on the alias
to free the alias (that WILL NOT free the memory, only the alias).

Lars

Heiko Nitzsche schrieb:
>>> Do you know if there are restrictions for using HMS memory objects in
>>> PM or GPI? I especially need this for bitmap creation and GpiDrawBits.
>>
>> GpiDrawBits works when passed high memory stuff, we use that in Mozilla,
>
> Excellent. I assume PM and GPI are pure 32bit and thus should have
> no problem dealing with high memory objects. Dos calls seem to be
> more critical (e.g. I think Vio stuff is still mostly 16bit).
>
>> too. If you want to know which functions do not work, take a look at the
>> wrapper header in kLibc. (Hmm, just wanted to point you to the online
>> repository, but that site doesn't answer at the moment. Tt's shipped
>> as os2safe.h with GCC 3.3.5 CSD3.)
>
> I found os2safe.h (have GCC installed) but how is it to be interpreted?
>
> Are the Safe* functions wrappers for non-safe API functions or do they
> declare them as safe?
>
> #define DosForceDelete SafeDosForceDelete
> #define DosQueryHType SafeDosQueryHType
> #define DosCreatePipe SafeDosCreatePipe
> #define DosQueryNPHState SafeDosQueryNPHState
> #define DosWaitNPipe SafeDosWaitNPipe
> #define DosSetFilePtr SafeDosSetFilePtr
> #define DosSetFilePtrL SafeDosSetFilePtrL
> #define DosDupHandle SafeDosDupHandle
> #define DosOpenL SafeDosOpenL
> #define DosQueryFHState SafeDosQueryFHState
> #define DosSetDateTime SafeDosSetDateTime
> #define DosStartSession SafeDosStartSession
> #define DosQueryAppType SafeDosQueryAppType
> #define DosWinUpper SafeDosWinUpper
>
> Assuming the Safe* functions are wrappers to make these API functions
> safe, can I assume that everything else is safe for high memory objects?
> It looks strange that especially DosOpenL is not high mem safe as
> it was introduced quite late. Maybe it is reusing old DosOpen code
> or, wait...ahh its probably due to 16bit IFS interface?

Thanks for the hints!

> 1.) The OS/2 Programming Guide and Reference Addendum lists APIs
> supporting high memory. I don't know if the list is complete.

Yes, I know this list. But all on it refers to Dos, nothing mentioned
about graphics or multimedia APIs which are the parts that would especially
require this support. To my experience IBM very often thought deeply
about application coverage of their products. So I have difficulties
to accept that they would have overseen high memory support coverage
for such kind of memory hungry applications on a server OS.

A Vio buffer of some KB or a file buffer of some hundred KB, I think
we can skip this as reason for introducing HMS support.

I see so far the following critical areas:
- network performance for many clients (many, many buffers for caching)
- large office applications
- high resolution graphics
- high resolution video

Lets have a look at graphics. Try working with a 5000x4000x32bpp bitmap.
Just decoding it, for instance from JPEG, will result in a final size of
about 80MB (add more memory for the decoding). Then it is just loaded
but you haven't done anything with it.
Digital SLRs will soon reach this image size and I suspect they will go up
to 48bpp in not to far future because of HDR (High Dynamic Range) support.
Many image manipulation algorithms require a lot of additional memory.
Trying to scale the above image by factor 2 with a resampling scaler will
fail because not enough memory can be allocated from the low pool.
If algorithms should scale well on SMP, even more memory would be required
to reduce synchronization locks between multiple worker threads trying to
access shared buffers.
If GPI would not support drawing an area of the bitmap from a large high
memory buffer, an additional drawing buffer in low memory (+processing power
for extracting the relevant rectangle) or an alias would be required.
This would make it complicated and somehow contradicts the initial goal.

Graphics is one area but video is critical as well. We will soon have
high resolution TV around the world and it's really memory intensive
to work with such recorded videos!

If the core OS APIs cannot provide support for such memory intensive
applications, the OS would die quite soon. Having a system with 3-4GB
of RAM would not help if it cannot be efficiently used. I think this
was the reason for IBM to introduce HMS support. If they really forgot
the graphics engine (I still doubt), HMS would have a very limited use.

As Peter told us, GpiDrawBits seems to work fine. I also have not
discovered any issues blitting areas from huge buffers (600MB). Also
GpiCreateBitmap showed no obvious problems so far. Well you cannot
create such a huge GPI bitmap (probably GPI allocates memory from
low memory for internal purposes or the graphics driver does not
support such huge bitmaps), but this is OK for most cases.

I think most of the graphics APIs are fully 32bit (for performance
reasons) and thus there is some hope

> 2.) Who would care about DosOpenL ? You don't pass large buffers to that
> API so what's the point ?

As long as DosWrite and DosRead support large buffers, there is no problem.
It just makes it error prone as you have to think twice before using every
single API function. As DosOpenL is new and was introduced in the same time
frame as HMS, it would have been wise to write it compatible to HMS.

> 3.) There is a way out of the misery: You can use DosAliasMem to get an
> alias in low memory for a high memory object. Then your memory is
> accessible via LDT selectors (and the compatibility region thunking
> algorithm applies).
> You can then safely pass that alias to the API function not supporting
> high memory. Just make sure you DO NOT use SEL_USE32 with DosAliasMem
> (because that would create a 32-bit LDT selector instead of 16-bit LDT
> selectors where 32-bit LDT selectors do not work with 16-bit code). Also
> be aware that LDT selectors are a scarce resource (because all of them
> are normally used to map low memory to 16-bit far pointers in a way that
> the compatibility region thunking algorithm applies). You should free
> the alias as soon as possible.

Interesting, I haven't heared about it so far.
But trying to reduce use of one scarce resource by using another scarce
resources is probably not the way to go for many applications. But it would
definitely help for my scenario (temporary use during drawing/creating a bitmap).
So far I see no issue with direct HMS usage, but it's good to know a fallback

Thanks,
Heiko

[A complimentary Cc of this posting was sent to
Heiko Nitzsche
], who wrote in article <47b37e36$0$23016$9b4e6d93@newsspool1.arcor-online.net>:
> Lets have a look at graphics. Try working with a 5000x4000x32bpp bitmap.
> Just decoding it, for instance from JPEG, will result in a final size of
> about 80MB (add more memory for the decoding). Then it is just loaded
> but you haven't done anything with it.
> Digital SLRs will soon reach this image size

Already there.

> and I suspect they will go up
> to 48bpp in not to far future because of HDR (High Dynamic Range) support.

The current State-of-Art is 42bpp (after demosaicing). Of course, to
manipulate this data (after gamma conversion), you need at least 48bpp.

> Graphics is one area but video is critical as well. We will soon have
> high resolution TV around the world and it's really memory intensive
> to work with such recorded videos!

Currently, without time-lapsing, it is hard to get above about 3K x 2K
video (you can rent 10MP video camera, but I think there is only a
handful in the pool).

> If the core OS APIs cannot provide support for such memory intensive
> applications, the OS would die quite soon.

LOL! OS/2 is dead for almost a decade already. If you need anything
up-to-date, you switch off it.

> > 2.) Who would care about DosOpenL ? You don't pass large buffers to that
> > API so what's the point ?

One still passes a buffer (a file name). IIRC, it is not highmem-supported.

Hope this helps,
Ilya

>>> 2.) Who would care about DosOpenL ? You don't pass large buffers to that
>>> API so what's the point ?
>
> One still passes a buffer (a file name). IIRC, it is not highmem-supported.

Come on. Copying a filename (< 260 bytes) into a buffer in low memory or
on the stack is not a problem. If yes, use DosAliasMem.

Lars

Heiko Nitzsche writes:

> Yes, I know this list. But all on it refers to Dos, nothing mentioned
> about graphics or multimedia APIs which are the parts that would especially
> require this support.

I have enabled high memory support in my audio editing software. I have
succeeded in loading a .WAV file over 800 MB in size. Use of multimedia
APIs is limited to just what I need to play an audio clip, but it has no
trouble playing the 800 MB clip.

On a pleasant day while strolling in
comp.os.os2.programmer.misc, a person by the name of
Heiko Nitzsche exclaimed:
> require this support. To my experience IBM very often thought deeply
> about application coverage of their products. So I have difficulties
> to accept that they would have overseen high memory support coverage
> for such kind of memory hungry applications on a server OS.

The only kind they probably cared about would be
database, Java and other server applications, where
those pesky big customers keep using it and want to use
modern hardware.

OS/2 GUI? That's Java right?

--
aaronl at consultant dot com
For every expert, there is an equal and
opposite expert. - Arthur C. Clarke

[A complimentary Cc of this posting was NOT [per weedlist] sent to
Lars Erdmann
], who wrote in article <47b3e31a$0$378$9b4e6d93@newsspool2.arcor-online.net>:
> >>> 2.) Who would care about DosOpenL ? You don't pass large buffers to that
> >>> API so what's the point ?
> >
> > One still passes a buffer (a file name). IIRC, it is not highmem-supported.

> Come on. Copying a filename (< 260 bytes) into a buffer in low memory or
> on the stack is not a problem.

Of course it is not a problem, IF one REMEMBERS that there is a
problem when one does not do it. ;-)

Yours,
Ilya

Thanks for your help and hints!

I finally settled with the following implementation for
allocations larger than 4KB. I hope ERROR_INVALID_PARAMETER
will catch the missing highmem support on old systems.
Unfortunately I cannot test it on such a system.

void * gbmmem_malloc(size_t size)
{
void * pBuffer = NULL;

/* initially try to allocate from High Memory Pool */
static ULONG highmemAlloc = OBJ_ANY;

/* allocate from high memory first */
APIRET apiRet = DosAllocMem(&pBuffer, (ULONG)size,
PAG_READ | PAG_WRITE | PAG_COMMIT | highmemAlloc);
if (apiRet == ERROR_INVALID_PARAMETER)
{
/* try again without highmem request as fallback for older systems */
apiRet = DosAllocMem(&pBuffer, (ULONG)size,
PAG_READ | PAG_WRITE | PAG_COMMIT);
if (apiRet != NO_ERROR)
{
return NULL;
}
/* Allocation from high memory pool failed but low memory was OK.
* Switching off highmem allocation retries for further requests.
*/
highmemAlloc = 0;
}
if (apiRet != NO_ERROR)
{
return NULL;
}
return pBuffer;
}

Heiko Nitzsche schrieb:
> Thanks for your help and hints!
>
> I finally settled with the following implementation for
> allocations larger than 4KB. I hope ERROR_INVALID_PARAMETER
> will catch the missing highmem support on old systems.
> Unfortunately I cannot test it on such a system.

Yes you can. On your system, set VIRTUALADDRESSLIMIT=512 (in
config.sys), then you have the old behaviour (without high memory).

Lars

On Thu, 14 Feb 2008 23:40:12 +0100, Heiko Nitzsche
wrote:

> if (apiRet == ERROR_INVALID_PARAMETER)
> {
> /* try again without highmem request as fallback for older systems */
> apiRet = DosAllocMem(&pBuffer, (ULONG)size,
> PAG_READ | PAG_WRITE | PAG_COMMIT);
> if (apiRet != NO_ERROR)
> {
> return NULL;
> }
> /* Allocation from high memory pool failed but low memory was OK.
> * Switching off highmem allocation retries for further requests.
> */
> highmemAlloc = 0;
> }
> if (apiRet != NO_ERROR)
> {
> return NULL;
> }
> return pBuffer;

Optimizing the above bit to:

if (apiRet == ERROR_INVALID_PARAMETER)
{
/* Allocation from high memory pool failed but low memory was OK.
* Switching off highmem allocation retries for further requests.
*/
highmemAlloc = 0;
/* try again without highmem request as fallback for older systems */
apiRet = DosAllocMem(&pBuffer, (ULONG)size,
PAG_READ | PAG_WRITE | PAG_COMMIT);
}

return (apiRet == NO_ERROR ? pBuffer : NULL);

Heiko Nitzsche wrote:
> Lets have a look at graphics. Try working with a 5000x4000x32bpp bitmap.
> ...

[snipped for brevity, but read in full]

> As Peter told us, GpiDrawBits seems to work fine. I also have not
> discovered any issues blitting areas from huge buffers (600MB). Also
> GpiCreateBitmap showed no obvious problems so far. Well you cannot
> create such a huge GPI bitmap (probably GPI allocates memory from
> low memory for internal purposes or the graphics driver does not
> support such huge bitmaps), but this is OK for most cases.
>
> I think most of the graphics APIs are fully 32bit (for performance
> reasons) and thus there is some hope

To my recollection, there is a portion of Gpi's implementation that can
occur in the video device driver itself (as well as some portion of
DIVE). In this case, it is possible to wind up in 16 bit code. We're
lucky that GRADD drivers like SNAP are so pervasive (and actually
necessary at this stage) that this has become less of a concern, but it
still could cause some issues with legacy drivers.

--
[Reverse the parts of the e-mail address to reply.]

tholen@antispam.ham wrote:
> Heiko Nitzsche writes:
>
>>Yes, I know this list. But all on it refers to Dos, nothing mentioned
>>about graphics or multimedia APIs which are the parts that would especially
>>require this support.
>
> I have enabled high memory support in my audio editing software. I have
> succeeded in loading a .WAV file over 800 MB in size. Use of multimedia
> APIs is limited to just what I need to play an audio clip, but it has no
> trouble playing the 800 MB clip.

Are you using UniAud or another 32-bit audio driver? Your results are
positive, but may be unintentionally misleading in this case. I use
Sander's 32-bit SBLive driver, so I can't augment your findings
unfortunately.

--
[Reverse the parts of the e-mail address to reply.]