>> Or are you discussing memory-pre-digestion by DD to translate memory
>> addresses as-usable-by-ring3 into something usable-by-ring-0?
>
> No. DDs can always address memory that is addressable from ring3. No
> additional steps are necessary.

Oops, that is not correct. A DD needs to use VMProcessToGlobal to map
process memory to global address space.
Note: Global address space is everything beyond what is specified with
VIRTUALADDRESSLIMIT config.sys setting, that is, everything above
per-process and shared high memory address space.

Lars

> low memory if the video BIOS code is pure 16-bit. If the user buffer is in
> high memory, the caller will have to use DosAliasMem to get a memory alias
> in low memory.

Yes, but what makes you think the caller has to be the application?

The whole functionality and the interface looks more like something IBM needed
mainly for internal purposes. I could imagine that the critical areas within
the API (e.g. GPI) call DosAliasMem by themselves if necessary. Maybe they forgot
to adapt some APIs (the ones that show problems). This would make much more sense
than moving the task to the application level. I'd be astonished if IBM would
break the rules for their well designed API so badly.

The technical background discussed by you together with the DosAliasMem description
requires for my point of view a very deep knowledge about the system architecture.
I don't think that this is appropriate for a higher level API which usually hides
these details. An average developer should not have to deal with these details and
usually cannot deal with them because of either lack of knowledge or just because
he/she cannot know how the API is internally implemented.
Look at the parameters of DosAliasMem, based on what should an application developer
decide which parameters to chose? It is not possible without knowing the API
implementation and their requirements.
For driver level development this might be a bit different as here the knowledge is
probably required anyway but when drivers call APIs they also cannot look into a
crystal ball to know what the API internally requires.

Summarizing the above I think it makes the most sense that IBM targeted DosAliasMem
for drivers (or direct driver callers) to allow them to map high memory for their
purpose because they know when and how this would be required while the normal
application cannot know this.

Hallo,

> The whole functionality and the interface looks more like something IBM
> needed
> mainly for internal purposes. I could imagine that the critical areas
> within
> the API (e.g. GPI) call DosAliasMem by themselves if necessary. Maybe they
> forgot
> to adapt some APIs (the ones that show problems). This would make much
> more sense
> than moving the task to the application level. I'd be astonished if IBM
> would
> break the rules for their well designed API so badly.

Don't forget that the GPI is much older than high memory.High memory is not
really well designed into the OS at all. That's what makes it so awkward,
sad as it is ...

> Look at the parameters of DosAliasMem, based on what should an application
> developer
> decide which parameters to chose? It is not possible without knowing the
> API
> implementation and their requirements.

My questions were just for technical interest. You don't have to know in
detail how it works. You just have to know that DosAliasMem will give you an
alias in low memory region for memory allocated in high memory region.

> For driver level development this might be a bit different as here the
> knowledge is
> probably required anyway but when drivers call APIs they also cannot look
> into a
> crystal ball to know what the API internally requires.
>
> Summarizing the above I think it makes the most sense that IBM targeted
> DosAliasMem
> for drivers (or direct driver callers) to allow them to map high memory
> for their
> purpose because they know when and how this would be required while the
> normal
> application cannot know this.

DosAliasMem can only be called by Ring-3 code, that is, application level
code. Not by device drivers.

Lars

> My questions were just for technical interest. You don't have to know in
> detail how it works. You just have to know that DosAliasMem will give you an
> alias in low memory region for memory allocated in high memory region.

But one has to know which parameters have to be chosen. And this is not
really obvious based on the documentation. It is not even documented for
what the function is needed, no link to DosAllocMem or other hints.

If this is really thought to be used by applications the way we think,
most implementations would end up as follows (I mean almost always!):

Every call to DosAllocMem with OBJ_ANY will be followed by a DosAliasMem
if the buffer is passed to an API call. In fact they could have built the
mechanism then directly into DosAllocMem. In a simple application it might
be possible to follow up where the allocated buffer finally is used but in
a more complex application this is almost impossible to predict, especially
if C++ or something similar is used to hide implementation details.
For the cases you specifically know by design that the buffer will never be
passed to a system API, a suppression flag for DosAllocMem would have been
much more usable. The default would be to always create the alias which would
automatically put the application on the safe side.

Example:
Assume there is an object that loads a bitmap file and provides it later
on for instance to algorithms. Well, here the alias would not be required.
BUT, the bitmap data could also be used for drawing. In this case the alias
would be required to be on the safe side. Because the bitmap file handler
cannot know for what the buffer is finally used, it would always need to
alias the buffer.
The other way round would be thinkable as well, I mean that the user of the
buffer (the algorithms or the drawing code) creates the alias. There are 2
problems. 1st there seems to be no unalias functionality, the alias is only
freed if the buffer is freed. So the buffer would be at least aliased once
(assuming multiple DosAliasMem reuses the selector for multiple calls).
2nd is that the buffer address must fulfill certain alignment requirements.
But the buffer user has no control of this. It would need to imply that
DosAllocMem has been used and not malloc from the compiler runtime library.
You don't want this!

Ahhhhhhhha..., and maybe scenario 2 describes the reason why the alias cannot
be created within the system API when needed. The API would have the same problem.
The buffer passed to the API does not always have to be aligned as required by
DosAliasMem.

Anyway, in both scenarios you end up having an alias, so why is it not implied
in DosAllocMem when OBJ_ANY is used? Is the waste of resources really so big
to rectify the significantly increased complexity?

In <47bcc73f$0$23013$9b4e6d93@newsspool1.arcor-online.net>, on 02/21/2008
at 01:34 AM, Heiko Nitzsche said:

Hi,

>Anyway, in both scenarios you end up having an alias, so why is it not
>implied in DosAllocMem when OBJ_ANY is used? Is the waste of resources
>really so big to rectify the significantly increased complexity?

All of what you say is true, but you have to remember the probable context
in which IBM created the API. They had a large base of installed code
that they wanted to avoid breaking at almost any cost. At the same time
they had a need for the functionality DosAliasMem provided. The result is
an API that has limitiations but that implements what they needed. I
really don't know when the API was implemeneted, but TTBOMK it was private
until the Addendum was published.

I tend to read between the lines. Since the docs show an example of using
DosAliasMem to implement buffers that are read-only to the givee and
read-write to the owner, this is the reason the API exists. The API was
implemented in a way that affected a minimal amount of existing kernel
code. The fact that the alias needed to be in low memory and use an LDT
descriptor was an unavoidable side effect to the implementation.

Steven

--
--------------------------------------------------------------------------------------------
Steven Levine MR2/ICE 3.00 beta 11pre #10183
eCS/Warp/DIY/14.103a_W4 www.scoug.com irc.ca.webbnet.info #scoug (Wed 7pm PST)
--------------------------------------------------------------------------------------------

[A complimentary Cc of this posting was sent to
Heiko Nitzsche
], who wrote in article <47bca203$0$377$9b4e6d93@newsspool2.arcor-online.net>:

> The whole functionality and the interface looks more like something
> IBM needed mainly for internal purposes. I could imagine that the
> critical areas within the API (e.g. GPI) call DosAliasMem by
> themselves if necessary. Maybe they forgot to adapt some APIs (the
> ones that show problems). This would make much more sense than
> moving the task to the application level. I'd be astonished if IBM
> would break the rules for their well designed API so badly.

Your suspicions are correct. There is about 60 OS/2 APIs which are
known to be safe to use with high memory. There are wrappers about a
dozen other APIs. The remaining 6000 APIs are NOT (known to be) safe
to use with high memory.

It is this simple: high memory as implemented on OS/2 is not intended
for general use.

Hope this helps,
Ilya

[A complimentary Cc of this posting was NOT [per weedlist] sent to
Lars Erdmann
], who wrote in article <47bc982e$0$374$9b4e6d93@newsspool2.arcor-online.net>:
> > Or are you discussing memory-pre-digestion by DD to translate memory
> > addresses as-usable-by-ring3 into something usable-by-ring-0?
>
> No. DDs can always address memory that is addressable from ring3. No
> additional steps are necessary.

As I said, my understanding of DD architecture is very limited, but
STILL I think you must be wrong. ;-)

Different processes have different virtual memory, so if DD wants to
read/write a buffer supplied by the application, and it needs to do it
in interrupt time, it must somehow massage the address, right?

The virtual addressing in use of interrupt time is just the addressing
of the "currently running thread", right?

Thanks,
Ilya

On Thu, 21 Feb 2008 06:52:28 UTC, Ilya Zakharevich
wrote:

> Different processes have different virtual memory, so if DD wants to
> read/write a buffer supplied by the application, and it needs to do it
> in interrupt time, it must somehow massage the address, right?
>
> The virtual addressing in use of interrupt time is just the addressing
> of the "currently running thread", right?

If a driver needs access to applicaton memory at interrupt time (or to
be more correct: outside the application task's context), it needs to
map
this memory into the system area using VMProcessToGlobal. The result is
an alias that is accessable independend of the current context. The
system
area is located above VIRTUALADDRESSLIMIT and that is why we get
problems
when people are setting VIRTUALADDRESSLIMIT too high.

BTW, many drivers don't need this addressability during interrupt,
because modern DMA-driven hardware will require physical addresses
either in form of a contiguous block or an array of page lists.



--
Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
http://www.s-t.de
Please remove all characters left of the "R" in my email address

On a pleasant day while strolling in
comp.os.os2.programmer.misc, a person by the name of
Ruediger Ihle exclaimed:
> On Mon, 18 Feb 2008 19:20:31 UTC, no_spam@outgoing.net (Jeffrey Smick)
> wrote:
>
> > DosAliasMem....
> > Does not exist in help! You spend time reading .h files, I guess!
>
> It's documented in \book\addenum.inf


Amusing that the remaining OS/2 developers could not
figure out how to update the (rather complex) toolkit
help files

> DosSubAllocMem is evil anyway ;-).

But the only way to suballocate shared memory ?


--
aaronl at consultant dot com
For every expert, there is an equal and
opposite expert. - Arthur C. Clarke

[A complimentary Cc of this posting was sent to
Aaron Lawrence
], who wrote in article :
> > DosSubAllocMem is evil anyway ;-).

> But the only way to suballocate shared memory ?

In ring 3, it is trivial to do. IIRC, EMX may even have the already
shipped API to suballocate any pool...

Hope this helps,
Ilya

> I tend to read between the lines. Since the docs show an example of using
> DosAliasMem to implement buffers that are read-only to the givee and
> read-write to the owner, this is the reason the API exists. The API was
> implemented in a way that affected a minimal amount of existing kernel
> code. The fact that the alias needed to be in low memory and use an LDT
> descriptor was an unavoidable side effect to the implementation.

And meanhwile I think I figured out why the alias cannot be done by
default in DosAllocMem. Obviously you can only map up to 512MB in sum
for the whole process, not per block. I tried to create an alias for two
370MB blocks, first one was OK but for the second I got OUT_OF_MEMORY.
So this mechanism is only usable for smaller chunks used temporary.
And this explains why the API is separated.

My scenario is a bit different. I allocate a huge block (512MB and up),
put a bitmap in and draw a rectangle from it for each WM_PAINT message.
With some address calculations it might be possible to fulfill the alignment
requirements to map a partial linear area of the block but unfortunately
this does not help for bitmaps. An additional render buffer would be
required and some code that extracts the drawing region data to it.

As this somehow contradicts the idea to reduce memory usage and also hits
performance (goal is quick interactive responsive), I'll first stick with
drawing from highmem and monitor whether really problems show up and then
go the other way. With SNAP, the latest Matrox drivers and Panorama there
seems to be no problem, otherwise this would have been an issue for Firefox
and SeaMonkey already. I think almost all users use one of the drivers above.
I'll try with plain VGA driver but I think nobody will use it for more than
installation and especially not for working with graphics.

Hallo Rudi,

> If a driver needs access to applicaton memory at interrupt time (or to
> be more correct: outside the application task's context), it needs to
> map
> this memory into the system area using VMProcessToGlobal. The result is
> an alias that is accessable independend of the current context.

Does that also mean that if I only need addressability at task time (for
example if I can handle all access to a user buffer provided in an IOCTL
from within the strat routine before returning from it) that I DO NOT need
to call VMProcessToGlobal ? Does the Ring-0 DS FLAT selector also work with
"process space" addresses ?

More concrete:
FAT32.IFS should really use 48-bit far addresses to access the cache. That
would allow much larger cache sizes than 2 Megs as no allocation of GDT
selectors would be necessary.
"Today", FAT32.IFS allocates cache with VMAlloc using VMDHA_FIXED |
VMDHA_CONTIG |VMDHA_USEHIGHMEM flags (is fixed and contiguous really
necessary ? I doubt it.) but NOT with the VMDHA_PROCESS flag. That is
already allocated in system arena address space, therefore addressable out
of any process context, correct ?
Can you briefly repeat the implications of the VMDHA_USEHIGHMEM flag ? I
remember something about EARLYMEMINIT ...

Lars

On Sun, 2 Mar 2008 03:36:27 UTC, "Lars Erdmann"
wrote:

Hi Lars,

> Does that also mean that if I only need addressability at task time (for
> example if I can handle all access to a user buffer provided in an IOCTL
> from within the strat routine before returning from it) that I DO NOT need
> to call VMProcessToGlobal ? Does the Ring-0 DS FLAT selector also work with
> "process space" addresses ?

Yes.


> "Today", FAT32.IFS allocates cache with VMAlloc using VMDHA_FIXED |
> VMDHA_CONTIG |VMDHA_USEHIGHMEM flags (is fixed and contiguous really
> necessary ? I doubt it.)

Depends on what type of DMA operations are to be expected to/from that
area. The "fixed" attribute is probably O.K. to prevent the FAT32 cache
from beeing paged out.


> but NOT with the VMDHA_PROCESS flag. That is already allocated in
> system arena address space, therefore addressable out of any process
> context, correct ?

Yes. Use DOS32FLATSEL.


> Can you briefly repeat the implications of the VMDHA_USEHIGHMEM flag ?
> I remember something about EARLYMEMINIT ...

You remember correctly. There appear to be two memory pools. One below
16M and one above. When a driver allocates memory during INIT or
INIT_COMPLETE and VMDHA_USEHIGHMEM is not set, the request is fullfilled
by taking it from the "below 16M" pool. This made good sense at times,
where slave DMA was state of the art technique. Today its gets more and
more problematic, because this pool is becoming a scarce ressource. This
can be avoided by specifying VMDHA_USEHIGHMEM. In this case, the
allocation
is taken from the "above 16M" pool.

So far so good.

But unfortunately the internal workings of EARLYMEMINIT create an
unexpected side effect. The setting basically "pumps up" the
"below 16M" pool to full size of installed memory by reducing the
"above 16M" pool to zero. The means, that allocations will fail,
if EARLYMEMINIT is active and VMDHA_USEHIGHMEM is set.

The obvious workaround is to do the allocation first with
VMDHA_USEHIGHMEM set and if that fails retry it without.




--
Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
http://www.s-t.de
Please remove all characters left of the "R" in my email address