On 19 May 2007 13:15:32 GMT phil-news-nospam@ipal.net wrote:
> It seems a program cannot access BOTH the old (without large file support)
> and new (with large file support) interfaces at the same time.

Not through standard interfaces, no.

> For example,
> when large file support is not turned on, type off_t is 32 bits and off64_t
> is not defined. But when LFS is turned on, off64_t does get defined, and
> off_t is redefined as 64 bits.
>
> Is there a way to have both definitions active in their respective ways?

That can be done, but you have to call the 32-bit and 64-bit versions
directly, instead of macro'd through LFS. See lfcompile64(5) on Solaris.

Alternatively, you can nearly simulate this by always calling the
64-bit versions in your library, "promoting" args from 32-bit callers
as needed. The difference then would be that you'd have to emulate
failures related to the 32-bit limits, eg max size of a file.

> The purpose is building a library which provides certain kinds of wrapping
> around system or library calls that do deal with interface changes due to
> large file support.

If your library is an interposer, you don't need to do anything
special at all though. 32-bit and 64-bit (LFS) library calls are
independent from each other, with the appropriate call invoked via
macros at compile time when building with LFS support. So you just
need 2 versions of each function you are interposing on.

-frank

On Sat, 19 May 2007 13:00:53 -0700 Frank Cusack wrote:
| On 19 May 2007 13:15:32 GMT phil-news-nospam@ipal.net wrote:
|> It seems a program cannot access BOTH the old (without large file support)
|> and new (with large file support) interfaces at the same time.
|
| Not through standard interfaces, no.
|
|> For example,
|> when large file support is not turned on, type off_t is 32 bits and off64_t
|> is not defined. But when LFS is turned on, off64_t does get defined, and
|> off_t is redefined as 64 bits.
|>
|> Is there a way to have both definitions active in their respective ways?
|
| That can be done, but you have to call the 32-bit and 64-bit versions
| directly, instead of macro'd through LFS. See lfcompile64(5) on Solaris.
|
| Alternatively, you can nearly simulate this by always calling the
| 64-bit versions in your library, "promoting" args from 32-bit callers
| as needed. The difference then would be that you'd have to emulate
| failures related to the 32-bit limits, eg max size of a file.
|
|> The purpose is building a library which provides certain kinds of wrapping
|> around system or library calls that do deal with interface changes due to
|> large file support.
|
| If your library is an interposer, you don't need to do anything
| special at all though. 32-bit and 64-bit (LFS) library calls are
| independent from each other, with the appropriate call invoked via
| macros at compile time when building with LFS support. So you just
| need 2 versions of each function you are interposing on.

That's what I want to avoid. I don't want to create a "permanent transition"
like LFS has become. I don't want two different interfaces. I want to make
things work as they should, where the usage of an interface involves using
the correct ADT which is defined to the correct size and time. With the
exception of the fseek and ftell functions, which got replaced by fseeko
and ftello, the whole standard interface _should_ have been done by having
simply redefined all the data types. There should never have been "64"
versions of any syscalls, library functions, data types, structs, etc.
They ended up doing this for various reasons related to legacy programs.

My library has no legacy so I don't want to create any such mess for it.
So I'm just trying to understand the specifics of the existant mess to see
how I might be able to accomplish this (exactly one way to access each
element of the interface, and yet have it always work no matter how the
calling programs are accessing the POSIX interface).

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-05-19-2047@ipal.net |
|------------------------------------/-------------------------------------|

On 20 May 2007 01:53:19 GMT phil-news-nospam@ipal.net wrote:
> | If your library is an interposer, you don't need to do anything
> | special at all though. 32-bit and 64-bit (LFS) library calls are
> | independent from each other, with the appropriate call invoked via
> | macros at compile time when building with LFS support. So you just
> | need 2 versions of each function you are interposing on.
>
> That's what I want to avoid. I don't want to create a "permanent transition"
> like LFS has become. I don't want two different interfaces.

That's not possible, if backwards compatibility is to be retained.

> I want to make
> things work as they should, where the usage of an interface involves using
> the correct ADT which is defined to the correct size and time. With the
> exception of the fseek and ftell functions, which got replaced by fseeko
> and ftello, the whole standard interface _should_ have been done by having
> simply redefined all the data types. There should never have been "64"
> versions of any syscalls, library functions, data types, structs, etc.
> They ended up doing this for various reasons related to legacy programs.

I disagree strongly; it was done the correct way. Supporting
so-called legacy programs is absolutely essential.

That's not to say that the purist in me doesn't appreciate your position.

> My library has no legacy so I don't want to create any such mess for it.
> So I'm just trying to understand the specifics of the existant mess to see
> how I might be able to accomplish this (exactly one way to access each
> element of the interface, and yet have it always work no matter how the
> calling programs are accessing the POSIX interface).

I can't imagine how this could be done.

-frank

On Sat, 19 May 2007 19:18:24 -0700 Frank Cusack wrote:
| On 20 May 2007 01:53:19 GMT phil-news-nospam@ipal.net wrote:
|> | If your library is an interposer, you don't need to do anything
|> | special at all though. 32-bit and 64-bit (LFS) library calls are
|> | independent from each other, with the appropriate call invoked via
|> | macros at compile time when building with LFS support. So you just
|> | need 2 versions of each function you are interposing on.
|>
|> That's what I want to avoid. I don't want to create a "permanent transition"
|> like LFS has become. I don't want two different interfaces.
|
| That's not possible, if backwards compatibility is to be retained.

Let me clarify. There are two different ways to make reference to having
more than one interface. Clearly if someone compiles my library on a
64-bit machine, that is a truly different interface at the ABI level than
one compiled on a 32-bit machine. Similarly, at the ABI level, interfaces
would be different for different endianness, as well as padding/alignment
issues where structs could be involved, and possibly even the argument
list itself.

The above is one kind of "two different interfaces". The other is having
what might be better termed "overloaded interface" where for each part of
the interface (a function, a struct, a data type, etc), or some subset of
those parts, a corresponding variant definition is created that only makes
sense, or is only needed on, certain categories of architectures. What I
mean here are things like the added "64" functions in LFS, new data types
with "64" in them, and various other things. On a 32 bit machine these are
different. On a 64 bit machine they are pointless.

The kind of thing I want to avoid is the latter.

I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
type of architecture that provides compatibility for legacy programs with
an extra legacy library that provides the ABI for older programs that expect
"32-bit pointer / 32-bit file reference". They could also make an alternate
toolchain to compile programs in the legacy architecture (for example the
legacy library itself which can still need maintenance). But the default
should be the forward architecture.

A very small amount of the POSIX standard failed to have the ability to
"just recompile" to get correctly written programs working on the new
"32-bit pointer / 64-bit file reference" architecture (e.g. why we had
to have fseeko and ftello because of the lack of ODTs). Programs using
fseeko and ftello in lieu of fseek and ftell should be able to simply be
recompiled for the default architecture.

Too many programs would have to be fixed. So someone decided rather than
recode those programs that were broken, we'd have to recode all programs
to add large file support in the transition. So in an effort to avoid
more work, we have a lot more work. It was a bad idea from the get-go.
Someone called it a transition. Looks like it will be around for many
years, maybe decades.


|> I want to make
|> things work as they should, where the usage of an interface involves using
|> the correct ADT which is defined to the correct size and time. With the
|> exception of the fseek and ftell functions, which got replaced by fseeko
|> and ftello, the whole standard interface _should_ have been done by having
|> simply redefined all the data types. There should never have been "64"
|> versions of any syscalls, library functions, data types, structs, etc.
|> They ended up doing this for various reasons related to legacy programs.
|
| I disagree strongly; it was done the correct way. Supporting
| so-called legacy programs is absolutely essential.

This depends on the definition of support. If you want original binary
programs that source is unavailable for to also work with large files,
then that is quite a stretch. But adding the "64" extensions is in no way
a form of supporting legacy programs, either with or without source.
With two different ABIs, you have legacy support. No need to bloat the
API to do so. You might have 2 library binaries around, and intimate
libraries like libc might well have to be coded to understand which it
is being compiled for (because the kernel might only have the LFS KABI).


| That's not to say that the purist in me doesn't appreciate your position.

But it's more than just a purist issue. It's also a practical issue.
Why force recoding all programs when it would have only been necessary
to recode the broken subset (which I am not convinced is so large).


|> My library has no legacy so I don't want to create any such mess for it.
|> So I'm just trying to understand the specifics of the existant mess to see
|> how I might be able to accomplish this (exactly one way to access each
|> element of the interface, and yet have it always work no matter how the
|> calling programs are accessing the POSIX interface).
|
| I can't imagine how this could be done.

This is my quest.

At this point it seems the way I will do so is to have my API be always
of the "implicit large file" type when either implicit or explicit large
files are used by calling programs. A calling program using implicit
would not really see a difference. A calling program using explicit is
going to have to take notice and deal with API elements with great care.
And those in my library would be the large file type but without the large
file names. It's the ability to be able to call both that will be lost.
So a program might be able to call both lstat() and lstat64() in libc to
reach the kernel in two different ways (and libc may well be translating
lstat() to a 64-bit underlying KABI where lstat64() would be a straight
call). But it would only have the 64-bit API in my library. A program
being compiled entirely without LFS would get the non-LFS API, and have
to link with a non-LFS binary (assuming one gets compiled).

I will have to decide if I want to make the default make configuration
create both 32 bit and 64 bit library binaries, or just the 64 bit.

One thing I think I need to detect on a given platform is if LFS was a
choice at all. If the machine is a true 64-bit machine, that might not
be entirely the obvious case, since there may still be 32-bit program
emulation going on, with support for a 32-bit pointer ABI which could
have either 32-bit or 64-bit file access data sizes, or both.

But I definitely do NOT want to see "64" variations in an API on a true
64-bit machine for programs compiled to the full architecture. Yet with
the POSIX LFS way, we apparently will be stuck with that. It's not so
much a transition as it is a trap.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-05-20-0813@ipal.net |
|------------------------------------/-------------------------------------|

phil-news-nospam@ipal.net writes:

[...]

> I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
> type of architecture that provides compatibility for legacy programs with
> an extra legacy library that provides the ABI for older programs that expect
> "32-bit pointer / 32-bit file reference". They could also make an alternate
> toolchain to compile programs in the legacy architecture (for example the
> legacy library itself which can still need maintenance). But the default
> should be the forward architecture.

Reusing my example from colds: Could you please provide a single
reason why a DNS-server of some kind, running on a 32-bit
architecture, should use 64-bit variables for its UDP communication
calls?

NB: A reason, meaning "some advantage to be gained by doing so', not
only 'I want to process my recorded television shows and don't give a
****", which is all I have read so far.

On Sun, 20 May 2007 16:01:43 +0200 Rainer Weikusat wrote:
| phil-news-nospam@ipal.net writes:
|
| [...]
|
|> I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
|> type of architecture that provides compatibility for legacy programs with
|> an extra legacy library that provides the ABI for older programs that expect
|> "32-bit pointer / 32-bit file reference". They could also make an alternate
|> toolchain to compile programs in the legacy architecture (for example the
|> legacy library itself which can still need maintenance). But the default
|> should be the forward architecture.
|
| Reusing my example from colds: Could you please provide a single
| reason why a DNS-server of some kind, running on a 32-bit
| architecture, should use 64-bit variables for its UDP communication
| calls?

No. Why should I provide such an example? I've never suggested that it
is necessary to do this.


| NB: A reason, meaning "some advantage to be gained by doing so', not
| only 'I want to process my recorded television shows and don't give a
| ****", which is all I have read so far.

How about a reason for asking for this?

Or is it that you are not making the distinction between 64-bit variables
in general, and the few 64-bit variable data types needed to access file
positions and sizes when files might be (because the system supports it)
larger than 2GB or 4GB?

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-05-20-1042@ipal.net |
|------------------------------------/-------------------------------------|

phil-news-nospam@ipal.net writes:
> On Sun, 20 May 2007 16:01:43 +0200 Rainer Weikusat wrote:
> | phil-news-nospam@ipal.net writes:
> |
> | [...]
> |
> |> I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
> |> type of architecture that provides compatibility for legacy programs with
> |> an extra legacy library that provides the ABI for older programs that expect
> |> "32-bit pointer / 32-bit file reference". They could also make an alternate
> |> toolchain to compile programs in the legacy architecture (for example the
> |> legacy library itself which can still need maintenance). But the default
> |> should be the forward architecture.
> |
> | Reusing my example from colds: Could you please provide a single
> | reason why a DNS-server of some kind, running on a 32-bit
> | architecture, should use 64-bit variables for its UDP communication
> | calls?
>
> No. Why should I provide such an example? I've never suggested that it
> is necessary to do this.
>
>
> | NB: A reason, meaning "some advantage to be gained by doing so', not
> | only 'I want to process my recorded television shows and don't give a
> | ****", which is all I have read so far.
>
> How about a reason for asking for this?
>
> Or is it that you are not making the distinction between 64-bit variables
> in general, and the few 64-bit variable data types needed to access file
> positions and sizes when files might be (because the system supports it)
> larger than 2GB or 4GB?

I was making the erronous assumption that 'these few variable data
types' would include 64-bit-I/O support as well (ie ssize_t and
size_t) which they (at least according to the headers on the system I
am currently using) don't.

On Sun, 20 May 2007 18:56:27 +0200 Rainer Weikusat wrote:
| phil-news-nospam@ipal.net writes:
|> On Sun, 20 May 2007 16:01:43 +0200 Rainer Weikusat wrote:
|> | phil-news-nospam@ipal.net writes:
|> |
|> | [...]
|> |
|> |> I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
|> |> type of architecture that provides compatibility for legacy programs with
|> |> an extra legacy library that provides the ABI for older programs that expect
|> |> "32-bit pointer / 32-bit file reference". They could also make an alternate
|> |> toolchain to compile programs in the legacy architecture (for example the
|> |> legacy library itself which can still need maintenance). But the default
|> |> should be the forward architecture.
|> |
|> | Reusing my example from colds: Could you please provide a single
|> | reason why a DNS-server of some kind, running on a 32-bit
|> | architecture, should use 64-bit variables for its UDP communication
|> | calls?
|>
|> No. Why should I provide such an example? I've never suggested that it
|> is necessary to do this.
|>
|>
|> | NB: A reason, meaning "some advantage to be gained by doing so', not
|> | only 'I want to process my recorded television shows and don't give a
|> | ****", which is all I have read so far.
|>
|> How about a reason for asking for this?
|>
|> Or is it that you are not making the distinction between 64-bit variables
|> in general, and the few 64-bit variable data types needed to access file
|> positions and sizes when files might be (because the system supports it)
|> larger than 2GB or 4GB?
|
| I was making the erronous assumption that 'these few variable data
| types' would include 64-bit-I/O support as well (ie ssize_t and
| size_t) which they (at least according to the headers on the system I
| am currently using) don't.

It would be useful to have 64-bit size_t and ssize_t if it were useful to
read in more than 4GB of a file at once. On a 32-bit architecture, that
cannot be done (at least within the scope of normal I/O call interfaces).
I suppose someone might find a cause to do something like that on a 64-bit
architecture, and indeed, you do find size_t and ssize_t (and pointers and
the like) that big (and part of the reason you do see a sudden jump up in
the memory needed when moving from 32-bit to 64-bit architecture).

What was all the purpose of having opaque data types that could be defined
in a different size on another machine? Was it just as a means to document
what a given variable was trying to be? Or was it a means to be able to
recompile a well written program to another architecture in a language that
was supposed to make this easier to do than say assembly language?

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-05-21-0751@ipal.net |
|------------------------------------/-------------------------------------|

phil-news-nospam wrote:

> So I'm just trying to understand the specifics of the existant mess to see
> how I might be able to accomplish this (exactly one way to access each
> element of the interface, and yet have it always work no matter how the
> calling programs are accessing the POSIX interface).

You can't, at least not portably.

As far as POSIX is concerned, ILP32_OFF32 and ILP32_OFFBIG are
separate programming environments, just as ILP32_OFF32 and
LP64_OFF64 are separate. You don't expect to be able to mix
ILP32_OFF32 and LP64_OFF64 object files, so you shouldn't expect to
be able to do it with ILP32_OFF32 and ILP32_OFFBIG either.

The fact that the usual way that ILP32_OFFBIG is implemented is as
a variation of ILP32_OFF32 with some identifiers translated "behind
the scenes" is an internal implementation detail. There is no
reason why a POSIX implementation that offers both programming
environments couldn't have separate headers and libraries for each,
and report errors if you try to mix them.

--
Geoff Clare

On Mon, 21 May 2007 14:16:48 +0100 Geoff Clare wrote:
| phil-news-nospam wrote:
|
|> So I'm just trying to understand the specifics of the existant mess to see
|> how I might be able to accomplish this (exactly one way to access each
|> element of the interface, and yet have it always work no matter how the
|> calling programs are accessing the POSIX interface).
|
| You can't, at least not portably.
|
| As far as POSIX is concerned, ILP32_OFF32 and ILP32_OFFBIG are
| separate programming environments, just as ILP32_OFF32 and
| LP64_OFF64 are separate. You don't expect to be able to mix
| ILP32_OFF32 and LP64_OFF64 object files, so you shouldn't expect to
| be able to do it with ILP32_OFF32 and ILP32_OFFBIG either.
|
| The fact that the usual way that ILP32_OFFBIG is implemented is as
| a variation of ILP32_OFF32 with some identifiers translated "behind
| the scenes" is an internal implementation detail. There is no
| reason why a POSIX implementation that offers both programming
| environments couldn't have separate headers and libraries for each,
| and report errors if you try to mix them.

I don't have a problem with more than one programming environment. What
I have a problem with is the mixed programming environment where some of
the calls have added versions specifically for 64-bit file offsets. I do
not want to have some of my functions have 64-bit versions. If my library
is used in a pure 32-bit environment or pure 64-bit environment, even on
the same machine, it should just work fine. The problem is when there is
the mixed environment where a program could call either syscall, such as
both stat() and stat64(). What I am thinking of doing in that case is to
make my functions be only the 64-bit version, but with the normal name.
The complication is that the headers will have to change under this case
so the prototypes use the 64-bit variant names.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-05-21-1341@ipal.net |
|------------------------------------/-------------------------------------|

There's a lot here that will take time to write an adequate reply, and
I doubt I'll find the time. Just a quick point:

On 20 May 2007 13:47:01 GMT phil-news-nospam@ipal.net wrote:
> At this point it seems the way I will do so is to have my API be always
> of the "implicit large file" type when either implicit or explicit large
> files are used by calling programs. A calling program using implicit
> would not really see a difference.

One thing I still don't understand is if your API is a shim or similar
for the POSIX API, or if it's something completely different.

If it's an equivalent/shim/etc, then having implicit large files is a
mistake if your goal is to have existing programs (say, 'diff') use
your library instead of or on top of POSIX. You would find that
(e.g.) programs would happily open 2.1G files instead of failing, and
then get lost as to their position in the file when they go past the
2G mark. Now of course (for this specific example) whether or not
this is a problem depends on the nature of the program, but in general
you'd have to audit any program you want to link with your library.

If a program has to be re-written or modified to use your API, it is
of course a different story.

....
> But I definitely do NOT want to see "64" variations in an API on a true
> 64-bit machine for programs compiled to the full architecture. Yet with
> the POSIX LFS way, we apparently will be stuck with that. It's not so
> much a transition as it is a trap.

POSIX LFS has a large legacy of programs to support! You may not have
that defining handicap.

-frank

On Sat, 19 May 2007 13:15:32 +0000, phil-news-nospam wrote:

> It seems a program cannot access BOTH the old (without large file
> support) and new (with large file support) interfaces at the same time.
> For example, when large file support is not turned on, type off_t is 32
> bits and off64_t is not defined. But when LFS is turned on, off64_t
> does get defined, and off_t is redefined as 64 bits.

Doesn't _just_ having _LARGEFILE64_SOURCE=1 do the right thing here? All
of the (informal) documentation I can find implies it does.

> Is there a way to have both definitions active in their respective ways?
>
> The purpose is building a library which provides certain kinds of
> wrapping around system or library calls that do deal with interface
> changes due to large file support.

Well I did the above for a library and it seems to work, although
probably all of the users want LFS with full 64bits ... but I thought I
tested it with off_t == 32bit as well, a little bit at least.

--
James Antill -- james@and.org
http://www.and.org/and-httpd/ -- $2,000 security guarantee
http://www.and.org/vstr/

On Mon, 21 May 2007 14:43:44 -0700 Frank Cusack wrote:
| There's a lot here that will take time to write an adequate reply, and
| I doubt I'll find the time. Just a quick point:
|
| On 20 May 2007 13:47:01 GMT phil-news-nospam@ipal.net wrote:
|> At this point it seems the way I will do so is to have my API be always
|> of the "implicit large file" type when either implicit or explicit large
|> files are used by calling programs. A calling program using implicit
|> would not really see a difference.
|
| One thing I still don't understand is if your API is a shim or similar
| for the POSIX API, or if it's something completely different.

It's not exactly a shim. Some of the functions are simplistic changes
to syscalls, such as a version of stat() that takes multiple string
arguments and forms the path by inserting "/" between each argument
and joining that all together as one string for the real syscall. It
is obviously nor just a look-alike shim, as the API design is different.
Another is a very different way to do file tree recursion than either
the existing ftw or fts functions. It would be affected in terms of
the struct stat it returns to callers for each object returned (unlike
ftw it returns to the caller with each object).

Examples:
http://libh.slashusr.org/source/io/s...tat_join_sep.c
http://libh.slashusr.org/source/ftr/...h/ftr_header.h

See macros ftr_stat() and ftr_stat_ptr() in the latter URL. Those macros
and many others like them provide info about the file last found by the
ftr_get() function, for the given ftr object.


| If it's an equivalent/shim/etc, then having implicit large files is a
| mistake if your goal is to have existing programs (say, 'diff') use
| your library instead of or on top of POSIX. You would find that
| (e.g.) programs would happily open 2.1G files instead of failing, and
| then get lost as to their position in the file when they go past the
| 2G mark. Now of course (for this specific example) whether or not
| this is a problem depends on the nature of the program, but in general
| you'd have to audit any program you want to link with your library.

It's not a shim in the sense of causing existing programs to effectively
call my library, even if recompiled to do so (as in a macro shim). It
is a different API that requires deliberate programming to use. A lot
of it provides programming conveneience (useful for new programs, but
not worth recoding old programs). A lot of it provides new features.


| If a program has to be re-written or modified to use your API, it is
| of course a different story.

It does have to be re-written or modified. It is a different API. But
it does make use of POSIX data types, structs, and functions (some even
in macros that would be compiled at caller compile time).


|> But I definitely do NOT want to see "64" variations in an API on a true
|> 64-bit machine for programs compiled to the full architecture. Yet with
|> the POSIX LFS way, we apparently will be stuck with that. It's not so
|> much a transition as it is a trap.
|
| POSIX LFS has a large legacy of programs to support! You may not have
| that defining handicap.

No, I certainly do not have that legacy. But even if I did I would not
take the approach POSIX LFS did. I would want to make it work for both
old and new programs, even if that meant having 2 or even 3 different
binary libraries (at least one being there for legacy ABI compatibility
for programs w/o source code available). I think 2 would be sufficient,
though. My objective is that none of my API have any "64" functions or
data types of my own creation. Everything would work with large file
PSOXI API elements, unless the calling programs don't use LFS. Then my
library might need a legacy binary so that it can properly fail when the
LFS functions would otherwise succeed, such as failing on a 5GB file being
found in a tree recursion, which I believe should be the correct behaviour
when the stat info being returned to the caller is from the legacy API.
If it didn't fail in such a case, the information cannot be properly
returned.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-05-21-1719@ipal.net |
|------------------------------------/-------------------------------------|

On Mon, 21 May 2007 22:18:30 -0000 James Antill wrote:
| On Sat, 19 May 2007 13:15:32 +0000, phil-news-nospam wrote:
|
|> It seems a program cannot access BOTH the old (without large file
|> support) and new (with large file support) interfaces at the same time.
|> For example, when large file support is not turned on, type off_t is 32
|> bits and off64_t is not defined. But when LFS is turned on, off64_t
|> does get defined, and off_t is redefined as 64 bits.
|
| Doesn't _just_ having _LARGEFILE64_SOURCE=1 do the right thing here? All
| of the (informal) documentation I can find implies it does.

But that is not the only way a calling program might do this.


|> Is there a way to have both definitions active in their respective ways?
|>
|> The purpose is building a library which provides certain kinds of
|> wrapping around system or library calls that do deal with interface
|> changes due to large file support.
|
| Well I did the above for a library and it seems to work, although
| probably all of the users want LFS with full 64bits ... but I thought I
| tested it with off_t == 32bit as well, a little bit at least.

There are at least 8 different variations of doing LFS, based on what can
be selected through CPP definitions. Apparently they fall into 3 major
groups: no LFS, explicit LFS, implicit LFS. I think implicit LFS will
be the easiest. But apparently a lot of programs have chosen to go with
explicit LFS. My thinking at the moment is to just treat explicit the
same as implicit, since there is very little legacy in usage of my library.
That is, if you choose to have both stat() and stat64(), and also use my
library, you've already committed to doing some amount of recoding, and
committed to conding usage of my library, so you must make sure all usage
of my library is consistent with the LFS support you are using. My macros
will probably have to make use of the "64" variants of POSIX symbols in
the explicit cases, but I think that won't be a problem.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-05-21-1743@ipal.net |
|------------------------------------/-------------------------------------|

On 21 May 2007 18:50:56 GMT
phil-news-nospam@ipal.net wrote:

> I don't have a problem with more than one programming environment.
> What I have a problem with is the mixed programming environment where
> some of the calls have added versions specifically for 64-bit file
> offsets. I do not want to have some of my functions have 64-bit
> versions.

This was introduced to allow 32-bit systems to access large files. The
transition from 32-bit systems via 32-bit systems with large file
support, to 64-bit systems was handled with considerable deftness by
the (then relevant) Unix vendors.

The only time you would need a 32 and 64 bit version of a system call
is when you're running a 32-bit machine with large file support.
Most Unix systems -even though all are now 64-bit- continue to support
32 bit programs, and 32-bit programs with large file support. This is a
feature.

If you restrict your library to a 64-bit version only, there is no need
to worry about large file support.

--
Stefaan A Eeckels
--
He who will not reason, is a bigot;
he who cannot is a fool;
and he who dares not is a slave. (Sir William Drummond)

On 21 May 2007 22:49:25 GMT
phil-news-nospam@ipal.net wrote:

> My macros will probably have to make use of the "64" variants of
> POSIX symbols in the explicit cases,...

Why? We're talking about a transitional interface here. The only time
you need to use the lf64 functions is when a 32-bit program needs to
access large files.

There are valid reasons to continue to use 32-bit programs (on SPARC,
they are marginally faster, the program needs to run on old 32-bit
systems, etc), but there are not many compelling reasons to go out of
one's way to support a transitional interface in new developments.

Just go 64-bit, and be happy.

--
Stefaan A Eeckels
--
You know, it is almost always the case in the real world that something
is "fair" when you like it and "unfair" when you don't.
-- Jeffrey Siegal in gnu.misc.discuss

On Mon, 21 May 2007 22:18:30 -0000 James Antill wrote:
> On Sat, 19 May 2007 13:15:32 +0000, phil-news-nospam wrote:
>
>> It seems a program cannot access BOTH the old (without large file
>> support) and new (with large file support) interfaces at the same time.
>> For example, when large file support is not turned on, type off_t is 32
>> bits and off64_t is not defined. But when LFS is turned on, off64_t
>> does get defined, and off_t is redefined as 64 bits.
>
> Doesn't _just_ having _LARGEFILE64_SOURCE=1 do the right thing here? All
> of the (informal) documentation I can find implies it does.

For an application, yes. But the OP is trying to avoid having two
interfaces (the LFS one and the non-LFS one) for each function, in a
library. What _LARGEFILE64_SOURCE=1 (or more correctly, `getconf
LFS_CFLAGS`) does is select the LFS interfaces for various functions.

>> Is there a way to have both definitions active in their respective ways?
>>
>> The purpose is building a library which provides certain kinds of
>> wrapping around system or library calls that do deal with interface
>> changes due to large file support.
>
> Well I did the above for a library and it seems to work, although
> probably all of the users want LFS with full 64bits ... but I thought I
> tested it with off_t == 32bit as well, a little bit at least.

There is no [good] way for a function to determine if its caller used
an LFS or non-LFS datatype. Even if you want to do the nasty stuff to
figure it out, LFS programs call different functions than non-LFS
functions; e.g. open64 instead of open. So to work with existing
programs and not require a recompile, a library must provide both
interfaces. Even if you are adding some brand new interface, if you
are using the types that are changed with LFS (off_t et al.) you
must provide 2 interfaces to deal with the different sized args.

libelf is a good example of a library that has LFS problems, but that
isn't itself part of the LFS "standard". This is unfortunate, since
implementors have been reluctant (or just didn't think of it) to
"extend" the LFS interface to libelf, ie to define multiple versions
of the libelf routines based on LFS or non-LFS settings. Therefore
however you or your vendor compiles libelf is the only way your
program that uses libelf can be compiled, regardless of the other
(LFS) functionality you may want. Well, you could write a shim to
translate the data types. Horrible.

-frank

On Tue, 22 May 2007 01:03:58 +0200 Stefaan A Eeckels wrote:
| On 21 May 2007 18:50:56 GMT
| phil-news-nospam@ipal.net wrote:
|
|> I don't have a problem with more than one programming environment.
|> What I have a problem with is the mixed programming environment where
|> some of the calls have added versions specifically for 64-bit file
|> offsets. I do not want to have some of my functions have 64-bit
|> versions.
|
| This was introduced to allow 32-bit systems to access large files. The
| transition from 32-bit systems via 32-bit systems with large file
| support, to 64-bit systems was handled with considerable deftness by
| the (then relevant) Unix vendors.

I don't thinks so. I think it was done the way they did it so they would
not have to undo some misguided attempts to handle large files that had
already be utilized.


| The only time you would need a 32 and 64 bit version of a system call
| is when you're running a 32-bit machine with large file support.
| Most Unix systems -even though all are now 64-bit- continue to support
| 32 bit programs, and 32-bit programs with large file support. This is a
| feature.

I cannot figure out when a program would need both the small file and large
file calls in the very same program.


| If you restrict your library to a 64-bit version only, there is no need
| to worry about large file support.

There is the need to work with those programs that select the explicit LFS
mode, which means stat() is still for small files and stat64() is for large
files (although some systems might only do one of them at the KABI).

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-05-21-1944@ipal.net |
|------------------------------------/-------------------------------------|

On Tue, 22 May 2007 01:23:37 +0200 Stefaan A Eeckels wrote:
| On 21 May 2007 22:49:25 GMT
| phil-news-nospam@ipal.net wrote:
|
|> My macros will probably have to make use of the "64" variants of
|> POSIX symbols in the explicit cases,...
|
| Why? We're talking about a transitional interface here. The only time
| you need to use the lf64 functions is when a 32-bit program needs to
| access large files.

And that seems to be widely used.


| There are valid reasons to continue to use 32-bit programs (on SPARC,
| they are marginally faster, the program needs to run on old 32-bit
| systems, etc), but there are not many compelling reasons to go out of
| one's way to support a transitional interface in new developments.
|
| Just go 64-bit, and be happy.

When I'm writing a program, that concept works fine. But for a library
one has to deal with how the prgorams calling it are written.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-05-21-1948@ipal.net |
|------------------------------------/-------------------------------------|