large file support && ! large file support - Unix

This is a discussion on large file support && ! large file support - Unix ; It seems a program cannot access BOTH the old (without large file support) and new (with large file support) interfaces at the same time. For example, when large file support is not turned on, type off_t is 32 bits and ...

+ Reply to Thread
Page 1 of 3 1 2 3 LastLast
Results 1 to 20 of 54

Thread: large file support && ! large file support

  1. large file support && ! large file support

    It seems a program cannot access BOTH the old (without large file support)
    and new (with large file support) interfaces at the same time. For example,
    when large file support is not turned on, type off_t is 32 bits and off64_t
    is not defined. But when LFS is turned on, off64_t does get defined, and
    off_t is redefined as 64 bits.

    Is there a way to have both definitions active in their respective ways?

    The purpose is building a library which provides certain kinds of wrapping
    around system or library calls that do deal with interface changes due to
    large file support.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-19-0809@ipal.net |
    |------------------------------------/-------------------------------------|

  2. Re: large file support && ! large file support

    On 19 May 2007 13:15:32 GMT phil-news-nospam@ipal.net wrote:
    > It seems a program cannot access BOTH the old (without large file support)
    > and new (with large file support) interfaces at the same time.


    Not through standard interfaces, no.

    > For example,
    > when large file support is not turned on, type off_t is 32 bits and off64_t
    > is not defined. But when LFS is turned on, off64_t does get defined, and
    > off_t is redefined as 64 bits.
    >
    > Is there a way to have both definitions active in their respective ways?


    That can be done, but you have to call the 32-bit and 64-bit versions
    directly, instead of macro'd through LFS. See lfcompile64(5) on Solaris.

    Alternatively, you can nearly simulate this by always calling the
    64-bit versions in your library, "promoting" args from 32-bit callers
    as needed. The difference then would be that you'd have to emulate
    failures related to the 32-bit limits, eg max size of a file.

    > The purpose is building a library which provides certain kinds of wrapping
    > around system or library calls that do deal with interface changes due to
    > large file support.


    If your library is an interposer, you don't need to do anything
    special at all though. 32-bit and 64-bit (LFS) library calls are
    independent from each other, with the appropriate call invoked via
    macros at compile time when building with LFS support. So you just
    need 2 versions of each function you are interposing on.

    -frank

  3. Re: large file support && ! large file support

    On Sat, 19 May 2007 13:00:53 -0700 Frank Cusack wrote:
    | On 19 May 2007 13:15:32 GMT phil-news-nospam@ipal.net wrote:
    |> It seems a program cannot access BOTH the old (without large file support)
    |> and new (with large file support) interfaces at the same time.
    |
    | Not through standard interfaces, no.
    |
    |> For example,
    |> when large file support is not turned on, type off_t is 32 bits and off64_t
    |> is not defined. But when LFS is turned on, off64_t does get defined, and
    |> off_t is redefined as 64 bits.
    |>
    |> Is there a way to have both definitions active in their respective ways?
    |
    | That can be done, but you have to call the 32-bit and 64-bit versions
    | directly, instead of macro'd through LFS. See lfcompile64(5) on Solaris.
    |
    | Alternatively, you can nearly simulate this by always calling the
    | 64-bit versions in your library, "promoting" args from 32-bit callers
    | as needed. The difference then would be that you'd have to emulate
    | failures related to the 32-bit limits, eg max size of a file.
    |
    |> The purpose is building a library which provides certain kinds of wrapping
    |> around system or library calls that do deal with interface changes due to
    |> large file support.
    |
    | If your library is an interposer, you don't need to do anything
    | special at all though. 32-bit and 64-bit (LFS) library calls are
    | independent from each other, with the appropriate call invoked via
    | macros at compile time when building with LFS support. So you just
    | need 2 versions of each function you are interposing on.

    That's what I want to avoid. I don't want to create a "permanent transition"
    like LFS has become. I don't want two different interfaces. I want to make
    things work as they should, where the usage of an interface involves using
    the correct ADT which is defined to the correct size and time. With the
    exception of the fseek and ftell functions, which got replaced by fseeko
    and ftello, the whole standard interface _should_ have been done by having
    simply redefined all the data types. There should never have been "64"
    versions of any syscalls, library functions, data types, structs, etc.
    They ended up doing this for various reasons related to legacy programs.

    My library has no legacy so I don't want to create any such mess for it.
    So I'm just trying to understand the specifics of the existant mess to see
    how I might be able to accomplish this (exactly one way to access each
    element of the interface, and yet have it always work no matter how the
    calling programs are accessing the POSIX interface).

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-19-2047@ipal.net |
    |------------------------------------/-------------------------------------|

  4. Re: large file support && ! large file support

    On 20 May 2007 01:53:19 GMT phil-news-nospam@ipal.net wrote:
    > | If your library is an interposer, you don't need to do anything
    > | special at all though. 32-bit and 64-bit (LFS) library calls are
    > | independent from each other, with the appropriate call invoked via
    > | macros at compile time when building with LFS support. So you just
    > | need 2 versions of each function you are interposing on.
    >
    > That's what I want to avoid. I don't want to create a "permanent transition"
    > like LFS has become. I don't want two different interfaces.


    That's not possible, if backwards compatibility is to be retained.

    > I want to make
    > things work as they should, where the usage of an interface involves using
    > the correct ADT which is defined to the correct size and time. With the
    > exception of the fseek and ftell functions, which got replaced by fseeko
    > and ftello, the whole standard interface _should_ have been done by having
    > simply redefined all the data types. There should never have been "64"
    > versions of any syscalls, library functions, data types, structs, etc.
    > They ended up doing this for various reasons related to legacy programs.


    I disagree strongly; it was done the correct way. Supporting
    so-called legacy programs is absolutely essential.

    That's not to say that the purist in me doesn't appreciate your position.

    > My library has no legacy so I don't want to create any such mess for it.
    > So I'm just trying to understand the specifics of the existant mess to see
    > how I might be able to accomplish this (exactly one way to access each
    > element of the interface, and yet have it always work no matter how the
    > calling programs are accessing the POSIX interface).


    I can't imagine how this could be done.

    -frank

  5. Re: large file support && ! large file support

    On Sat, 19 May 2007 19:18:24 -0700 Frank Cusack wrote:
    | On 20 May 2007 01:53:19 GMT phil-news-nospam@ipal.net wrote:
    |> | If your library is an interposer, you don't need to do anything
    |> | special at all though. 32-bit and 64-bit (LFS) library calls are
    |> | independent from each other, with the appropriate call invoked via
    |> | macros at compile time when building with LFS support. So you just
    |> | need 2 versions of each function you are interposing on.
    |>
    |> That's what I want to avoid. I don't want to create a "permanent transition"
    |> like LFS has become. I don't want two different interfaces.
    |
    | That's not possible, if backwards compatibility is to be retained.

    Let me clarify. There are two different ways to make reference to having
    more than one interface. Clearly if someone compiles my library on a
    64-bit machine, that is a truly different interface at the ABI level than
    one compiled on a 32-bit machine. Similarly, at the ABI level, interfaces
    would be different for different endianness, as well as padding/alignment
    issues where structs could be involved, and possibly even the argument
    list itself.

    The above is one kind of "two different interfaces". The other is having
    what might be better termed "overloaded interface" where for each part of
    the interface (a function, a struct, a data type, etc), or some subset of
    those parts, a corresponding variant definition is created that only makes
    sense, or is only needed on, certain categories of architectures. What I
    mean here are things like the added "64" functions in LFS, new data types
    with "64" in them, and various other things. On a 32 bit machine these are
    different. On a 64 bit machine they are pointless.

    The kind of thing I want to avoid is the latter.

    I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
    type of architecture that provides compatibility for legacy programs with
    an extra legacy library that provides the ABI for older programs that expect
    "32-bit pointer / 32-bit file reference". They could also make an alternate
    toolchain to compile programs in the legacy architecture (for example the
    legacy library itself which can still need maintenance). But the default
    should be the forward architecture.

    A very small amount of the POSIX standard failed to have the ability to
    "just recompile" to get correctly written programs working on the new
    "32-bit pointer / 64-bit file reference" architecture (e.g. why we had
    to have fseeko and ftello because of the lack of ODTs). Programs using
    fseeko and ftello in lieu of fseek and ftell should be able to simply be
    recompiled for the default architecture.

    Too many programs would have to be fixed. So someone decided rather than
    recode those programs that were broken, we'd have to recode all programs
    to add large file support in the transition. So in an effort to avoid
    more work, we have a lot more work. It was a bad idea from the get-go.
    Someone called it a transition. Looks like it will be around for many
    years, maybe decades.


    |> I want to make
    |> things work as they should, where the usage of an interface involves using
    |> the correct ADT which is defined to the correct size and time. With the
    |> exception of the fseek and ftell functions, which got replaced by fseeko
    |> and ftello, the whole standard interface _should_ have been done by having
    |> simply redefined all the data types. There should never have been "64"
    |> versions of any syscalls, library functions, data types, structs, etc.
    |> They ended up doing this for various reasons related to legacy programs.
    |
    | I disagree strongly; it was done the correct way. Supporting
    | so-called legacy programs is absolutely essential.

    This depends on the definition of support. If you want original binary
    programs that source is unavailable for to also work with large files,
    then that is quite a stretch. But adding the "64" extensions is in no way
    a form of supporting legacy programs, either with or without source.
    With two different ABIs, you have legacy support. No need to bloat the
    API to do so. You might have 2 library binaries around, and intimate
    libraries like libc might well have to be coded to understand which it
    is being compiled for (because the kernel might only have the LFS KABI).


    | That's not to say that the purist in me doesn't appreciate your position.

    But it's more than just a purist issue. It's also a practical issue.
    Why force recoding all programs when it would have only been necessary
    to recode the broken subset (which I am not convinced is so large).


    |> My library has no legacy so I don't want to create any such mess for it.
    |> So I'm just trying to understand the specifics of the existant mess to see
    |> how I might be able to accomplish this (exactly one way to access each
    |> element of the interface, and yet have it always work no matter how the
    |> calling programs are accessing the POSIX interface).
    |
    | I can't imagine how this could be done.

    This is my quest.

    At this point it seems the way I will do so is to have my API be always
    of the "implicit large file" type when either implicit or explicit large
    files are used by calling programs. A calling program using implicit
    would not really see a difference. A calling program using explicit is
    going to have to take notice and deal with API elements with great care.
    And those in my library would be the large file type but without the large
    file names. It's the ability to be able to call both that will be lost.
    So a program might be able to call both lstat() and lstat64() in libc to
    reach the kernel in two different ways (and libc may well be translating
    lstat() to a 64-bit underlying KABI where lstat64() would be a straight
    call). But it would only have the 64-bit API in my library. A program
    being compiled entirely without LFS would get the non-LFS API, and have
    to link with a non-LFS binary (assuming one gets compiled).

    I will have to decide if I want to make the default make configuration
    create both 32 bit and 64 bit library binaries, or just the 64 bit.

    One thing I think I need to detect on a given platform is if LFS was a
    choice at all. If the machine is a true 64-bit machine, that might not
    be entirely the obvious case, since there may still be 32-bit program
    emulation going on, with support for a 32-bit pointer ABI which could
    have either 32-bit or 64-bit file access data sizes, or both.

    But I definitely do NOT want to see "64" variations in an API on a true
    64-bit machine for programs compiled to the full architecture. Yet with
    the POSIX LFS way, we apparently will be stuck with that. It's not so
    much a transition as it is a trap.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-20-0813@ipal.net |
    |------------------------------------/-------------------------------------|

  6. Re: large file support && ! large file support

    phil-news-nospam@ipal.net writes:

    [...]

    > I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
    > type of architecture that provides compatibility for legacy programs with
    > an extra legacy library that provides the ABI for older programs that expect
    > "32-bit pointer / 32-bit file reference". They could also make an alternate
    > toolchain to compile programs in the legacy architecture (for example the
    > legacy library itself which can still need maintenance). But the default
    > should be the forward architecture.


    Reusing my example from colds: Could you please provide a single
    reason why a DNS-server of some kind, running on a 32-bit
    architecture, should use 64-bit variables for its UDP communication
    calls?

    NB: A reason, meaning "some advantage to be gained by doing so', not
    only 'I want to process my recorded television shows and don't give a
    ****", which is all I have read so far.

  7. Re: large file support && ! large file support

    On Sun, 20 May 2007 16:01:43 +0200 Rainer Weikusat wrote:
    | phil-news-nospam@ipal.net writes:
    |
    | [...]
    |
    |> I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
    |> type of architecture that provides compatibility for legacy programs with
    |> an extra legacy library that provides the ABI for older programs that expect
    |> "32-bit pointer / 32-bit file reference". They could also make an alternate
    |> toolchain to compile programs in the legacy architecture (for example the
    |> legacy library itself which can still need maintenance). But the default
    |> should be the forward architecture.
    |
    | Reusing my example from colds: Could you please provide a single
    | reason why a DNS-server of some kind, running on a 32-bit
    | architecture, should use 64-bit variables for its UDP communication
    | calls?

    No. Why should I provide such an example? I've never suggested that it
    is necessary to do this.


    | NB: A reason, meaning "some advantage to be gained by doing so', not
    | only 'I want to process my recorded television shows and don't give a
    | ****", which is all I have read so far.

    How about a reason for asking for this?

    Or is it that you are not making the distinction between 64-bit variables
    in general, and the few 64-bit variable data types needed to access file
    positions and sizes when files might be (because the system supports it)
    larger than 2GB or 4GB?

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-20-1042@ipal.net |
    |------------------------------------/-------------------------------------|

  8. Re: large file support && ! large file support

    phil-news-nospam@ipal.net writes:
    > On Sun, 20 May 2007 16:01:43 +0200 Rainer Weikusat wrote:
    > | phil-news-nospam@ipal.net writes:
    > |
    > | [...]
    > |
    > |> I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
    > |> type of architecture that provides compatibility for legacy programs with
    > |> an extra legacy library that provides the ABI for older programs that expect
    > |> "32-bit pointer / 32-bit file reference". They could also make an alternate
    > |> toolchain to compile programs in the legacy architecture (for example the
    > |> legacy library itself which can still need maintenance). But the default
    > |> should be the forward architecture.
    > |
    > | Reusing my example from colds: Could you please provide a single
    > | reason why a DNS-server of some kind, running on a 32-bit
    > | architecture, should use 64-bit variables for its UDP communication
    > | calls?
    >
    > No. Why should I provide such an example? I've never suggested that it
    > is necessary to do this.
    >
    >
    > | NB: A reason, meaning "some advantage to be gained by doing so', not
    > | only 'I want to process my recorded television shows and don't give a
    > | ****", which is all I have read so far.
    >
    > How about a reason for asking for this?
    >
    > Or is it that you are not making the distinction between 64-bit variables
    > in general, and the few 64-bit variable data types needed to access file
    > positions and sizes when files might be (because the system supports it)
    > larger than 2GB or 4GB?


    I was making the erronous assumption that 'these few variable data
    types' would include 64-bit-I/O support as well (ie ssize_t and
    size_t) which they (at least according to the headers on the system I
    am currently using) don't.

  9. Re: large file support && ! large file support

    On Sun, 20 May 2007 18:56:27 +0200 Rainer Weikusat wrote:
    | phil-news-nospam@ipal.net writes:
    |> On Sun, 20 May 2007 16:01:43 +0200 Rainer Weikusat wrote:
    |> | phil-news-nospam@ipal.net writes:
    |> |
    |> | [...]
    |> |
    |> |> I see nothing wrong with having a "32-bit pointer / 64-bit file reference"
    |> |> type of architecture that provides compatibility for legacy programs with
    |> |> an extra legacy library that provides the ABI for older programs that expect
    |> |> "32-bit pointer / 32-bit file reference". They could also make an alternate
    |> |> toolchain to compile programs in the legacy architecture (for example the
    |> |> legacy library itself which can still need maintenance). But the default
    |> |> should be the forward architecture.
    |> |
    |> | Reusing my example from colds: Could you please provide a single
    |> | reason why a DNS-server of some kind, running on a 32-bit
    |> | architecture, should use 64-bit variables for its UDP communication
    |> | calls?
    |>
    |> No. Why should I provide such an example? I've never suggested that it
    |> is necessary to do this.
    |>
    |>
    |> | NB: A reason, meaning "some advantage to be gained by doing so', not
    |> | only 'I want to process my recorded television shows and don't give a
    |> | ****", which is all I have read so far.
    |>
    |> How about a reason for asking for this?
    |>
    |> Or is it that you are not making the distinction between 64-bit variables
    |> in general, and the few 64-bit variable data types needed to access file
    |> positions and sizes when files might be (because the system supports it)
    |> larger than 2GB or 4GB?
    |
    | I was making the erronous assumption that 'these few variable data
    | types' would include 64-bit-I/O support as well (ie ssize_t and
    | size_t) which they (at least according to the headers on the system I
    | am currently using) don't.

    It would be useful to have 64-bit size_t and ssize_t if it were useful to
    read in more than 4GB of a file at once. On a 32-bit architecture, that
    cannot be done (at least within the scope of normal I/O call interfaces).
    I suppose someone might find a cause to do something like that on a 64-bit
    architecture, and indeed, you do find size_t and ssize_t (and pointers and
    the like) that big (and part of the reason you do see a sudden jump up in
    the memory needed when moving from 32-bit to 64-bit architecture).

    What was all the purpose of having opaque data types that could be defined
    in a different size on another machine? Was it just as a means to document
    what a given variable was trying to be? Or was it a means to be able to
    recompile a well written program to another architecture in a language that
    was supposed to make this easier to do than say assembly language?

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-21-0751@ipal.net |
    |------------------------------------/-------------------------------------|

  10. Re: large file support && ! large file support

    phil-news-nospam wrote:

    > So I'm just trying to understand the specifics of the existant mess to see
    > how I might be able to accomplish this (exactly one way to access each
    > element of the interface, and yet have it always work no matter how the
    > calling programs are accessing the POSIX interface).


    You can't, at least not portably.

    As far as POSIX is concerned, ILP32_OFF32 and ILP32_OFFBIG are
    separate programming environments, just as ILP32_OFF32 and
    LP64_OFF64 are separate. You don't expect to be able to mix
    ILP32_OFF32 and LP64_OFF64 object files, so you shouldn't expect to
    be able to do it with ILP32_OFF32 and ILP32_OFFBIG either.

    The fact that the usual way that ILP32_OFFBIG is implemented is as
    a variation of ILP32_OFF32 with some identifiers translated "behind
    the scenes" is an internal implementation detail. There is no
    reason why a POSIX implementation that offers both programming
    environments couldn't have separate headers and libraries for each,
    and report errors if you try to mix them.

    --
    Geoff Clare

  11. Re: large file support && ! large file support

    On Mon, 21 May 2007 14:16:48 +0100 Geoff Clare wrote:
    | phil-news-nospam wrote:
    |
    |> So I'm just trying to understand the specifics of the existant mess to see
    |> how I might be able to accomplish this (exactly one way to access each
    |> element of the interface, and yet have it always work no matter how the
    |> calling programs are accessing the POSIX interface).
    |
    | You can't, at least not portably.
    |
    | As far as POSIX is concerned, ILP32_OFF32 and ILP32_OFFBIG are
    | separate programming environments, just as ILP32_OFF32 and
    | LP64_OFF64 are separate. You don't expect to be able to mix
    | ILP32_OFF32 and LP64_OFF64 object files, so you shouldn't expect to
    | be able to do it with ILP32_OFF32 and ILP32_OFFBIG either.
    |
    | The fact that the usual way that ILP32_OFFBIG is implemented is as
    | a variation of ILP32_OFF32 with some identifiers translated "behind
    | the scenes" is an internal implementation detail. There is no
    | reason why a POSIX implementation that offers both programming
    | environments couldn't have separate headers and libraries for each,
    | and report errors if you try to mix them.

    I don't have a problem with more than one programming environment. What
    I have a problem with is the mixed programming environment where some of
    the calls have added versions specifically for 64-bit file offsets. I do
    not want to have some of my functions have 64-bit versions. If my library
    is used in a pure 32-bit environment or pure 64-bit environment, even on
    the same machine, it should just work fine. The problem is when there is
    the mixed environment where a program could call either syscall, such as
    both stat() and stat64(). What I am thinking of doing in that case is to
    make my functions be only the 64-bit version, but with the normal name.
    The complication is that the headers will have to change under this case
    so the prototypes use the 64-bit variant names.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-21-1341@ipal.net |
    |------------------------------------/-------------------------------------|

  12. Re: large file support && ! large file support

    There's a lot here that will take time to write an adequate reply, and
    I doubt I'll find the time. Just a quick point:

    On 20 May 2007 13:47:01 GMT phil-news-nospam@ipal.net wrote:
    > At this point it seems the way I will do so is to have my API be always
    > of the "implicit large file" type when either implicit or explicit large
    > files are used by calling programs. A calling program using implicit
    > would not really see a difference.


    One thing I still don't understand is if your API is a shim or similar
    for the POSIX API, or if it's something completely different.

    If it's an equivalent/shim/etc, then having implicit large files is a
    mistake if your goal is to have existing programs (say, 'diff') use
    your library instead of or on top of POSIX. You would find that
    (e.g.) programs would happily open 2.1G files instead of failing, and
    then get lost as to their position in the file when they go past the
    2G mark. Now of course (for this specific example) whether or not
    this is a problem depends on the nature of the program, but in general
    you'd have to audit any program you want to link with your library.

    If a program has to be re-written or modified to use your API, it is
    of course a different story.

    ....
    > But I definitely do NOT want to see "64" variations in an API on a true
    > 64-bit machine for programs compiled to the full architecture. Yet with
    > the POSIX LFS way, we apparently will be stuck with that. It's not so
    > much a transition as it is a trap.


    POSIX LFS has a large legacy of programs to support! You may not have
    that defining handicap.

    -frank

  13. Re: large file support && ! large file support

    On Sat, 19 May 2007 13:15:32 +0000, phil-news-nospam wrote:

    > It seems a program cannot access BOTH the old (without large file
    > support) and new (with large file support) interfaces at the same time.
    > For example, when large file support is not turned on, type off_t is 32
    > bits and off64_t is not defined. But when LFS is turned on, off64_t
    > does get defined, and off_t is redefined as 64 bits.


    Doesn't _just_ having _LARGEFILE64_SOURCE=1 do the right thing here? All
    of the (informal) documentation I can find implies it does.

    > Is there a way to have both definitions active in their respective ways?
    >
    > The purpose is building a library which provides certain kinds of
    > wrapping around system or library calls that do deal with interface
    > changes due to large file support.


    Well I did the above for a library and it seems to work, although
    probably all of the users want LFS with full 64bits ... but I thought I
    tested it with off_t == 32bit as well, a little bit at least.

    --
    James Antill -- james@and.org
    http://www.and.org/and-httpd/ -- $2,000 security guarantee
    http://www.and.org/vstr/

  14. Re: large file support && ! large file support

    On Mon, 21 May 2007 14:43:44 -0700 Frank Cusack wrote:
    | There's a lot here that will take time to write an adequate reply, and
    | I doubt I'll find the time. Just a quick point:
    |
    | On 20 May 2007 13:47:01 GMT phil-news-nospam@ipal.net wrote:
    |> At this point it seems the way I will do so is to have my API be always
    |> of the "implicit large file" type when either implicit or explicit large
    |> files are used by calling programs. A calling program using implicit
    |> would not really see a difference.
    |
    | One thing I still don't understand is if your API is a shim or similar
    | for the POSIX API, or if it's something completely different.

    It's not exactly a shim. Some of the functions are simplistic changes
    to syscalls, such as a version of stat() that takes multiple string
    arguments and forms the path by inserting "/" between each argument
    and joining that all together as one string for the real syscall. It
    is obviously nor just a look-alike shim, as the API design is different.
    Another is a very different way to do file tree recursion than either
    the existing ftw or fts functions. It would be affected in terms of
    the struct stat it returns to callers for each object returned (unlike
    ftw it returns to the caller with each object).

    Examples:
    http://libh.slashusr.org/source/io/s...tat_join_sep.c
    http://libh.slashusr.org/source/ftr/...h/ftr_header.h

    See macros ftr_stat() and ftr_stat_ptr() in the latter URL. Those macros
    and many others like them provide info about the file last found by the
    ftr_get() function, for the given ftr object.


    | If it's an equivalent/shim/etc, then having implicit large files is a
    | mistake if your goal is to have existing programs (say, 'diff') use
    | your library instead of or on top of POSIX. You would find that
    | (e.g.) programs would happily open 2.1G files instead of failing, and
    | then get lost as to their position in the file when they go past the
    | 2G mark. Now of course (for this specific example) whether or not
    | this is a problem depends on the nature of the program, but in general
    | you'd have to audit any program you want to link with your library.

    It's not a shim in the sense of causing existing programs to effectively
    call my library, even if recompiled to do so (as in a macro shim). It
    is a different API that requires deliberate programming to use. A lot
    of it provides programming conveneience (useful for new programs, but
    not worth recoding old programs). A lot of it provides new features.


    | If a program has to be re-written or modified to use your API, it is
    | of course a different story.

    It does have to be re-written or modified. It is a different API. But
    it does make use of POSIX data types, structs, and functions (some even
    in macros that would be compiled at caller compile time).


    |> But I definitely do NOT want to see "64" variations in an API on a true
    |> 64-bit machine for programs compiled to the full architecture. Yet with
    |> the POSIX LFS way, we apparently will be stuck with that. It's not so
    |> much a transition as it is a trap.
    |
    | POSIX LFS has a large legacy of programs to support! You may not have
    | that defining handicap.

    No, I certainly do not have that legacy. But even if I did I would not
    take the approach POSIX LFS did. I would want to make it work for both
    old and new programs, even if that meant having 2 or even 3 different
    binary libraries (at least one being there for legacy ABI compatibility
    for programs w/o source code available). I think 2 would be sufficient,
    though. My objective is that none of my API have any "64" functions or
    data types of my own creation. Everything would work with large file
    PSOXI API elements, unless the calling programs don't use LFS. Then my
    library might need a legacy binary so that it can properly fail when the
    LFS functions would otherwise succeed, such as failing on a 5GB file being
    found in a tree recursion, which I believe should be the correct behaviour
    when the stat info being returned to the caller is from the legacy API.
    If it didn't fail in such a case, the information cannot be properly
    returned.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-21-1719@ipal.net |
    |------------------------------------/-------------------------------------|

  15. Re: large file support && ! large file support

    On Mon, 21 May 2007 22:18:30 -0000 James Antill wrote:
    | On Sat, 19 May 2007 13:15:32 +0000, phil-news-nospam wrote:
    |
    |> It seems a program cannot access BOTH the old (without large file
    |> support) and new (with large file support) interfaces at the same time.
    |> For example, when large file support is not turned on, type off_t is 32
    |> bits and off64_t is not defined. But when LFS is turned on, off64_t
    |> does get defined, and off_t is redefined as 64 bits.
    |
    | Doesn't _just_ having _LARGEFILE64_SOURCE=1 do the right thing here? All
    | of the (informal) documentation I can find implies it does.

    But that is not the only way a calling program might do this.


    |> Is there a way to have both definitions active in their respective ways?
    |>
    |> The purpose is building a library which provides certain kinds of
    |> wrapping around system or library calls that do deal with interface
    |> changes due to large file support.
    |
    | Well I did the above for a library and it seems to work, although
    | probably all of the users want LFS with full 64bits ... but I thought I
    | tested it with off_t == 32bit as well, a little bit at least.

    There are at least 8 different variations of doing LFS, based on what can
    be selected through CPP definitions. Apparently they fall into 3 major
    groups: no LFS, explicit LFS, implicit LFS. I think implicit LFS will
    be the easiest. But apparently a lot of programs have chosen to go with
    explicit LFS. My thinking at the moment is to just treat explicit the
    same as implicit, since there is very little legacy in usage of my library.
    That is, if you choose to have both stat() and stat64(), and also use my
    library, you've already committed to doing some amount of recoding, and
    committed to conding usage of my library, so you must make sure all usage
    of my library is consistent with the LFS support you are using. My macros
    will probably have to make use of the "64" variants of POSIX symbols in
    the explicit cases, but I think that won't be a problem.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-21-1743@ipal.net |
    |------------------------------------/-------------------------------------|

  16. Re: large file support && ! large file support

    On 21 May 2007 18:50:56 GMT
    phil-news-nospam@ipal.net wrote:

    > I don't have a problem with more than one programming environment.
    > What I have a problem with is the mixed programming environment where
    > some of the calls have added versions specifically for 64-bit file
    > offsets. I do not want to have some of my functions have 64-bit
    > versions.


    This was introduced to allow 32-bit systems to access large files. The
    transition from 32-bit systems via 32-bit systems with large file
    support, to 64-bit systems was handled with considerable deftness by
    the (then relevant) Unix vendors.

    The only time you would need a 32 and 64 bit version of a system call
    is when you're running a 32-bit machine with large file support.
    Most Unix systems -even though all are now 64-bit- continue to support
    32 bit programs, and 32-bit programs with large file support. This is a
    feature.

    If you restrict your library to a 64-bit version only, there is no need
    to worry about large file support.

    --
    Stefaan A Eeckels
    --
    He who will not reason, is a bigot;
    he who cannot is a fool;
    and he who dares not is a slave. (Sir William Drummond)

  17. Re: large file support && ! large file support

    On 21 May 2007 22:49:25 GMT
    phil-news-nospam@ipal.net wrote:

    > My macros will probably have to make use of the "64" variants of
    > POSIX symbols in the explicit cases,...


    Why? We're talking about a transitional interface here. The only time
    you need to use the lf64 functions is when a 32-bit program needs to
    access large files.

    There are valid reasons to continue to use 32-bit programs (on SPARC,
    they are marginally faster, the program needs to run on old 32-bit
    systems, etc), but there are not many compelling reasons to go out of
    one's way to support a transitional interface in new developments.

    Just go 64-bit, and be happy.

    --
    Stefaan A Eeckels
    --
    You know, it is almost always the case in the real world that something
    is "fair" when you like it and "unfair" when you don't.
    -- Jeffrey Siegal in gnu.misc.discuss

  18. Re: large file support && ! large file support

    On Mon, 21 May 2007 22:18:30 -0000 James Antill wrote:
    > On Sat, 19 May 2007 13:15:32 +0000, phil-news-nospam wrote:
    >
    >> It seems a program cannot access BOTH the old (without large file
    >> support) and new (with large file support) interfaces at the same time.
    >> For example, when large file support is not turned on, type off_t is 32
    >> bits and off64_t is not defined. But when LFS is turned on, off64_t
    >> does get defined, and off_t is redefined as 64 bits.

    >
    > Doesn't _just_ having _LARGEFILE64_SOURCE=1 do the right thing here? All
    > of the (informal) documentation I can find implies it does.


    For an application, yes. But the OP is trying to avoid having two
    interfaces (the LFS one and the non-LFS one) for each function, in a
    library. What _LARGEFILE64_SOURCE=1 (or more correctly, `getconf
    LFS_CFLAGS`) does is select the LFS interfaces for various functions.

    >> Is there a way to have both definitions active in their respective ways?
    >>
    >> The purpose is building a library which provides certain kinds of
    >> wrapping around system or library calls that do deal with interface
    >> changes due to large file support.

    >
    > Well I did the above for a library and it seems to work, although
    > probably all of the users want LFS with full 64bits ... but I thought I
    > tested it with off_t == 32bit as well, a little bit at least.


    There is no [good] way for a function to determine if its caller used
    an LFS or non-LFS datatype. Even if you want to do the nasty stuff to
    figure it out, LFS programs call different functions than non-LFS
    functions; e.g. open64 instead of open. So to work with existing
    programs and not require a recompile, a library must provide both
    interfaces. Even if you are adding some brand new interface, if you
    are using the types that are changed with LFS (off_t et al.) you
    must provide 2 interfaces to deal with the different sized args.

    libelf is a good example of a library that has LFS problems, but that
    isn't itself part of the LFS "standard". This is unfortunate, since
    implementors have been reluctant (or just didn't think of it) to
    "extend" the LFS interface to libelf, ie to define multiple versions
    of the libelf routines based on LFS or non-LFS settings. Therefore
    however you or your vendor compiles libelf is the only way your
    program that uses libelf can be compiled, regardless of the other
    (LFS) functionality you may want. Well, you could write a shim to
    translate the data types. Horrible.

    -frank

  19. Re: large file support && ! large file support

    On Tue, 22 May 2007 01:03:58 +0200 Stefaan A Eeckels wrote:
    | On 21 May 2007 18:50:56 GMT
    | phil-news-nospam@ipal.net wrote:
    |
    |> I don't have a problem with more than one programming environment.
    |> What I have a problem with is the mixed programming environment where
    |> some of the calls have added versions specifically for 64-bit file
    |> offsets. I do not want to have some of my functions have 64-bit
    |> versions.
    |
    | This was introduced to allow 32-bit systems to access large files. The
    | transition from 32-bit systems via 32-bit systems with large file
    | support, to 64-bit systems was handled with considerable deftness by
    | the (then relevant) Unix vendors.

    I don't thinks so. I think it was done the way they did it so they would
    not have to undo some misguided attempts to handle large files that had
    already be utilized.


    | The only time you would need a 32 and 64 bit version of a system call
    | is when you're running a 32-bit machine with large file support.
    | Most Unix systems -even though all are now 64-bit- continue to support
    | 32 bit programs, and 32-bit programs with large file support. This is a
    | feature.

    I cannot figure out when a program would need both the small file and large
    file calls in the very same program.


    | If you restrict your library to a 64-bit version only, there is no need
    | to worry about large file support.

    There is the need to work with those programs that select the explicit LFS
    mode, which means stat() is still for small files and stat64() is for large
    files (although some systems might only do one of them at the KABI).

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-21-1944@ipal.net |
    |------------------------------------/-------------------------------------|

  20. Re: large file support && ! large file support

    On Tue, 22 May 2007 01:23:37 +0200 Stefaan A Eeckels wrote:
    | On 21 May 2007 22:49:25 GMT
    | phil-news-nospam@ipal.net wrote:
    |
    |> My macros will probably have to make use of the "64" variants of
    |> POSIX symbols in the explicit cases,...
    |
    | Why? We're talking about a transitional interface here. The only time
    | you need to use the lf64 functions is when a 32-bit program needs to
    | access large files.

    And that seems to be widely used.


    | There are valid reasons to continue to use 32-bit programs (on SPARC,
    | they are marginally faster, the program needs to run on old 32-bit
    | systems, etc), but there are not many compelling reasons to go out of
    | one's way to support a transitional interface in new developments.
    |
    | Just go 64-bit, and be happy.

    When I'm writing a program, that concept works fine. But for a library
    one has to deal with how the prgorams calling it are written.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-21-1948@ipal.net |
    |------------------------------------/-------------------------------------|

+ Reply to Thread
Page 1 of 3 1 2 3 LastLast