large file support && ! large file support - Unix

This is a discussion on large file support && ! large file support - Unix ; On 28 May 2007 20:44:46 GMT phil-news-nospam@ipal.net wrote: > On Mon, 28 May 2007 11:03:12 -0700 Frank Cusack wrote: > | On 28 May 2007 12:46:03 GMT phil-news-nospam@ipal.net wrote: > |> On Mon, 28 May 2007 00:00:50 -0700 Frank Cusack ...

+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3
Results 41 to 54 of 54

Thread: large file support && ! large file support

  1. Re: large file support && ! large file support

    On 28 May 2007 20:44:46 GMT phil-news-nospam@ipal.net wrote:
    > On Mon, 28 May 2007 11:03:12 -0700 Frank Cusack wrote:
    > | On 28 May 2007 12:46:03 GMT phil-news-nospam@ipal.net wrote:
    > |> On Mon, 28 May 2007 00:00:50 -0700 Frank Cusack wrote:
    > |> | Part of the reason it's difficult because the Linux folks haven't
    > |> | really learned of 'getconf LFS_CFLAGS'.
    > |>
    > |> ================================================== ==============
    > |> phil@varuna:/home/phil 916> getconf LFS_CFLAGS
    > |> -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
    > |> phil@varuna:/home/phil 917>
    > |> ================================================== ==============
    > |>
    > |> It seems to do something. But what is that telling me? I thought that
    > |> the macros that provide information from the system to the program had
    > |> different names, e.g. _LFS_LARGEFILE and _LFS64_LARGEFILE.
    > |
    > | Maybe you're thinking of sysconf()?
    >
    > No. I was referring to the macros _LFS_LARGEFILE and _LFS64_LARGEFILE as
    > defined in the POSIX LFS extensions.
    >
    > http://www.unix.org/version2/whatsnew/lfs20mar.html


    Ah, sure. But those aren't necessarily what you, the programmer, have
    to define. Like how _GNU_SOURCE defines other things,
    _LARGEFILE_SOURCE defines the right stuff for you. In fact, section
    3.3.1 and 3.3.2 seem to say that you (the programmer) should *not*
    define _LFS_LARGEFILE, rather unistd.h will define it for you.

    -frank

  2. Re: large file support && ! large file support

    On Mon, 28 May 2007 20:25:14 -0700 Frank Cusack wrote:
    | On 28 May 2007 20:44:46 GMT phil-news-nospam@ipal.net wrote:
    |> On Mon, 28 May 2007 11:03:12 -0700 Frank Cusack wrote:
    |> | On 28 May 2007 12:46:03 GMT phil-news-nospam@ipal.net wrote:
    |> |> On Mon, 28 May 2007 00:00:50 -0700 Frank Cusack wrote:
    |> |> | Part of the reason it's difficult because the Linux folks haven't
    |> |> | really learned of 'getconf LFS_CFLAGS'.
    |> |>
    |> |> ================================================== ==============
    |> |> phil@varuna:/home/phil 916> getconf LFS_CFLAGS
    |> |> -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
    |> |> phil@varuna:/home/phil 917>
    |> |> ================================================== ==============
    |> |>
    |> |> It seems to do something. But what is that telling me? I thought that
    |> |> the macros that provide information from the system to the program had
    |> |> different names, e.g. _LFS_LARGEFILE and _LFS64_LARGEFILE.
    |> |
    |> | Maybe you're thinking of sysconf()?
    |>
    |> No. I was referring to the macros _LFS_LARGEFILE and _LFS64_LARGEFILE as
    |> defined in the POSIX LFS extensions.
    |>
    |> http://www.unix.org/version2/whatsnew/lfs20mar.html
    |
    | Ah, sure. But those aren't necessarily what you, the programmer, have
    | to define. Like how _GNU_SOURCE defines other things,
    | _LARGEFILE_SOURCE defines the right stuff for you. In fact, section
    | 3.3.1 and 3.3.2 seem to say that you (the programmer) should *not*
    | define _LFS_LARGEFILE, rather unistd.h will define it for you.

    But my point is, _LARGEFILE_SOURCE, _LARGEFILE64_SOURCE, _FILE_OFFSET_BITS
    should be defined by the program source. Things like _LFS_LARGEFILE and
    _LFS64_LARGEFILE, get defined by the system to indicate what is, or could
    be, available. What I have not discovered, yet (too many things to try),
    is if _LFS_LARGEFILE and _LFS64_LARGEFILE are defined to say what _could_
    be requested, or defined to say what _has_ been requested.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-29-0353@ipal.net |
    |------------------------------------/-------------------------------------|

  3. Re: large file support && ! large file support

    phil-news-nospam wrote:

    > |> No. I was referring to the macros _LFS_LARGEFILE and _LFS64_LARGEFILE as
    > |> defined in the POSIX LFS extensions.
    > |>
    > |> http://www.unix.org/version2/whatsnew/lfs20mar.html


    I'm afraid you may be misunderstanding the status of the above
    document. It was a white paper which did two things:

    1. It proposed additions to SUS (the Single UNIX Specification) related
    to large file support.

    2. It proposed some transitional interfaces which implementors could
    provide to help convert existing programs to be large file capable.

    The transitional interfaces are not, and never have been, included in
    POSIX or SUS. I seem to recall that some implementors provided
    different transitional interfaces (lf_open() instead of open64()).

    The proposed additions to SUS were included in SUSv2 (possibly with
    minor modifications). SUSv2 has been superseded by SUSv3/POSIX.1-2001
    and some changes to the LFS-related interfaces have occurred.

    Thus the white paper is really of no use in writing current
    applications; it is purely of historical interest.

    > |
    > | Ah, sure. But those aren't necessarily what you, the programmer, have
    > | to define. Like how _GNU_SOURCE defines other things,
    > | _LARGEFILE_SOURCE defines the right stuff for you. In fact, section
    > | 3.3.1 and 3.3.2 seem to say that you (the programmer) should *not*
    > | define _LFS_LARGEFILE, rather unistd.h will define it for you.
    >
    > But my point is, _LARGEFILE_SOURCE, _LARGEFILE64_SOURCE, _FILE_OFFSET_BITS
    > should be defined by the program source. Things like _LFS_LARGEFILE and
    > _LFS64_LARGEFILE, get defined by the system to indicate what is, or could
    > be, available. What I have not discovered, yet (too many things to try),
    > is if _LFS_LARGEFILE and _LFS64_LARGEFILE are defined to say what _could_
    > be requested, or defined to say what _has_ been requested.


    None of these things are in POSIX or SUS. You should try to stick to
    the standard interfaces rather than using the obsolete, non-standard,
    transitional interfaces.

    The standard specifies two different programming environments for
    32-bit systems. One has a 32-bit off_t, the other has a 64-bit (or
    greater) off_t. You can use sysconf() or getconf to query which of
    the programming environments are supported, and confstr() or getconf
    to find out what compiler flags, linker flags, and libraries are
    needed for each environment.

    --
    Geoff Clare

  4. Re: large file support && ! large file support

    On Tue, 29 May 2007 14:07:09 +0100 Geoff Clare wrote:
    | phil-news-nospam wrote:
    |
    |> |> No. I was referring to the macros _LFS_LARGEFILE and _LFS64_LARGEFILE as
    |> |> defined in the POSIX LFS extensions.
    |> |>
    |> |> http://www.unix.org/version2/whatsnew/lfs20mar.html
    |
    | I'm afraid you may be misunderstanding the status of the above
    | document. It was a white paper which did two things:
    |
    | 1. It proposed additions to SUS (the Single UNIX Specification) related
    | to large file support.
    |
    | 2. It proposed some transitional interfaces which implementors could
    | provide to help convert existing programs to be large file capable.
    |
    | The transitional interfaces are not, and never have been, included in
    | POSIX or SUS. I seem to recall that some implementors provided
    | different transitional interfaces (lf_open() instead of open64()).
    |
    | The proposed additions to SUS were included in SUSv2 (possibly with
    | minor modifications). SUSv2 has been superseded by SUSv3/POSIX.1-2001
    | and some changes to the LFS-related interfaces have occurred.
    |
    | Thus the white paper is really of no use in writing current
    | applications; it is purely of historical interest.

    So where is a clear and correct document that tells specifically what does
    or does not happen with the specific macros defined or not?



    |> | Ah, sure. But those aren't necessarily what you, the programmer, have
    |> | to define. Like how _GNU_SOURCE defines other things,
    |> | _LARGEFILE_SOURCE defines the right stuff for you. In fact, section
    |> | 3.3.1 and 3.3.2 seem to say that you (the programmer) should *not*
    |> | define _LFS_LARGEFILE, rather unistd.h will define it for you.
    |>
    |> But my point is, _LARGEFILE_SOURCE, _LARGEFILE64_SOURCE, _FILE_OFFSET_BITS
    |> should be defined by the program source. Things like _LFS_LARGEFILE and
    |> _LFS64_LARGEFILE, get defined by the system to indicate what is, or could
    |> be, available. What I have not discovered, yet (too many things to try),
    |> is if _LFS_LARGEFILE and _LFS64_LARGEFILE are defined to say what _could_
    |> be requested, or defined to say what _has_ been requested.
    |
    | None of these things are in POSIX or SUS. You should try to stick to
    | the standard interfaces rather than using the obsolete, non-standard,
    | transitional interfaces.
    |
    | The standard specifies two different programming environments for
    | 32-bit systems. One has a 32-bit off_t, the other has a 64-bit (or
    | greater) off_t. You can use sysconf() or getconf to query which of
    | the programming environments are supported, and confstr() or getconf
    | to find out what compiler flags, linker flags, and libraries are
    | needed for each environment.

    What is the value of sysconf() or getconf for each of the cases:

    1. If only 32-bit is supported
    2. If only 64-bit is supported
    3. If both are supported

    For my macros that are part of my library, how should they test for which
    environment is being used? What if the calling program has selected a
    transitional environment?

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-29-2005@ipal.net |
    |------------------------------------/-------------------------------------|

  5. Re: large file support && ! large file support

    phil-news-nospam wrote:

    > |> |> http://www.unix.org/version2/whatsnew/lfs20mar.html
    > |
    > | I'm afraid you may be misunderstanding the status of the above
    > | document. It was a white paper which did two things:
    > |
    > | 1. It proposed additions to SUS (the Single UNIX Specification) related
    > | to large file support.
    > |
    > | 2. It proposed some transitional interfaces which implementors could
    > | provide to help convert existing programs to be large file capable.
    > |
    > | The transitional interfaces are not, and never have been, included in
    > | POSIX or SUS. I seem to recall that some implementors provided
    > | different transitional interfaces (lf_open() instead of open64()).
    > |
    > | The proposed additions to SUS were included in SUSv2 (possibly with
    > | minor modifications). SUSv2 has been superseded by SUSv3/POSIX.1-2001
    > | and some changes to the LFS-related interfaces have occurred.
    > |
    > | Thus the white paper is really of no use in writing current
    > | applications; it is purely of historical interest.
    >
    > So where is a clear and correct document that tells specifically what does
    > or does not happen with the specific macros defined or not?


    http://www.unix.org/version3/online.html

    > |> | Ah, sure. But those aren't necessarily what you, the programmer, have
    > |> | to define. Like how _GNU_SOURCE defines other things,
    > |> | _LARGEFILE_SOURCE defines the right stuff for you. In fact, section
    > |> | 3.3.1 and 3.3.2 seem to say that you (the programmer) should *not*
    > |> | define _LFS_LARGEFILE, rather unistd.h will define it for you.
    > |>
    > |> But my point is, _LARGEFILE_SOURCE, _LARGEFILE64_SOURCE, _FILE_OFFSET_BITS
    > |> should be defined by the program source. Things like _LFS_LARGEFILE and
    > |> _LFS64_LARGEFILE, get defined by the system to indicate what is, or could
    > |> be, available. What I have not discovered, yet (too many things to try),
    > |> is if _LFS_LARGEFILE and _LFS64_LARGEFILE are defined to say what _could_
    > |> be requested, or defined to say what _has_ been requested.
    > |
    > | None of these things are in POSIX or SUS. You should try to stick to
    > | the standard interfaces rather than using the obsolete, non-standard,
    > | transitional interfaces.
    > |
    > | The standard specifies two different programming environments for
    > | 32-bit systems. One has a 32-bit off_t, the other has a 64-bit (or
    > | greater) off_t. You can use sysconf() or getconf to query which of
    > | the programming environments are supported, and confstr() or getconf
    > | to find out what compiler flags, linker flags, and libraries are
    > | needed for each environment.
    >
    > What is the value of sysconf() or getconf for each of the cases:
    >
    > 1. If only 32-bit is supported
    > 2. If only 64-bit is supported
    > 3. If both are supported


    sysconf(_SC_V6_ILP32_OFF32) or getconf _POSIX_V6_ILP32_OFF32
    tells you if an "ILP32" (32-bit int, long and pointer) programming
    environment with 32-bit off_t is supported.

    sysconf(_SC_V6_ILP32_OFFBIG) or getconf _POSIX_V6_ILP32_OFFBIG
    tells you if an ILP32 programming environment with 64-bit (or
    greater) off_t is supported.

    Combining the two results will answer each of your three questions
    (unless you specifically wanted 64-bit off_t, not 64-bit or greater).

    > For my macros that are part of my library, how should they test for which
    > environment is being used?


    I don't know of any direct way, nor can I see the need for such a
    thing. Perhaps you need to look in a different way at the problem
    you are trying to solve.

    > What if the calling program has selected a
    > transitional environment?


    Non-standard environments are (naturally) not covered by the standard.

    --
    Geoff Clare

  6. Re: large file support && ! large file support

    On Wed, 30 May 2007 13:52:29 +0100 Geoff Clare wrote:

    |> So where is a clear and correct document that tells specifically what does
    |> or does not happen with the specific macros defined or not?
    |
    | http://www.unix.org/version3/online.html

    I'll give that a look later on.


    |> |> | Ah, sure. But those aren't necessarily what you, the programmer, have
    |> |> | to define. Like how _GNU_SOURCE defines other things,
    |> |> | _LARGEFILE_SOURCE defines the right stuff for you. In fact, section
    |> |> | 3.3.1 and 3.3.2 seem to say that you (the programmer) should *not*
    |> |> | define _LFS_LARGEFILE, rather unistd.h will define it for you.
    |> |>
    |> |> But my point is, _LARGEFILE_SOURCE, _LARGEFILE64_SOURCE, _FILE_OFFSET_BITS
    |> |> should be defined by the program source. Things like _LFS_LARGEFILE and
    |> |> _LFS64_LARGEFILE, get defined by the system to indicate what is, or could
    |> |> be, available. What I have not discovered, yet (too many things to try),
    |> |> is if _LFS_LARGEFILE and _LFS64_LARGEFILE are defined to say what _could_
    |> |> be requested, or defined to say what _has_ been requested.
    |> |
    |> | None of these things are in POSIX or SUS. You should try to stick to
    |> | the standard interfaces rather than using the obsolete, non-standard,
    |> | transitional interfaces.
    |> |
    |> | The standard specifies two different programming environments for
    |> | 32-bit systems. One has a 32-bit off_t, the other has a 64-bit (or
    |> | greater) off_t. You can use sysconf() or getconf to query which of
    |> | the programming environments are supported, and confstr() or getconf
    |> | to find out what compiler flags, linker flags, and libraries are
    |> | needed for each environment.
    |>
    |> What is the value of sysconf() or getconf for each of the cases:
    |>
    |> 1. If only 32-bit is supported
    |> 2. If only 64-bit is supported
    |> 3. If both are supported
    |
    | sysconf(_SC_V6_ILP32_OFF32) or getconf _POSIX_V6_ILP32_OFF32
    | tells you if an "ILP32" (32-bit int, long and pointer) programming
    | environment with 32-bit off_t is supported.
    |
    | sysconf(_SC_V6_ILP32_OFFBIG) or getconf _POSIX_V6_ILP32_OFFBIG
    | tells you if an ILP32 programming environment with 64-bit (or
    | greater) off_t is supported.
    |
    | Combining the two results will answer each of your three questions
    | (unless you specifically wanted 64-bit off_t, not 64-bit or greater).

    If off_t is greater, I don't think that will be an issue.

    But the problem is, I want to be able to do this test during the
    compile using CPP tests like #ifdef so that my macro definitions
    can be done correctly.


    |
    |> For my macros that are part of my library, how should they test for which
    |> environment is being used?
    |
    | I don't know of any direct way, nor can I see the need for such a
    | thing. Perhaps you need to look in a different way at the problem
    | you are trying to solve.

    The macros will need to know which functions to call within my own library
    to get the correct behaviour semantics. An example is the lstat() syscall,
    which in a 32-bit off_t environment will fail for a file which cannot be
    described in the corresponding struct stat.

    Otherwise, I would have to go back to the idea of having 2 different .so
    files (and probably 2 different .a files) for my library, one for 32-bit
    off_t and one for 64-bit off_t.


    |> What if the calling program has selected a
    |> transitional environment?
    |
    | Non-standard environments are (naturally) not covered by the standard.

    Nevertheless, it seems they are as prevalent as if it had been standardized.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-30-2010@ipal.net |
    |------------------------------------/-------------------------------------|

  7. Re: large file support && ! large file support

    phil-news-nospam wrote:

    > |> | The standard specifies two different programming environments for
    > |> | 32-bit systems. One has a 32-bit off_t, the other has a 64-bit (or
    > |> | greater) off_t. You can use sysconf() or getconf to query which of
    > |> | the programming environments are supported, and confstr() or getconf
    > |> | to find out what compiler flags, linker flags, and libraries are
    > |> | needed for each environment.
    > |>
    > |> What is the value of sysconf() or getconf for each of the cases:
    > |>
    > |> 1. If only 32-bit is supported
    > |> 2. If only 64-bit is supported
    > |> 3. If both are supported
    > |
    > | sysconf(_SC_V6_ILP32_OFF32) or getconf _POSIX_V6_ILP32_OFF32
    > | tells you if an "ILP32" (32-bit int, long and pointer) programming
    > | environment with 32-bit off_t is supported.
    > |
    > | sysconf(_SC_V6_ILP32_OFFBIG) or getconf _POSIX_V6_ILP32_OFFBIG
    > | tells you if an ILP32 programming environment with 64-bit (or
    > | greater) off_t is supported.
    > |
    > | Combining the two results will answer each of your three questions
    > | (unless you specifically wanted 64-bit off_t, not 64-bit or greater).
    >
    > If off_t is greater, I don't think that will be an issue.
    >
    > But the problem is, I want to be able to do this test during the
    > compile using CPP tests like #ifdef so that my macro definitions
    > can be done correctly.
    >
    >
    > |
    > |> For my macros that are part of my library, how should they test for which
    > |> environment is being used?
    > |
    > | I don't know of any direct way, nor can I see the need for such a
    > | thing. Perhaps you need to look in a different way at the problem
    > | you are trying to solve.
    >
    > The macros will need to know which functions to call within my own library
    > to get the correct behaviour semantics. An example is the lstat() syscall,
    > which in a 32-bit off_t environment will fail for a file which cannot be
    > described in the corresponding struct stat.
    >
    > Otherwise, I would have to go back to the idea of having 2 different .so
    > files (and probably 2 different .a files) for my library, one for 32-bit
    > off_t and one for 64-bit off_t.


    That brings us full circle to the point I made in my first post in
    this thread. The only fully portable solution (i.e. to all POSIX
    systems, both current and future) is to have two separate libraries.

    Mixing ILP32_OFF32 and ILP32_OFFBIG objects may work on (some) current
    systems, but as far as POSIX is concerned they are separate programming
    environments and implementations can refuse to allow you to mix them.

    --
    Geoff Clare

  8. Re: large file support && ! large file support

    On Thu, 31 May 2007 13:43:40 +0100 Geoff Clare wrote:

    | That brings us full circle to the point I made in my first post in
    | this thread. The only fully portable solution (i.e. to all POSIX
    | systems, both current and future) is to have two separate libraries.

    OK, so suppose I do that. How will you distinguish these libraries?

    One probem is there is no real mechanism for handling subarchitectures
    with different ABIs in a distinct way.


    | Mixing ILP32_OFF32 and ILP32_OFFBIG objects may work on (some) current
    | systems, but as far as POSIX is concerned they are separate programming
    | environments and implementations can refuse to allow you to mix them.

    What's important is the ability to match up binary executables compiled
    for each environment (subarchitecture) with the appropropriate library.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-05-31-2045@ipal.net |
    |------------------------------------/-------------------------------------|

  9. Re: large file support && ! large file support

    phil-news-nospam wrote:

    > | That brings us full circle to the point I made in my first post in
    > | this thread. The only fully portable solution (i.e. to all POSIX
    > | systems, both current and future) is to have two separate libraries.
    >
    > OK, so suppose I do that. How will you distinguish these libraries?


    That's entirely up to you. One common convention is to put them
    in separate directories and use -L to choose the directory, so
    that the library can still have its normal name in each case.
    (Of course for shared libraries you also need to set rpath in the
    executable so the right library version will be loaded at run time.)

    --
    Geoff Clare

  10. Re: large file support && ! large file support

    On Fri, 1 Jun 2007 13:28:39 +0100 Geoff Clare wrote:
    | phil-news-nospam wrote:
    |
    |> | That brings us full circle to the point I made in my first post in
    |> | this thread. The only fully portable solution (i.e. to all POSIX
    |> | systems, both current and future) is to have two separate libraries.
    |>
    |> OK, so suppose I do that. How will you distinguish these libraries?
    |
    | That's entirely up to you. One common convention is to put them
    | in separate directories and use -L to choose the directory, so
    | that the library can still have its normal name in each case.
    | (Of course for shared libraries you also need to set rpath in the
    | executable so the right library version will be loaded at run time.)

    One issue is that whatever is done to an executable, that executable must
    be able to run on any system that supports executables of that pointer
    size and that offset size, whether or not other sizes are supported. So,
    clearly, if an rapth setting is used for this, it would have to be a broad
    standard. This is yet another area that POSIX has stumbled on by not
    having established a uniform standard in the first place. I'm sure their
    execuse will be that POSIX is only about APIs, not about the underlying
    mechanisms that various systems use. But if POSIX doesn't want to take
    the lead on this, who should? LSB for Linux? Who for other Unix systems?

    Maybe we need to have a standard "ABI connectivity" naming scheme somewhat
    like C++ does for their object class interface linkages. But in this case
    it would be for libraries. It would need to address both pointer size and
    offset size (although I suspect there won't be any real life usage of 64
    bit pointer and 32 bit offset, although I would not be 100% sure). How
    about names like: libc.32.64.so (first is pointer size, second is offset
    size)?

    BTW, it's actually possible on the 32-bit x86 to use pointers up to 48 bits
    without needing any additional instructions from the 64-bit architecture.
    The catch is it innvolves some segment register juggling and corresponding
    instruction generation from the compiler. AFAIK, no toolchain or kernel
    has been made to support this hacked mode, so maybe we'll never need to
    deal with it. I don't know if any other 32-bit CPU has any such capability
    to go beyond 32 bits for pointers and address space.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-06-02-1137@ipal.net |
    |------------------------------------/-------------------------------------|

  11. Re: large file support && ! large file support

    On 2 Jun 2007 17:23:49 GMT phil-news-nospam@ipal.net wrote:
    > BTW, it's actually possible on the 32-bit x86 to use pointers up to 48 bits
    > without needing any additional instructions from the 64-bit architecture.
    > The catch is it innvolves some segment register juggling and corresponding
    > instruction generation from the compiler. AFAIK, no toolchain or kernel
    > has been made to support this hacked mode, so maybe we'll never need to
    > deal with it. I don't know if any other 32-bit CPU has any such capability
    > to go beyond 32 bits for pointers and address space.


    Applications can still only see 32 bits worth of address space, so this
    isn't useful for LFS (e.g. you can't mmap a very large file in).

    It's called PAE and is supported by Linux. I think also by Solaris,
    but I'm not 100% sure.

    -frank

  12. Re: large file support && ! large file support

    On Sat, 02 Jun 2007 15:51:43 -0700 Frank Cusack wrote:
    | On 2 Jun 2007 17:23:49 GMT phil-news-nospam@ipal.net wrote:
    |> BTW, it's actually possible on the 32-bit x86 to use pointers up to 48 bits
    |> without needing any additional instructions from the 64-bit architecture.
    |> The catch is it innvolves some segment register juggling and corresponding
    |> instruction generation from the compiler. AFAIK, no toolchain or kernel
    |> has been made to support this hacked mode, so maybe we'll never need to
    |> deal with it. I don't know if any other 32-bit CPU has any such capability
    |> to go beyond 32 bits for pointers and address space.
    |
    | Applications can still only see 32 bits worth of address space, so this
    | isn't useful for LFS (e.g. you can't mmap a very large file in).

    If your goal is to be able to randomly access different parts of a file
    without having to call mmap() more than once, then it certainly is an
    advantage to be accessing the large file from a 64-bit address space.
    But many of the advantages of using mmap() still work when mapping only
    a portion of a file at one time. Programs accessing large files in a
    32-bit address space would be restricted to that methd.


    | It's called PAE and is supported by Linux. I think also by Solaris,
    | but I'm not 100% sure.

    I was referring to using it at the process/VM layer. It could be done.
    AFAIK, this isn't implemented in Linux. I don't know about Solaris.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-06-03-1051@ipal.net |
    |------------------------------------/-------------------------------------|

  13. Re: large file support && ! large file support

    On 3 Jun 2007 15:57:36 GMT phil-news-nospam@ipal.net wrote:
    > | It's called PAE and is supported by Linux. I think also by Solaris,
    > | but I'm not 100% sure.
    >
    > I was referring to using it at the process/VM layer. It could be done.
    > AFAIK, this isn't implemented in Linux. I don't know about Solaris.


    How could it possibly be done at the process layer? The application
    expects a contigous memory layout.

    -frank

  14. Re: large file support && ! large file support

    On Sun, 03 Jun 2007 11:31:20 -0700 Frank Cusack wrote:
    | On 3 Jun 2007 15:57:36 GMT phil-news-nospam@ipal.net wrote:
    |> | It's called PAE and is supported by Linux. I think also by Solaris,
    |> | but I'm not 100% sure.
    |>
    |> I was referring to using it at the process/VM layer. It could be done.
    |> AFAIK, this isn't implemented in Linux. I don't know about Solaris.
    |
    | How could it possibly be done at the process layer? The application
    | expects a contigous memory layout.

    The process layer ABI would have to not expect a contiguous memory layout.
    It would be faked for the C programming layer by having the compiler for
    this kind of sub-architecture present the appearance of a flat layout by
    the use of code that loads 6 byte or 8 byte pointer types into appropriate
    segment and offset registers. It could allow applications to run in a VM
    that in theory could be as large as nearly 281474976710656 bytes, at the
    expense of being a bit slower due to all the extra instructions handling
    pointers in an odd way.

    One of the C compiler memory models for the 16-bit real mode 8086 allowed
    using a 32-bit pointer with values up to 1MB (or modulo 1MB) by splitting
    the pointer bits between a 16-bit segment and a 16-bit address. I wrote
    and ran programs with that memory model under MS-DOS.

    How can you have a 64-bit integer (long long) on a machine that only has
    instructions for integers up to 32 bits? You just use compiler tricks.
    There's no reason we can't have int128_t and uint128_t other than no one
    has modified the C compilers to do it (yet).

    Actually, we could build a completely fake 64-bit pointer size environment
    at the C programming level, complete with 64-bit long, 64-bit off_t, and
    many other things at the same size as in a real 64-bit machine, and generate
    the code strictly for a 32-bit machine. It would have limitations, such as
    the largest addressable memory, depending on the ABI layer sub-architecture
    choice involved. It wouldn't have a great deal of value, as it would waste
    memory for no real benefit, other than as a test realm for making sure code
    works correctly for the comping 64-bit machines. Now that the latter really
    are here, this benefit is no more. But it could have been done a long time
    ago if enough of the right people had thought out of the box.

    --
    |---------------------------------------/----------------------------------|
    | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
    | first name lower case at ipal.net / spamtrap-2007-06-03-1504@ipal.net |
    |------------------------------------/-------------------------------------|

+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3