bzip2 + large files ? - Aix

This is a discussion on bzip2 + large files ? - Aix ; group, I just came across a bz2-file that couldn't be decompressed because of "file too large". In compressed form it's just below 2GB, decompressed just above 2GB. (ulimit is OK, large file enabled JFS, AIX 5.2). I fetched the latest ...

+ Reply to Thread
Results 1 to 11 of 11

Thread: bzip2 + large files ?

  1. bzip2 + large files ?

    group,

    I just came across a bz2-file that couldn't be decompressed
    because of "file too large".
    In compressed form it's just below 2GB, decompressed just above 2GB.
    (ulimit is OK, large file enabled JFS, AIX 5.2).
    I fetched the latest bzip2 version (1.0.5), compiled, but needed
    cc -q64 and ar -X64 to obtain a binary which solved my problem.
    This does, however, not run on 32bit hardware (of course).

    Are those 64-bit flags mandatory to get large file support for
    user apps ?
    ISTR that large files were supported under AIX before the hardware and
    the OS went 64-bit ?


  2. Re: bzip2 + large files ?

    Michael Kraemer schrieb:
    > group,
    >
    > I just came across a bz2-file that couldn't be decompressed
    > because of "file too large".
    > In compressed form it's just below 2GB, decompressed just above 2GB.
    > (ulimit is OK, large file enabled JFS, AIX 5.2).
    > I fetched the latest bzip2 version (1.0.5), compiled, but needed
    > cc -q64 and ar -X64 to obtain a binary which solved my problem.
    > This does, however, not run on 32bit hardware (of course).
    >
    > Are those 64-bit flags mandatory to get large file support for
    > user apps ?
    > ISTR that large files were supported under AIX before the hardware and
    > the OS went 64-bit ?
    >

    32-bit programs can handle large files too.
    But they need to use open64, stat64...
    PS: I just had inutoc failing on a bff file >2GB.
    All needed to fix was adding 3 64 to enable it
    to work with large files.

  3. Re: bzip2 + large files ?

    Thomas Braunbeck schrieb:

    > 32-bit programs can handle large files too.
    > But they need to use open64, stat64...


    that's what I expected somehow,
    but it probably means I have to wade through bzip2's
    sources to look what's going on :-(




  4. Re: bzip2 + large files ?

    Michael Kraemer schrieb:
    > Thomas Braunbeck schrieb:
    >
    > that's what I expected somehow,
    > but it probably means I have to wade through bzip2's
    > sources to look what's going on :-(
    >


    yes. However in the bzip2 case I think it fails on
    a write() when the file gets >2GB. So you need to
    find the open() and change it to open64(). A grep
    over all source file to find all open() calls may
    do it. Changing them all into open64() should be
    no problem.

  5. Re: bzip2 + large files ?

    Michael Kraemer schrieb:

    > that's what I expected somehow,
    > but it probably means I have to wade through bzip2's
    > sources to look what's going on :-(
    >


    There is another option in case the problem is only
    open() which needs to be changed to open64().
    Build the 32bit version. Then dump -nv bzip2, look
    for the open (coming from libc.a) and note the function
    names listed before and after.
    Use a binary editor on the executable, find this open
    and change it to open64. You may want to work on a copy,
    then dump -nv and see if the open changed to open64.
    I once got a binary failing on files >2GB. The source
    was no longer available to correct the open() calls.
    I used od to locate the 'open' and dd to change it into
    open64. The fixed binary is still working, now can handle
    files >2GB.
    The includes files defining the a.out format are shipped
    with AIX and may give you an idea where in the binary the
    functions from the shared libs are located.

  6. Re: bzip2 + large files ?

    Thomas Braunbeck schrieb:

    > yes. However in the bzip2 case I think it fails on
    > a write() when the file gets >2GB. So you need to
    > find the open() and change it to open64(). A grep
    > over all source file to find all open() calls may
    > do it. Changing them all into open64() should be
    > no problem.


    That's the kind of work I tried to avoid in the first place.
    But if it's necessary to get a clean version ...


  7. Re: bzip2 + large files ?

    Thomas Braunbeck schrieb:
    > Michael Kraemer schrieb:
    >
    >> that's what I expected somehow,
    >> but it probably means I have to wade through bzip2's
    >> sources to look what's going on :-(
    >>

    >
    > There is another option in case the problem is only
    > open() which needs to be changed to open64().


    (snip)

    Appears like a hack to me.
    I think I would prefer a clean solution,
    probably other Unices (especially older ones)
    will give similar 32/64 bit issues.


  8. Re: bzip2 + large files ?

    On 2008-04-08, Michael Kraemer wrote:
    > I just came across a bz2-file that couldn't be decompressed
    > because of "file too large".


    Have you tried bzip2 -dc file ?

    --
    Darren Tucker (dtucker at zip.com.au)
    GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
    Good judgement comes with experience. Unfortunately, the experience
    usually comes from bad judgement.

  9. Re: bzip2 + large files ?

    Michael Kraemer wrote:
    > group,
    >
    > I just came across a bz2-file that couldn't be decompressed
    > because of "file too large".
    > In compressed form it's just below 2GB, decompressed just above 2GB.
    > (ulimit is OK, large file enabled JFS, AIX 5.2).
    > I fetched the latest bzip2 version (1.0.5), compiled, but needed
    > cc -q64 and ar -X64 to obtain a binary which solved my problem.
    > This does, however, not run on 32bit hardware (of course).
    >
    > Are those 64-bit flags mandatory to get large file support for
    > user apps ?
    > ISTR that large files were supported under AIX before the hardware and
    > the OS went 64-bit ?
    >



    The system header files define system calls and data structures
    according to the directive _LARGE_FILES.
    Defining this at compile time should allow your C programs to handle
    large files.

    Regards

  10. Re: bzip2 + large files ?

    On Apr 11, 5:48 am, Joachim Gann wrote:
    > Michael Kraemer wrote:
    > > group,

    >
    > > I just came across a bz2-file that couldn't be decompressed
    > > because of "file too large".
    > > In compressed form it's just below 2GB, decompressed just above 2GB.
    > > (ulimit is OK, large file enabled JFS, AIX 5.2).
    > > I fetched the latest bzip2 version (1.0.5), compiled, but needed
    > > cc -q64 and ar -X64 to obtain a binary which solved my problem.
    > > This does, however, not run on 32bit hardware (of course).

    >
    > > Are those 64-bit flags mandatory to get large file support for
    > > user apps ?
    > > ISTR that large files were supported under AIX before the hardware and
    > > the OS went 64-bit ?

    >
    > The system header files define system calls and data structures
    > according to the directive _LARGE_FILES.
    > Defining this at compile time should allow your C programs to handle
    > large files.
    >
    > Regards


    If you just want to get it to work, the cat the file through bunzip,
    i.e

    cat largefile.bz2 | bunzip2 > uncompressed_file

    Regards

  11. Re: bzip2 + large files ?

    bakaneko201@googlemail.com schrieb:

    >
    > If you just want to get it to work, the cat the file through bunzip,
    > i.e
    >
    > cat largefile.bz2 | bunzip2 > uncompressed_file


    Just for the records:
    all of the workarounds were functional,
    and I could also produce a large file enabled native version
    (http://bio.gsi.de/DOCS/SOFTWARE/DOWNLOAD/AIX/POWER/52/).

    thanks to all.


+ Reply to Thread