gunzip and send to a program - Unix

This is a discussion on gunzip and send to a program - Unix ; Hi, I have some pretty big file compressed that I want to use for a program. The program works like this: ../variscan bigfile.phy config.conf Reading the firt parameter, bigfile.phy, and a config.conf file which is the second parameter. I would ...

+ Reply to Thread
Results 1 to 6 of 6

Thread: gunzip and send to a program

  1. gunzip and send to a program

    Hi,

    I have some pretty big file compressed that I want to use for a
    program. The program works like this:

    ../variscan bigfile.phy config.conf

    Reading the firt parameter, bigfile.phy, and a config.conf file which
    is the second parameter.

    I would like to use a compressed version of bigfile.phy,
    bigfile.phy.gz, and only expand it temporarily for variscan to read
    it, and throw it away once the program is finished.

    How can I do this? Can I use a pipe?

  2. Re: gunzip and send to a program

    Begin <5e7776a2-a030-4c84-a174-cb227ca0a48d@l64g2000hse.googlegroups.com>
    On Sat, 3 May 2008 09:36:32 -0700 (PDT), avilella wrote:
    > ./variscan bigfile.phy config.conf

    [snip]
    > I would like to use a compressed version of bigfile.phy,
    > bigfile.phy.gz, and only expand it temporarily for variscan to read
    > it, and throw it away once the program is finished.
    >
    > How can I do this? Can I use a pipe?


    See the documentation of the program you want to use (`man variscan');
    it should tell you whether you can make it accept input from stdin
    instead of as an argument. If yes, then you can use a pipe.


    --
    j p d (at) d s b (dot) t u d e l f t (dot) n l .
    This message was originally posted on Usenet in plain text.
    Any other representation, additions, or changes do not have my
    consent and may be a violation of international copyright law.

  3. Re: gunzip and send to a program

    On May 3, 5:45 pm, jpd wrote:
    > Begin <5e7776a2-a030-4c84-a174-cb227ca0a...@l64g2000hse.googlegroups.com>
    >
    > On Sat, 3 May 2008 09:36:32 -0700 (PDT), avilella wrote:
    > > ./variscan bigfile.phy config.conf

    > [snip]
    > > I would like to use a compressed version of bigfile.phy,
    > > bigfile.phy.gz, and only expand it temporarily for variscan to read
    > > it, and throw it away once the program is finished.

    >
    > > How can I do this? Can I use a pipe?

    >
    > See the documentation of the program you want to use (`man variscan');
    > it should tell you whether you can make it accept input from stdin
    > instead of as an argument. If yes, then you can use a pipe.


    no, it does not. I wrote the program and this option is not
    implemented. The program
    does not load the entire bigfile.phy, it reads chunks and throws away
    the analysed parts.
    So I am interested in an option that creates something that looks like
    a file but is simply
    the output of a "gunzip -c".

    >
    > --
    > j p d (at) d s b (dot) t u d e l f t (dot) n l .
    > This message was originally posted on Usenet in plain text.
    > Any other representation, additions, or changes do not have my
    > consent and may be a violation of international copyright law.



  4. Re: gunzip and send to a program

    Begin <42beaa64-33a4-486d-9159-13d00f8a5ba0@i76g2000hsf.googlegroups.com>
    On Sat, 3 May 2008 10:41:14 -0700 (PDT), avilella wrote:
    > On May 3, 5:45 pm, jpd wrote:
    >> See the documentation of the program you want to use (`man variscan');
    >> it should tell you whether you can make it accept input from stdin
    >> instead of as an argument. If yes, then you can use a pipe.

    >
    > no, it does not. I wrote the program and this option is not
    > implemented.


    Well, then fix your progam so that it does. If you wrote the program,
    bringing it in line with accepted practice for filter programs is the
    most elegant solution.


    > The program does not load the entire bigfile.phy, it reads chunks and
    > throws away the analysed parts. So I am interested in an option that
    > creates something that looks like a file but is simply the output of a
    > "gunzip -c".


    You could create a named pipe (using mkfifo(1)), then use that to
    connect the two parts of the pipe. Or you could give your program
    zlib support. Or you could do a number of other things.

    But with source available making an option to read from stdin (typically
    by giving just `-' as the file argument) is the way to go. It keeps the
    program and its use simpler and would allow to use bunzip2 or another
    streaming decompressor, or yet other things like netcat and some source
    on the network, or whatnot else, as needed.

    The only real argument you have to not support streaming through stdin
    and/or stdout is if you need to extensively seek on a file, in which
    case a named pipe won't help either, but you already noted that you
    don't have that excuse.


    --
    j p d (at) d s b (dot) t u d e l f t (dot) n l .
    This message was originally posted on Usenet in plain text.
    Any other representation, additions, or changes do not have my
    consent and may be a violation of international copyright law.

  5. Re: gunzip and send to a program

    In article ,
    jpd wrote:

    > Begin <42beaa64-33a4-486d-9159-13d00f8a5ba0@i76g2000hsf.googlegroups.com>
    > On Sat, 3 May 2008 10:41:14 -0700 (PDT), avilella wrote:
    > > On May 3, 5:45 pm, jpd wrote:
    > >> See the documentation of the program you want to use (`man variscan');
    > >> it should tell you whether you can make it accept input from stdin
    > >> instead of as an argument. If yes, then you can use a pipe.

    > >
    > > no, it does not. I wrote the program and this option is not
    > > implemented.

    >
    > Well, then fix your progam so that it does. If you wrote the program,
    > bringing it in line with accepted practice for filter programs is the
    > most elegant solution.
    >
    >
    > > The program does not load the entire bigfile.phy, it reads chunks and
    > > throws away the analysed parts. So I am interested in an option that
    > > creates something that looks like a file but is simply the output of a
    > > "gunzip -c".

    >
    > You could create a named pipe (using mkfifo(1)), then use that to
    > connect the two parts of the pipe. Or you could give your program
    > zlib support. Or you could do a number of other things.


    Some shells, including bash, have syntax that automates this:

    progname <(cmd line)

    This creates a temporary named pipe or uses /dev/fd to make the output
    of the command line appear to be a file.

    However, this will only work if the program reads sequentially from the
    file. If his program performs random access (using lseek() or mmap())
    to read the chunks, it will only work with a real file on disk.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  6. Re: gunzip and send to a program

    >
    > Some shells, including bash, have syntax that automates this:
    >
    > progname <(cmd line)
    >
    > This creates a temporary named pipe or uses /dev/fd to make the output
    > of the command line appear to be a file.


    Works like an absolute charm!

    ../variscan <(bunzip2 -c 21.1.46944323.phylip.bz2) 12.conf

    Thanks :-)

    >
    > However, this will only work if the program reads sequentially from the
    > file. If his program performs random access (using lseek() or mmap())
    > to read the chunks, it will only work with a real file on disk.
    >
    > --
    > Barry Margolin, bar...@alum.mit.edu
    > Arlington, MA
    > *** PLEASE post questions in newsgroups, not directly to me ***
    > *** PLEASE don't copy me on replies, I'll read them in the group ***



+ Reply to Thread