Begin <5e7776a2-a030-4c84-a174-cb227ca0a48d@l64g2000hse.googlegroups.com>
On Sat, 3 May 2008 09:36:32 -0700 (PDT), avilella wrote:
> ./variscan bigfile.phy config.conf
[snip]
> I would like to use a compressed version of bigfile.phy,
> bigfile.phy.gz, and only expand it temporarily for variscan to read
> it, and throw it away once the program is finished.
>
> How can I do this? Can I use a pipe?

See the documentation of the program you want to use (`man variscan');
it should tell you whether you can make it accept input from stdin
instead of as an argument. If yes, then you can use a pipe.


--
j p d (at) d s b (dot) t u d e l f t (dot) n l .
This message was originally posted on Usenet in plain text.
Any other representation, additions, or changes do not have my
consent and may be a violation of international copyright law.

On May 3, 5:45 pm, jpd wrote:
> Begin <5e7776a2-a030-4c84-a174-cb227ca0a...@l64g2000hse.googlegroups.com>
>
> On Sat, 3 May 2008 09:36:32 -0700 (PDT), avilella wrote:
> > ./variscan bigfile.phy config.conf
> [snip]
> > I would like to use a compressed version of bigfile.phy,
> > bigfile.phy.gz, and only expand it temporarily for variscan to read
> > it, and throw it away once the program is finished.
>
> > How can I do this? Can I use a pipe?
>
> See the documentation of the program you want to use (`man variscan');
> it should tell you whether you can make it accept input from stdin
> instead of as an argument. If yes, then you can use a pipe.

no, it does not. I wrote the program and this option is not
implemented. The program
does not load the entire bigfile.phy, it reads chunks and throws away
the analysed parts.
So I am interested in an option that creates something that looks like
a file but is simply
the output of a "gunzip -c".

>
> --
> j p d (at) d s b (dot) t u d e l f t (dot) n l .
> This message was originally posted on Usenet in plain text.
> Any other representation, additions, or changes do not have my
> consent and may be a violation of international copyright law.

Begin <42beaa64-33a4-486d-9159-13d00f8a5ba0@i76g2000hsf.googlegroups.com>
On Sat, 3 May 2008 10:41:14 -0700 (PDT), avilella wrote:
> On May 3, 5:45 pm, jpd wrote:
>> See the documentation of the program you want to use (`man variscan');
>> it should tell you whether you can make it accept input from stdin
>> instead of as an argument. If yes, then you can use a pipe.
>
> no, it does not. I wrote the program and this option is not
> implemented.

Well, then fix your progam so that it does. If you wrote the program,
bringing it in line with accepted practice for filter programs is the
most elegant solution.


> The program does not load the entire bigfile.phy, it reads chunks and
> throws away the analysed parts. So I am interested in an option that
> creates something that looks like a file but is simply the output of a
> "gunzip -c".

You could create a named pipe (using mkfifo(1)), then use that to
connect the two parts of the pipe. Or you could give your program
zlib support. Or you could do a number of other things.

But with source available making an option to read from stdin (typically
by giving just `-' as the file argument) is the way to go. It keeps the
program and its use simpler and would allow to use bunzip2 or another
streaming decompressor, or yet other things like netcat and some source
on the network, or whatnot else, as needed.

The only real argument you have to not support streaming through stdin
and/or stdout is if you need to extensively seek on a file, in which
case a named pipe won't help either, but you already noted that you
don't have that excuse.


--
j p d (at) d s b (dot) t u d e l f t (dot) n l .
This message was originally posted on Usenet in plain text.
Any other representation, additions, or changes do not have my
consent and may be a violation of international copyright law.

In article ,
jpd wrote:

> Begin <42beaa64-33a4-486d-9159-13d00f8a5ba0@i76g2000hsf.googlegroups.com>
> On Sat, 3 May 2008 10:41:14 -0700 (PDT), avilella wrote:
> > On May 3, 5:45 pm, jpd wrote:
> >> See the documentation of the program you want to use (`man variscan');
> >> it should tell you whether you can make it accept input from stdin
> >> instead of as an argument. If yes, then you can use a pipe.
> >
> > no, it does not. I wrote the program and this option is not
> > implemented.
>
> Well, then fix your progam so that it does. If you wrote the program,
> bringing it in line with accepted practice for filter programs is the
> most elegant solution.
>
>
> > The program does not load the entire bigfile.phy, it reads chunks and
> > throws away the analysed parts. So I am interested in an option that
> > creates something that looks like a file but is simply the output of a
> > "gunzip -c".
>
> You could create a named pipe (using mkfifo(1)), then use that to
> connect the two parts of the pipe. Or you could give your program
> zlib support. Or you could do a number of other things.

Some shells, including bash, have syntax that automates this:

progname <(cmd line)

This creates a temporary named pipe or uses /dev/fd to make the output
of the command line appear to be a file.

However, this will only work if the program reads sequentially from the
file. If his program performs random access (using lseek() or mmap())
to read the chunks, it will only work with a real file on disk.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

>
> Some shells, including bash, have syntax that automates this:
>
> progname <(cmd line)
>
> This creates a temporary named pipe or uses /dev/fd to make the output
> of the command line appear to be a file.

Works like an absolute charm!

../variscan <(bunzip2 -c 21.1.46944323.phylip.bz2) 12.conf

Thanks :-)

>
> However, this will only work if the program reads sequentially from the
> file. If his program performs random access (using lseek() or mmap())
> to read the chunks, it will only work with a real file on disk.
>
> --
> Barry Margolin, bar...@alum.mit.edu
> Arlington, MA
> *** PLEASE post questions in newsgroups, not directly to me ***
> *** PLEASE don't copy me on replies, I'll read them in the group ***