STREAMS pipes - ioctl I_SENDFD

How do I send a file descriptor over a pipe with ioctl I_SENDFD?

Reading up on "STREAMS", I see the pipe must be a "STREAMS" file
descriptor, but I'm not sure what a "stream" is in that context -
is a pipe one, or must one do some magic to turn it into one?

The following program works on Solaris, but on Linux, OSF1 and HP
the ioctls say "Invalid argument" or "Not a typewriter":

#include <assert.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stropts.h>
#include <unistd.h>

/* Run either without args, or the read end with arg 'r' and
* when that is waiting the write end with arg 'w'. */
int main(int argc, char **argv)
{
int pp[2], fd, r = 0, w = 0;
char *pipename = "/tmp/thepipe", *filename = "/tmp/thefile";
# define CK(expr) ((expr) ? (void)0 : (perror(#expr), exit(1)))
/* Open pipe descriptors */
if (argc == 1) {
CK(pipe(pp) == 0);
w = r = 1;
} else if (argv[1][0] == 'r') {
unlink(pipename);
CK(mkfifo(pipename, 0644) == 0);
CK((pp[0] = open(pipename, O_RDONLY)) >= 0);
r = 1;
} else {
CK((pp[1] = open(pipename, O_WRONLY)) >= 0);
w = 1;
}
/* Handle write end */
if (w) {
static const char hello[] = "Hello, world!\n";
unlink(filename);
CK((fd = open(filename, O_CREAT|O_RDWR, 0644)) >= 0);
unlink(filename);
CK(write(fd, hello, sizeof(hello) - 1) == (int) sizeof(hello) - 1);
CK(lseek(fd, 0, SEEK_SET) == 0);
CK(ioctl(pp[1], I_SENDFD, fd) != -1);
CK(close(fd) == 0);
CK(close(pp[1]) == 0);
}
/* Handle read end */
if (r) {
struct strrecvfd fdinfo;
static char buf[20];
CK(ioctl(pp[0], I_RECVFD, &fdinfo) != -1);
CK(close(pp[0]) == 0);
if (!w)
unlink(pipename);
CK(read(fdinfo.fd, buf, sizeof(buf) - 1) >= 0);
fputs(buf, stdout);
CK(close(fdinfo.fd) == 0);
}
return 0;
}

--
Hallvard

On Wed, 28 Mar 2007 21:30:38 +0200, Hallvard B Furuseth
<h.b.furuseth@usit.uio.no> wrote:
[color=blue]
>How do I send a file descriptor over a pipe with ioctl I_SENDFD?
>
>Reading up on "STREAMS", I see the pipe must be a "STREAMS" file
>descriptor, but I'm not sure what a "stream" is in that context -
>is a pipe one, or must one do some magic to turn it into one?
>
>The following program works on Solaris, but on Linux, OSF1 and HP
>the ioctls say "Invalid argument" or "Not a typewriter":[/color]

Linux doesn't have STREAMS, which are the System V way of doing this
task. BSD defined a different mechanism using sockets and the
SCM_RIGHTS message type with snedmsg() and recvmsg(), and this method
is more or less standardised in POSIX.1-2001.

Cheers,

Michael

Michael Kerrisk writes:[color=blue]
> Linux doesn't have STREAMS, which are the System V way of doing this
> task.[/color]

Huh? Then why does it #define I_SENDFD and I_RECVFD? And Linux 'man 2
ioctl' says
DESCRIPTION
The ioctl() function shall perform a variety of control functions
on STREAMS devices. For non-STREAMS devices, the functions
performed by this call are unspecified. (...)
HPUX and OSF1 have a large 'man 7 streamio' page.

Maybe it's not provided by default, but only as an add-on, but
documented anyway? Is there an #ifdef I can use to test if STREAMS are
provided? Or do I just try some STREAMS ioctl?
[color=blue]
> BSD defined a different mechanism using sockets and the
> SCM_RIGHTS message type with snedmsg() and recvmsg(), and this method
> is more or less standardised in POSIX.1-2001.[/color]

Thanks, but currently I hope to use STREAMS. They do have a number of
nice features.

BTW, I wonder about something. If I send the descriptors for both ends
of a pipe or socketpair into itself with I_SENDFD or SCM_RIGHTS and then
exit the program, what happens? Does the OS garbage collect the file
descriptions, or are there some OSes that just use reference counting so
they keep themselves open until system shutdown?

--
Hallvard

On Thu, 29 Mar 2007 10:41:26 +0200, Hallvard B Furuseth
<h.b.furuseth@usit.uio.no> wrote:
[color=blue]
>Michael Kerrisk writes:[color=green]
>> Linux doesn't have STREAMS, which are the System V way of doing this
>> task.[/color]
>
>Huh? Then why does it #define I_SENDFD and I_RECVFD?[/color]

I can't quite say. Perhaps because "one day, just in case the kernel
ever did...".
[color=blue]
> And Linux 'man 2
>ioctl' says
> DESCRIPTION
> The ioctl() function shall perform a variety of control functions
> on STREAMS devices. For non-STREAMS devices, the functions
> performed by this call are unspecified. (...)[/color]

Actually that looks like "man 3p ioctl". My Linux ioctl(2) page looks
rather different, and makes no mention of STREAMS.
[color=blue]
>HPUX and OSF1 have a large 'man 7 streamio' page.[/color]

Yes, they've got the System V heritage.
[color=blue]
>Maybe it's not provided by default, but only as an add-on, but
>documented anyway?[/color]

There has never been a mainline STREAMS implementation for Linux. in
2.4 there was an implementation that was developed but only as an
add-on.
[color=blue]
> Is there an #ifdef I can use to test if STREAMS are
>provided? Or do I just try some STREAMS ioctl?[/color]

Linux does not provide STREAMS, and likely never will. The Linux
kernel developers don't like them (because they find the idea
inefficient as I recall).

Cheers,

Michael

Michael Kerrisk <michael.kerrisk.at.gmx.net@nospam.com> writes:[color=blue]
>On Thu, 29 Mar 2007 10:41:26 +0200, Hallvard B Furuseth
><h.b.furuseth@usit.uio.no> wrote:[color=green]
>>Huh? Then why does it #define I_SENDFD and I_RECVFD?[/color]
> (...)
> I can't quite say. Perhaps because "one day, just in case the kernel
> ever did...".
>[color=green]
>> And Linux 'man 2
>>ioctl' says (...)[/color]
> Actually that looks like "man 3p ioctl".[/color]

Oops, right. I'd browsed a bit too many manpages on different systems.
[color=blue][color=green]
>>HPUX and OSF1 have a large 'man 7 streamio' page.[/color]
>
> Yes, they've got the System V heritage.[/color]

Urque. Looks rather common to rip out STREAMS _and_ fail to clean up,
then:-( Since both have the #defines and the manpage, I mean.
[color=blue]
>(...)[color=green]
>> Is there an #ifdef I can use to test if STREAMS are
>>provided? Or do I just try some STREAMS ioctl?[/color]
>
> Linux does not provide STREAMS, and likely never will. The Linux
> kernel developers don't like them (because they find the idea
> inefficient as I recall).[/color]

Well, I'm not looking for Linux in particular. Oh well, if only 1 out
of 4 OSes that I tested have them, I don't suppose it's much point in
making use of them. I hoped I was just missing some magic invocations.

Thanks for the help.

--
Regards,
Hallvard

>>>HPUX and OSF1 have a large 'man 7 streamio' page.[color=blue][color=green]
>>
>> Yes, they've got the System V heritage.[/color]
>
>Urque. Looks rather common to rip out STREAMS _and_ fail to clean up,
>then:-( Since both have the #defines and the manpage, I mean.[/color]


I didn't mean to imply that contemporary HP-UX and OSF1 (Tru64) don't
have STREAMS. Offhand, I'm not sure of the state of STREAMS on
contemporary versions of those systems - it may be that they have them
but you need some link magic or sth else.

Cheers,

Michael

Michael Kerrisk <michael.kerrisk.at.gmx.net@nospam.com> wrote:[color=blue][color=green][color=darkred]
> >>>HPUX and OSF1 have a large 'man 7 streamio' page.
> >> Yes, they've got the System V heritage.[/color][/color][/color]

HP-UX also has BSD heritage. For example, up through and including
HP-UX 10.20 the TCP/IP stack was a variation on BSD.
[color=blue]
> I didn't mean to imply that contemporary HP-UX and OSF1 (Tru64)
> don't have STREAMS. Offhand, I'm not sure of the state of STREAMS
> on contemporary versions of those systems - it may be that they have
> them but you need some link magic or sth else.[/color]

Support for STREAMS (Streams really as I don't think it is an acryonym
:) was first available as an add-on to HP-UX 9.mumble and was included
in the core OS starting with 10.20. The TCP/IP stack became
"Streams-based" in HP-UX 11.0 and remains that way today.

rick jones
--
denial, anger, bargaining, depression, acceptance, rebirth...
where do you want to be today?
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

>> I didn't mean to imply that contemporary HP-UX and OSF1 (Tru64)[color=blue][color=green]
>> don't have STREAMS. Offhand, I'm not sure of the state of STREAMS
>> on contemporary versions of those systems - it may be that they have
>> them but you need some link magic or sth else.[/color]
>
>Support for STREAMS (Streams really as I don't think it is an acryonym[/color]

I though common usage was CAPS to distinguish from stdio streams?
It's what I recall seeing in various places.

Cheers,

Michael

Michael Kerrisk <michael.kerrisk.at.gmx.net@nospam.com> wrote:
[color=blue]
> Linux does not provide STREAMS, and likely never will. The Linux
> kernel developers don't like them (because they find the idea
> inefficient as I recall).[/color]

There is a STREAMS implementation for Linux. It's part of the
OpenSS7 project - [url]http://www.openss7.org/[/url]

Maybe it will never be included in the mainstream kernel, but
AFAIK there's no reason why a Linux distro couldn't include it.

--
Geoff Clare <netnews@gclare.org.uk>

Rick Jones writes:[color=blue]
> Support for STREAMS (Streams really as I don't think it is an acryonym
> :) was first available as an add-on to HP-UX 9.mumble and was included
> in the core OS starting with 10.20. The TCP/IP stack became
> "Streams-based" in HP-UX 11.0 and remains that way today.[/color]

The HPUP box I'm using is more recent than that. uname -a says
HP-UX hagrid B.11.11 U 9000/782 2002917847 unlimited-user license
yet the ioctls in my program still say "Not a typewriter".
Anyway, I'm taking this to comp.sys.hp.hpux to see if someone else
there knows, since nobody spotted a problem with the program itself.

(BTW, I don't know why STREAMS is spelled in uppercase, but the manpages
I've seen still do. Though some also refer to lowercase "streams" for
the individual descriptors. I guess STREAMS is the module name or
something.)

--
Regards,
Hallvard

Hallvard B Furuseth <h.b.furuseth@usit.uio.no> writes:[color=blue]
> (BTW, I don't know why STREAMS is spelled in uppercase, but the manpages
> I've seen still do. Though some also refer to lowercase "streams" for
> the individual descriptors. I guess STREAMS is the module name or
> something.)[/color]

No; it's part of the kernel itself, not a module.

It's just System V heritage. STREAMS shouted is not the same as
regular streams, in much the same way UNIX(tm) isn't the same as other
spellings. ;-}

So, for common usage:

"The STREAMS framework has functions to allocate memory for
..."

but:

"When I open /dev/foobar, a self-cloning node, I get a stream
with a distinct minor number, and ..."

It doesn't have to make sense.

--
James Carlson, Solaris Networking <james.d.carlson@sun.com>
Sun Microsystems / 1 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677

Hallvard B Furuseth <h.b.furuseth@usit.uio.no> wrote:[color=blue]
> The HPUP box I'm using is more recent than that. uname -a says
> HP-UX hagrid B.11.11 U 9000/782 2002917847 unlimited-user license
> yet the ioctls in my program still say "Not a typewriter".
> Anyway, I'm taking this to comp.sys.hp.hpux to see if someone else
> there knows, since nobody spotted a problem with the program itself.[/color]

Mild bitrot in the wetware dimm memory led to my not remembering that
while the TCP/IP stack is Streams based, "pipes" are not necessarily
Streams-based by default on 11.X - you have to tweak a kernel
parameter (the name of which escapes me) to have pipes be Streams.


--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Hallvard B Furuseth <h.b.furuseth@usit.uio.no> wrote:[color=blue]
> Michael Kerrisk writes:[color=green]
>> Linux doesn't have STREAMS, which are the System V way of doing this
>> task.[/color][/color]

See [url]http://www.openss7.org/[/url] for the Linux Fast-STREAMS implementation.
[color=blue]
> HPUX and OSF1 have a large 'man 7 streamio' page.[/color]

[url]http://www.openss7.org/man2html?streamio.7[/url]
[color=blue]
> documented anyway? Is there an #ifdef I can use to test if STREAMS are
> provided? Or do I just try some STREAMS ioctl?[/color]

_XOPEN_STREAMS and sysconf(8) variable _SC_XOPEN_STREAMS (unfortunately
syconf always returns false for Linux).

To check a particular file descriptor for whether it corresponds to a
STREAMS character special file, use isastream(2):

[url]http://www.openss7.org/man2html?isastream.2[/url]

isastream(2) can be used to distinguish an old SVR3-style unidirectional
pipe from a SVR4+ STREAMS-based full duplex pipe. It can also be used
to distinguish between an old VIII Edition or BSD-style terminal or
pseudo terminal and a modern SVR4+ STREAMS-based terminal or
pseudo-terminal.

isastream() is also supported by glibc, but that version does not know
about STREAMS and will always return false. (Linux mainline only
supports antiquated terminals and pipes, and has only crippled BSD-style
pseudo-terminals.)
[color=blue]
> Thanks, but currently I hope to use STREAMS. They do have a number of
> nice features.[/color]

Agreed. Without STREAMS, Linux is just another BSD (and probably not a
very good one).
[color=blue]
> BTW, I wonder about something. If I send the descriptors for both ends
> of a pipe or socketpair into itself with I_SENDFD or SCM_RIGHTS and then
> exit the program, what happens? Does the OS garbage collect the file
> descriptions, or are there some OSes that just use reference counting so
> they keep themselves open until system shutdown?[/color]

In fact what is sent using I_SENDFD is not the fd, but the file pointer.
The fd is simply used to find the file pointer and the file pointer is
passed in the M_PASSFP STREAMS message:

[url]http://www.openss7.org/man2html?M_PASSFP.9[/url]

The Linux Fast-STREAMS implementation uses an esballoc(9)'ed message
block with a free function that releases a reference on the file
pointer.

[url]http://www.openss7.org/man2html?esballoc.9[/url]

When a process calls I_SENDFD and then exits, the file descriptor is
released but not the file pointer which is held by the M_PASSFP message
(the file remains open). If the pipe is dismantled as a result of the
close, the M_PASSFP message will be flushed resulting in the release
(and possible close) of the associated file pointer.

--brian

--
Brian F. G. Bidulock
[email]bidulock@openss7.org[/email]
[url]http://www.openss7.org/[/url]

Michael Kerrisk <michael.kerrisk.at.gmx.net@nospam.com> wrote:[color=blue][color=green]
>>Huh? Then why does it #define I_SENDFD and I_RECVFD?[/color][/color]
[color=blue]
> I can't quite say. Perhaps because "one day, just in case the kernel
> ever did...".[/color]

I_SENDFD and I_RECVFD are defined in stropts.h and are necessary for
SVID SUSv1 SUSv2 and SUSv3 compliance. Their presence does not mean
that any particular file descriptor need support them. And Linux
doesn't.
[color=blue]
> Linux does not provide STREAMS, and likely never will. The Linux
> kernel developers don't like them (because they find the idea
> inefficient as I recall).[/color]

Perhaps you mean Alan Cox's FUD claims about poor STREAMS performance.

With particular regard to STREAMS-based pipes, Linux Fast-STREAMS
STREAMS-based pipes have always performed as good as the antiquated
SVR3-style pipes Linux provides:

[url]http://www.openss7.org/streams_perf.html[/url]

More recent performance test show that STREAMS-based pipes outperform
Linux SVR3-style pipes across a wide range of architectures and systems.

Also, more recent performance tests show that STREAMS-based TPI internet
components (UDP, SCTP) outperform Linux Native NET4 Sockets
implementations using netperf on loopback (thanks Rick). For the most
recent kernels, STREAMS passed 40% more throughput than Sockets across
the full range of packet sizes from 1 through 16368. I will post these
results on our website in several days time.

Any Linux kernel hacker that wants to make statements about STREAMS
performance had better fix their pipes, sockets and pseudo-terminals,
as ultimately it is Linux performance that has now proved to be dismal
in same-machine-same-kernel comparison tests against STREAMS.

--brian

--
Brian F. G. Bidulock
[email]bidulock@openss7.org[/email]
[url]http://www.openss7.org/[/url]

Hallvard B Furuseth <h.b.furuseth@usit.uio.no> wrote:[color=blue]
> Well, I'm not looking for Linux in particular. Oh well, if only 1 out
> of 4 OSes that I tested have them, I don't suppose it's much point in
> making use of them. I hoped I was just missing some magic invocations.[/color]

They all support STREAMS-based pipes in one form or another. Yes,
there is magic to invoking them. Some provide an s_pipe() library
call. For AIX and UnixWare you open /dev/spx and used I_FDINSERT.
(See, for example, [url]http://www.openss7.org/man2html?spx.4[/url] ). Also,
see Stevens Chapter 15 (Advanced Programming in the UNIX Environment
ISBN 0-201-56371-7).

--brian

--
Brian F. G. Bidulock
[email]bidulock@openss7.org[/email]
[url]http://www.openss7.org/[/url]

Thanks for all the info. One detail:

[email]bidulock@openss7.org[/email] writes:[color=blue]
>Hallvard B Furuseth <h.b.furuseth@usit.uio.no> wrote:[color=green]
>> BTW, I wonder about something. If I send the descriptors for both ends
>> of a pipe or socketpair into itself with I_SENDFD or SCM_RIGHTS and then
>> exit the program, what happens? Does the OS garbage collect the file
>> descriptions, or are there some OSes that just use reference counting so
>> they keep themselves open until system shutdown?[/color]
>
> In fact what is sent using I_SENDFD is not the fd, but the file pointer.
> The fd is simply used to find the file pointer and the file pointer is
> passed in the M_PASSFP STREAMS message:[/color]

Well, yes. I tried to use what's apparently the Posix terminology
(which I'm admittedly not clear on) - the integer descript_or_ refers to
the internal descript_ion_.
[color=blue]
> [url]http://www.openss7.org/man2html?M_PASSFP.9[/url]
>
> The Linux Fast-STREAMS implementation uses an esballoc(9)'ed message
> block with a free function that releases a reference on the file
> pointer.
>
> [url]http://www.openss7.org/man2html?esballoc.9[/url][/color]

Doesn't tell me much, I'm afraid. And in this case I'm anyway more
interested in if someone does it wrong than if someone does it right:-)
[color=blue]
> When a process calls I_SENDFD and then exits, the file descriptor is
> released but not the file pointer which is held by the M_PASSFP message
> (the file remains open). If the pipe is dismantled as a result of the
> close, the M_PASSFP message will be flushed resulting in the release
> (and possible close) of the associated file pointer.[/color]

Yes, _if_ it is dismantled due to close(). But close() doesn't
dismantle it if it is held open by something else - e.g. due to dup(),
fork() or I_SENDFD. So if a pipe or socketpair is holding _itself_
open, I assume garbage collection will catch it, reference counting will
not. Unless some additional magic is invoked.

--
Hallvard

[email]bidulock@openss7.org[/email] wrote:[color=blue]
> Also, more recent performance tests show that STREAMS-based TPI
> internet components (UDP, SCTP) outperform Linux Native NET4 Sockets
> implementations using netperf on loopback (thanks Rick). For the
> most recent kernels, STREAMS passed 40% more throughput than Sockets
> across the full range of packet sizes from 1 through 16368. I will
> post these results on our website in several days time.[/color]

I do not in any way wish to minimize loopback performance, but will
say that I am a bit more comfortable with claims about loopback
performance if they are backed-up with similar data showing
performance over a real link.

rick jones

I guess this means I shouldn't drop the XTI suite from netperf2 and I
should be asking folks to port something similar to netperf4 :)

--
The computing industry isn't as much a game of "Follow The Leader" as
it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
- Rick Jones
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Hallvard B Furuseth <h.b.furuseth@usit.uio.no> wrote:[color=blue]
> Yes, _if_ it is dismantled due to close(). But close() doesn't
> dismantle it if it is held open by something else - e.g. due to dup(),
> fork() or I_SENDFD. So if a pipe or socketpair is holding _itself_
> open, I assume garbage collection will catch it, reference counting will
> not. Unless some additional magic is invoked.[/color]

There is a problem with passing a file pointer that is associated with the
Stream head to Which it is passed. The problem is that if the M_PASSFP
message containing a file pointer to the Stream head sits on that Stream
head's read queue when the last file descriptor is closed, that Stream head
will never close. Therefore, if the file pointer passed is associated with
the Stream head to which it is passing the file pointer, I_SENDFD fails and
returns EINVAL. In general it is not very useful to pass a Stream head its
own file pointer.

--brian

--
Brian F. G. Bidulock
[email]bidulock@openss7.org[/email]
[url]http://www.openss7.org/[/url]

Rick Jones <rick.jones2@hp.com> wrote:[color=blue]
> I do not in any way wish to minimize loopback performance, but will
> say that I am a bit more comfortable with claims about loopback
> performance if they are backed-up with similar data showing
> performance over a real link.[/color]

I don't own a NIC that will pass 12 Gigibits per Second.

--brian

--
Brian F. G. Bidulock
[email]bidulock@openss7.org[/email]
[url]http://www.openss7.org/[/url]

[email]bidulock@openss7.org[/email] wrote:[color=blue]
> Rick Jones <rick.jones2@hp.com> wrote:[color=green]
> > I do not in any way wish to minimize loopback performance, but will
> > say that I am a bit more comfortable with claims about loopback
> > performance if they are backed-up with similar data showing
> > performance over a real link.[/color][/color]
[color=blue]
> I don't own a NIC that will pass 12 Gigibits per Second.[/color]

But that is why netperf can present service demand - so one can
compare the efficiency of two different stacks running on the same
hardware, even if doesn't have the hardware to 12 gigabits per second
:)

rick jones
--
portable adj, code that compiles under more than one compiler
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...