Read/Write behind 2G - OS2

This is a discussion on Read/Write behind 2G - OS2 ; Is it possible to a) Query a file descriptor state w.r.t. possibility to read/write behind 2G? b) Change this state (ReOpen as L)? Thanks, Ilya P.S. In the toolkit headers I found #define OPEN_SHARE_DENYLEGACY 0x10000000 /* 2GB */ Do not ...

+ Reply to Thread
Results 1 to 14 of 14

Thread: Read/Write behind 2G

  1. Read/Write behind 2G


    Is it possible to

    a) Query a file descriptor state w.r.t. possibility to read/write
    behind 2G?

    b) Change this state (ReOpen as L)?

    Thanks,
    Ilya

    P.S. In the toolkit headers I found

    #define OPEN_SHARE_DENYLEGACY 0x10000000 /* 2GB */

    Do not know what it means, probably something else...

  2. Re: Read/Write behind 2G

    On Thu, 3 Feb 2005 01:06:55 +0000 (UTC), Ilya Zakharevich
    wrote:

    > Is it possible to
    >
    > a) Query a file descriptor state w.r.t. possibility to read/write
    > behind 2G?


    Not that I'm aware of.

    > b) Change this state (ReOpen as L)?


    Not without closing and reopening.

    > P.S. In the toolkit headers I found
    >
    > #define OPEN_SHARE_DENYLEGACY 0x10000000 /* 2GB */
    >
    > Do not know what it means, probably something else...


    It means that if something has used DosOpenL with that flag, then something
    using DosOpen will fail. If you don't use that flag, then both will succeed
    (subject to other normal mode and sharing restrictions).
    If something has already DosOpen'd a file then you will not be able to
    write beyond 2GB even when using DosOpenL.

  3. Re: Read/Write behind 2G

    On Thu, 3 Feb 2005 01:06:55 UTC, Ilya Zakharevich
    wrote:

    > a) Query a file descriptor state w.r.t. possibility to read/write
    > behind 2G?


    What I do is DosSetFilePtrL() to 0x80000000. If this fails,
    large file support is not available. Of course, the file
    position before this operatiom has the be saved an restored.


    > b) Change this state (ReOpen as L)?


    I don't think so.


    --
    Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
    http://www.s-t.de
    Please remove all characters left of the "R" in my email address


  4. Re: Read/Write behind 2G

    [A complimentary Cc of this posting was sent to
    Ruediger Ihle
    ], who wrote in article :
    > On Thu, 3 Feb 2005 01:06:55 UTC, Ilya Zakharevich
    > wrote:
    >
    > > a) Query a file descriptor state w.r.t. possibility to read/write
    > > behind 2G?

    >
    > What I do is DosSetFilePtrL() to 0x80000000. If this fails,
    > large file support is not available. Of course, the file
    > position before this operatiom has the be saved an restored.


    There are so many possibilities why this call may fail; how do you
    distinguish them?

    Thanks,
    Ilya

  5. Re: Read/Write behind 2G

    On Wed, 9 Feb 2005 22:06:09 UTC, Ilya Zakharevich
    wrote:

    > There are so many possibilities why this call may fail;


    Which are (besides an invalid handle an a corrupted file
    system) ?


    > how do you distinguish them?


    Currently, I don't. I have assured, that the handle references
    a disk file and not a pipe). So if this call fails for another
    than the assumed reason, there is much more wrong with the
    handle passed in or with the system as a whole. In this case,
    it is really not important if the (unaccessable) file might be
    smaller or larger than 2G.


    --
    Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
    http://www.s-t.de
    Please remove all characters left of the "R" in my email address


  6. Re: Read/Write behind 2G

    [A complimentary Cc of this posting was sent to
    Ruediger Ihle
    ], who wrote in article :
    > On Wed, 9 Feb 2005 22:06:09 UTC, Ilya Zakharevich
    > wrote:
    > > There are so many possibilities why this call may fail;


    > Which are (besides an invalid handle an a corrupted file
    > system) ?


    Short file, pipe, filesystem which can seek only backwards (or only in
    small chunks)... These are just first things which come to mind...

    > it is really not important if the (unaccessable) file might be
    > smaller or larger than 2G.


    Suppose I do not care about the size of the file (if it is a file; all
    I have is a filehandle). What I care is that the program "will work
    with it". But if there is no way to flip the flag, all these
    considerations are, of course, moot...

    Thanks,
    Ilya

  7. Re: Read/Write behind 2G

    On Thu, 10 Feb 2005 20:24:39 UTC, Ilya Zakharevich
    wrote:

    > Short file,


    This is exactly what I want to find out. If the call
    fails the file cannot grow above 2G. This means that
    either it has not been opened with DosOpenL() or the
    underlying file system does not support large files.

    > pipe,


    Can be checked using DosQueryHType(). In my case, I
    don't do this because since I'm the one who opens the
    file, I'm sure that it's not a pipe.

    > filesystem which can seek only backwards (or only in
    > small chunks)...


    I have never heard of any filesystem on OS/2 that does
    this. Are you thinking about a tape file system ? If
    such a file system would really exists, how would one
    access it in a generic way i.e without knowing about
    these specialities ? There is no OS/2 error like: "I
    cannot seek forward, please do a dummy read" or "The
    distance it to big, please try a smaller one". If a
    device has such limitations, it is the job of the file
    system / device driver to hide them from the rest of
    the system.

    > But if there is no way to flip the flag, all these
    > considerations are, of course, moot...


    Depends on what you are trying to do. I use this function
    to determine if the output file I am about to create can
    grow above 2G. If not, my program will switch to a split
    file approach. Of course, I'm not using handles inherited
    from some other process...


    --
    Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
    http://www.s-t.de
    Please remove all characters left of the "R" in my email address


  8. Re: Read/Write behind 2G

    [A complimentary Cc of this posting was sent to
    Ruediger Ihle
    ], who wrote in article :
    > On Thu, 10 Feb 2005 20:24:39 UTC, Ilya Zakharevich
    > wrote:
    >
    > > Short file,

    >
    > This is exactly what I want to find out. If the call
    > fails the file cannot grow above 2G.


    Eh? I do not see it documented what happens if the position is after
    EOF. What your docs say?

    > > filesystem which can seek only backwards (or only in
    > > small chunks)...

    >
    > I have never heard of any filesystem on OS/2 that does
    > this.


    Then write one... ;-)

    > If such a file system would really exists, how would one
    > access it in a generic way i.e without knowing about
    > these specialities ?


    As usual. Presumably, some operations do not make sense; you will get
    some error when you do such an operation.

    > There is no OS/2 error like: "I
    > cannot seek forward, please do a dummy read" or "The
    > distance it to big, please try a smaller one".


    There are many situations when different error conditions translate to
    one error number. Certainly you know of this...

    > If a device has such limitations, it is the job of the file system /
    > device driver to hide them from the rest of the system.


    If the error is hideable, sure. If it is not - sure not.

    > file approach. Of course, I'm not using handles inherited
    > from some other process...


    Right, the moment you inherit something, problems appear.

    If it were possible to flip the flag, the first application would be
    to make pdksh's file redirections be L-ed; it fork()s to start an
    external program anyway, so I could flip the file descriptors to the
    long state immediately before exec()ing the external program.

    This would keep internally used 32-bit offsets without a change (since
    internally used files would be DosOpen()ed, not DosOpenL()ed), but
    would provide external program with 64-bit accessible redirections.

    Thanks,
    Ilya

  9. Re: Read/Write behind 2G

    On Sat, 12 Feb 2005 01:01:23 UTC, Ilya Zakharevich
    wrote:

    > Eh? I do not see it documented what happens if the
    > position is after EOF. What your docs say?


    The API docs are not specific, but they don't disallow it.
    They just clearly state, that negative file positions are
    not allowed. So what sense would it make to define a specific
    error number for seeking "below zero" and none for the "beyond
    EOF" case ? My obersevation on various OSes is, that seeking
    beyond EOF is generally a legal operation.

    The C standard library man pages say this:

    man> The lseek function allows the file offset to be set beyond
    man> the end of the existing end-of-file of the file. If data
    man> is later written at this point, subsequent reads of the
    man> data in the gap return bytes of zeros (until data is actuğ
    man> ally written into the gap).

    What they describe here, is actually the creation of a sparse
    file. There are file systems (for example NetWare), that honour
    this by not allocating physical disk space for the gap.


    > Then write one... ;-)


    That would be a buggy one ...


    > As usual. Presumably, some operations do not make sense; you
    > will get some error when you do such an operation.


    Which error code ? There is no specific error code for the
    situations you describe. Returning a generic error is not a
    solution in this case, because the operation may be valid
    for one device and invalid for another. So how should the
    application (which may only have a handle) react ?


    >If the error is hideable, sure. If it is not - sure not.


    Then the device is not suitable for that purpose at all.


    > If it were possible to flip the flag, the first application would be
    > to make pdksh's file redirections be L-ed; it fork()s to start an
    > external program anyway, so I could flip the file descriptors to the
    > long state immediately before exec()ing the external program.


    Hmm, I have not idea if it is possible to duplicate a non-"L" handle
    into an "L"-ed and let the child inherit the latter.


    --
    Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
    http://www.s-t.de
    Please remove all characters left of the "R" in my email address


  10. Re: Read/Write behind 2G

    [A complimentary Cc of this posting was sent to
    Ruediger Ihle
    ], who wrote in article :
    > > Eh? I do not see it documented what happens if the
    > > position is after EOF. What your docs say?

    >
    > The API docs are not specific, but they don't disallow it.

    .....
    > The C standard library man pages say this:


    This has no relevance on OS operation...

    > What they describe here, is actually the creation of a sparse
    > file. There are file systems (for example NetWare), that honour
    > this by not allocating physical disk space for the gap.


    You mean "NetWare on OS/2", right? Most "Unix" filesystems support gaps...

    > > Then write one... ;-)


    > That would be a buggy one ...


    ;-)

    > > As usual. Presumably, some operations do not make sense; you
    > > will get some error when you do such an operation.

    >
    > Which error code ?


    Whatever the file system will judge appropriate. In the worst case,
    ERROR_INVALID_PARAMETER.

    > So how should the application (which may only have a handle) react ?


    As usual: if it can't handle the error condition itself, it should
    propagate the error to the user.

    > >If the error is hideable, sure. If it is not - sure not.


    > Then the device is not suitable for that purpose at all.


    "That purpose"? I do not know what you have in your mind.

    > > If it were possible to flip the flag, the first application would be
    > > to make pdksh's file redirections be L-ed; it fork()s to start an
    > > external program anyway, so I could flip the file descriptors to the
    > > long state immediately before exec()ing the external program.


    > Hmm, I have not idea if it is possible to duplicate a non-"L" handle
    > into an "L"-ed and let the child inherit the latter.


    Pity. I hoped to avoid global changes...

    Thanks,
    Ilya

  11. Re: Read/Write behind 2G

    On Sat, 12 Feb 2005 21:01:15 UTC, Ilya Zakharevich wrote:

    > This has no relevance on OS operation...


    O.K. so how about this one, taken from the official IFS developer documentation, section
    "FS Service Routines", subfunction FS_CHGFILEPTR:

    IBM> The file system may want to take the seek operation as a hint that an I/O operation
    IBM> is about to take place at the new position and initiate a positioning operation on
    IBM> sequential access media or read-ahead operation on other media.
    IBM>
    IBM> Some DOS mode programs expect to be able to do a negative seek. OS/2 passes
    IBM> these requests on to the FSD and returns an error for OS/2 mode negative seek
    IBM> requests. Because a seek to a negative position is, effectively, a seek to a very
    IBM> large offset, it is suggested that the FSD return end-of-file for subsequent read
    IBM> requests.
    IBM>
    IBM> FSDs must allow seeks to positions beyond end-of-file.


    > You mean "NetWare on OS/2", right? Most "Unix" filesystems support gaps...


    No, I mean NetWare on a NetWare system.


    > As usual: if it can't handle the error condition itself, it should
    > propagate the error to the user.


    Cool. I'm about writing a document in my word processor. Saving the data to
    one drives says "OK". To another says "Sorry, something went wrong. Maybe
    you have to adjust your maximum seek distance or you cannot modify a chapter
    at the beginning of the document because the device can't seek backwards".
    I'd love such a system !


    > "That purpose"? I do not know what you have in your mind.


    I mean that such a device simply cannot be used to carry an (OS/2) file system.
    Maybe it's a character anyway...


    --
    Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
    http://www.s-t.de
    Please remove all characters left of the "R" in my email address


  12. Re: Read/Write behind 2G

    [A complimentary Cc of this posting was sent to
    Ruediger Ihle
    ], who wrote in article :
    > O.K. so how about this one, taken from the official IFS developer documentation, section
    > "FS Service Routines", subfunction FS_CHGFILEPTR:


    > IBM> FSDs must allow seeks to positions beyond end-of-file.


    Thanks, I missed the idea of looking into IFS documentation. However,
    since "must allow" gives no suggestion on what the semantic should be,
    this is not of substantial help.

    > > As usual: if it can't handle the error condition itself, it should
    > > propagate the error to the user.

    >
    > Cool. I'm about writing a document in my word processor. Saving the data to
    > one drives says "OK". To another says "Sorry, something went wrong. Maybe
    > you have to adjust your maximum seek distance or you cannot modify a chapter
    > at the beginning of the document because the device can't seek backwards".
    > I'd love such a system !


    Glad to be to your service.

    > > "That purpose"? I do not know what you have in your mind.


    > I mean that such a device simply cannot be used to carry an (OS/2)
    > file system. Maybe it's a character anyway...


    In my experience, combined intelligence of users of my software is
    always uncomparably stronger than mine. So allow me to disregard your
    judgement on usability of a FS you did not even see...

    Yours,
    Ilya

  13. Re: Read/Write behind 2G

    On Sun, 13 Feb 2005 10:13:17 UTC, Ilya Zakharevich wrote:

    > Thanks, I missed the idea of looking into IFS documentation. However,
    > since "must allow" gives no suggestion on what the semantic should be,
    > this is not of substantial help.


    For me this +
    the DosSetFilePtr() documentation +
    the behavior of existing OS/2 file systems +
    the behaviour of file systems on other platforms
    is substantial enough.


    > So allow me to disregard your judgement


    Feel free to do so.


    > on usability of a FS you did not even see...


    Show me, if it's not a hypothetical one.


    --
    Ruediger "Rudi" Ihle [S&T Systemtechnik GmbH, Germany]
    http://www.s-t.de
    Please remove all characters left of the "R" in my email address


  14. Re: Read/Write behind 2G

    [A complimentary Cc of this posting was sent to
    Ruediger Ihle
    ], who wrote in article :
    > On Sun, 13 Feb 2005 10:13:17 UTC, Ilya Zakharevich wrote:
    > For me this +
    > the DosSetFilePtr() documentation +
    > the behavior of existing OS/2 file systems +
    > the behaviour of file systems on other platforms
    > is substantial enough.


    I see. What I missed is that on "usual OS/2 suspects" a seek after
    EOF creates a sequence of 0-bytes on the next write (but, obviously,
    would not create a gap in a file). Thanks.

    > > So allow me to disregard your judgement on usability of a FS you
    > > did not even see...


    > Show me, if it's not a hypothetical one.


    Of course it is.

    Yours,
    Ilya

+ Reply to Thread