USB stick access freezing up system? - BSD

This is a discussion on USB stick access freezing up system? - BSD ; I've been playing with multiple USB sticks tonight, trying out RAID0 and RAID3 configurations. Under 6.3-BETA1 (I think it was #1) I've noticed a couple of things... 1) After several writes the speed slows to a dribble, and the transaction ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 26

Thread: USB stick access freezing up system?

  1. USB stick access freezing up system?

    I've been playing with multiple USB sticks tonight, trying out RAID0
    and RAID3 configurations. Under 6.3-BETA1 (I think it was #1) I've
    noticed a couple of things...

    1) After several writes the speed slows to a dribble, and the
    transaction delay time (watched with iostat -x) steadily increases. It
    wanders up past 40,000 milliseconds (40 seconds!!) then suddenly it
    clears and the delay is back to a few milliseconds. I presume this is
    due to some sort of garbage collection and shuffling around of blocks
    by the flash memory.

    2) A more serious concern is that both the console and ssh logins
    become non responsive for relatively long periods of time when an
    application is doing heavy access to a USB stick. Pressing enter on
    the console makes a new line, but there's no prompt or other response;
    pressing enter when logged in via SSH does nothing at all. Just now it
    was 'suspended' for several minutes, the only indication that it had
    not hard locked was that it responded to pings.

    Has anyone else experienced these problems? I'm playing with flash
    memory to try to increase the transaction rate (and increase the life
    of the poor HD thrashing heavily) but if the machine freezes for
    minutes at a time there's not much point!

    CPU: E2160
    m/b: ASUS PK5-SE
    RAM: 2 x A-DATA 1GB DDR-800
    USB stick: Sandisk 4GB / generic 2GB (doesn't seem to make any
    difference)

    Thanks in advance for any suggestions.

  2. Re: USB stick access freezing up system?

    I just noticed a transaction time going past 240 seconds, which
    probably explains the state of suspended animation. FreeBSD seems to
    be ignoring just about everything else (in user space??) until the
    write completes. What could cause a multiuser system to grind to a
    halt like this?

    Reminds me a little of Windows 3.1 when there was a bad sector on a
    floppy

  3. Re: USB stick access freezing up system?

    googlegroups@sensation.net.au wrote:
    > Has anyone else experienced these problems? I'm playing with flash
    > memory to try to increase the transaction rate (and increase the life
    > of the poor HD thrashing heavily) but if the machine freezes for
    > minutes at a time there's not much point!


    You are aware that flash memory in general (and therefore usb memory
    sticks) isn't a good candidate as a replacement for the normal
    filesystems that usually is put on a hard disk?

    > CPU: E2160
    > m/b: ASUS PK5-SE
    > RAM: 2 x A-DATA 1GB DDR-800
    > USB stick: Sandisk 4GB / generic 2GB (doesn't seem to make any
    > difference)
    >


    Provide more information. Like - what kind of usb controller does the
    machine have? uhci or ohci?
    What is the output of 'usbdevs -v' with the usb sticks connected?
    and so on.
    --
    Torfinn Ingolfsen,
    Norway

  4. Re: USB stick access freezing up system?

    On Dec 5, 10:46 am, Torfinn Ingolfsen wrote:
    > googlegro...@sensation.net.au wrote:
    > > Has anyone else experienced these problems? I'm playing with flash
    > > memory to try to increase the transaction rate (and increase the life
    > > of the poor HD thrashing heavily) but if the machine freezes for
    > > minutes at a time there's not much point!

    >
    > You are aware that flash memory in general (and therefore usb memory
    > sticks) isn't a good candidate as a replacement for the normal
    > filesystems that usually is put on a hard disk?


    Hello Torfinn,

    I'll explain a little about my application... for each object it
    examines it needs to check refererences to see whether to add further
    ones, which in some cases means 2000+ key searches of the db per
    object. As you can imagine the HD goes crazy. I've tried experimenting
    with increasing nbufs to use extra memory for disk caching but the
    size and scope of the db is just too large for it to have much overall
    benefit. (It will eventually have several hundred million objects in
    it)

    I decided to try flash in order to give my hard drive a break. I
    expected the r/w speed would be slower, but the average access time
    would be quicker. Needless to say it's not working so far, average
    write speed with the HD was 100+ keys per second, with the flash it's
    about 3-5/sec.

    Why isn't flash a good candidate?

    > > CPU: E2160
    > > m/b: ASUS PK5-SE
    > > RAM: 2 x A-DATA 1GB DDR-800
    > > USB stick: Sandisk 4GB / generic 2GB (doesn't seem to make any
    > > difference)

    >
    > Provide more information. Like - what kind of usb controller does the
    > machine have? uhci or ohci?
    > What is the output of 'usbdevs -v' with the usb sticks connected?
    > and so on.


    Here's the output, there are currently 3 sticks connected (presumably
    the entries under /dev/usb3)

    What's the difference between UHCI and EHCI? It's an Intel ICH9R
    chipset.

    Thanks for your help so far.

    # usbdevs -v
    Controller /dev/usb0:
    addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
    Intel(0x0000), rev 1.00
    port 1 powered
    port 2 powered
    Controller /dev/usb1:
    addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
    Intel(0x0000), rev 1.00
    port 1 powered
    port 2 powered
    Controller /dev/usb2:
    addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
    Intel(0x0000), rev 1.00
    port 1 powered
    port 2 powered
    Controller /dev/usb3:
    addr 1: high speed, self powered, config 1, EHCI root hub(0x0000),
    Intel(0x0000), rev 1.00
    port 1 addr 2: high speed, power 180 mA, config 1, product
    0x0020(0x0020), SanDisk(0x08ec), rev 2.00
    port 2 powered
    port 3 addr 3: high speed, power 180 mA, config 1, product
    0x0020(0x0020), SanDisk(0x08ec), rev 2.00
    port 4 addr 4: high speed, power 180 mA, config 1, product
    0x0020(0x0020), SanDisk(0x08ec), rev 2.00
    port 5 powered
    port 6 powered
    Controller /dev/usb4:
    addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
    Intel(0x0000), rev 1.00
    port 1 powered
    port 2 powered
    Controller /dev/usb5:
    addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
    Intel(0x0000), rev 1.00
    port 1 powered
    port 2 powered
    Controller /dev/usb6:
    addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
    Intel(0x0000), rev 1.00
    port 1 powered
    port 2 powered
    Controller /dev/usb7:
    addr 1: high speed, self powered, config 1, EHCI root hub(0x0000),
    Intel(0x0000), rev 1.00
    port 1 powered
    port 2 powered
    port 3 powered
    port 4 powered
    port 5 powered
    port 6 powered


  5. Re: USB stick access freezing up system?

    On Dec 5, 10:33 pm, jpd wrote:
    > Begin <660e30c8-e32d-412a-957b-554a93507...@d4g2000prg.googlegroups.com>
    > On Wed, 5 Dec 2007 02:44:10 -0800 (PST),
    >
    > googlegro...@sensation.net.au wrote:
    > > On Dec 5, 6:21 pm, Torfinn Ingolfsen wrote:
    > >> googlegro...@sensation.net.au wrote:
    > >> > I'll explain a little about my application... for each object it
    > >> > examines it needs to check refererences to see whether to add
    > >> > further ones, which in some cases means 2000+ key searches of the
    > >> > db per object. [...] (It will eventually have several hundred
    > >> > million objects in it)

    >
    > You can ameliorate it somewhat with caching the key lookups, but there
    > is a clear speedup gain in reducing the number of lookups needed.


    Caching is only useful to a certain point, because the lookup keys are
    generally not clustered in any way. The application is a web crawler;
    when it processes a page it needs to decide whether to add any URLs it
    finds to the fetch queue, so each unique URL on that page is a db
    query. This is fine for normal pages with only a few external links on
    average but it is choking on "blackhat" networks which have thousands
    of pages of junk text with thousands of links to their other pages.
    Crawling through a single network like this could result in several
    million db queries which could have been time better spent on
    processing a few hundred thousand normal pages. (Unfortunately,
    detecting and excluding the blackhat networks is not an option since
    this index needs to be as complete as possible. It may be possible to
    bump down their priority a little though.)


    > > I'm also confused why FreeBSD panics when the stick is removed. I
    > > would expect the application to fail, but the stick is just a mount in
    > > the app's directory tree, nothing essential...

    >
    > You know that but your system does not know that. FreeBSD does not make
    > a distinction between disks and flash, or cds, or whatever. A mount is a
    > mount. If the reader could signal the event you could set it up so that
    > it'll force-unmount the disk, but you risk data loss and even filesystem
    > corruption that way.


    I still don't understand why this is considered a "give up and
    restart" situation. Does this mean that if a non RAIDed drive fails a
    FreeBSD box will reboot (possibly repeatedly and endlessly), or is it
    only applicable to umass devices? I've been using FreeBSD for 11 years
    but I've never actually had a drive fail while a box is up and running.

  6. Re: USB stick access freezing up system?

    jpd wrote:
    >
    > > I'm also confused why FreeBSD panics when the stick is removed. I
    > > would expect the application to fail, but the stick is just a mount in
    > > the app's directory tree, nothing essential...

    >
    > You know that but your system does not know that. FreeBSD does not make
    > a distinction between disks and flash, or cds, or whatever. A mount is a
    > mount. If the reader could signal the event you could set it up so that
    > it'll force-unmount the disk, but you risk data loss and even filesystem
    > corruption that way.
    >


    Yes, but Linux does that. I have removed an USB stick by inadvertance
    without unmounting it, the system has taken care of doing automatically
    everything necessary, dismount the mount point (with perhaps loss of
    data), remove the icon on the desktop, etc. and i have seen no
    instability whatsoever after that. Thinking that a panic is a better
    solution seems to me quite strange, you have the same loss of data plus
    big inconvenience.

    >


    --

    Michel TALON


  7. Re: USB stick access freezing up system?

    Begin <4fa6365f-8c58-4e75-9cee-27a83f865085@a39g2000pre.googlegroups.com>
    On Wed, 5 Dec 2007 05:13:15 -0800 (PST),
    googlegroups@sensation.net.au wrote:
    > On Dec 5, 10:33 pm, jpd wrote:
    > The application is a web crawler; when it processes a page it needs
    > to decide whether to add any URLs it finds to the fetch queue, so
    > each unique URL on that page is a db query. This is fine for normal
    > pages with only a few external links on average but it is choking
    > on "blackhat" networks which have thousands of pages of junk text
    > with thousands of links to their other pages. [...] (Unfortunately,
    > detecting and excluding the blackhat networks is not an option since
    > this index needs to be as complete as possible. It may be possible to
    > bump down their priority a little though.)


    If it's for human consumption it'd make sense to prune them, provided you
    could do it with any accuracy at all. Your design decision, of course.


    > I still don't understand why this is considered a "give up and
    > restart" situation. Does this mean that if a non RAIDed drive fails a
    > FreeBSD box will reboot (possibly repeatedly and endlessly), or is it
    > only applicable to umass devices?


    Suppose you yanked the root disk. Could it run?

    It might reboot then panic on failing to mount root. If that causes it
    to reboot, it'll keep on cycling.

    The system expects mounted things to be *there* and neither it nor
    applications have an useful response if mounts vanish. As mentioned,
    it doesn't differentiate between types or grades of mount. Compare NFS
    mounts when the server suddenly hangs.

    It's a design decision, and an old one. Doing anything else is trying
    to handle errors that cannot be meaningfully handled, at minimum with
    regards to the data, in the majority of cases. The assumption is also
    very widespread in the system. Unix is designed to be always on, not to
    be turned off or have parts ripped out of it on a whim.

    Think about it. How would you handle it? Does that solution work for all
    cases? What does it mean in extra error codes and handling in programs?


    --
    j p d (at) d s b (dot) t u d e l f t (dot) n l .
    This message was originally posted on Usenet in plain text.
    Any other representation, additions, or changes do not have my
    consent and may be a violation of international copyright law.

  8. Re: USB stick access freezing up system?

    In article ,
    jpd wrote:

    >> I'm also confused why FreeBSD panics when the stick is removed. I
    >> would expect the application to fail, but the stick is just a mount in
    >> the app's directory tree, nothing essential...


    >You know that but your system does not know that. FreeBSD does not make
    >a distinction between disks and flash, or cds, or whatever. A mount is a
    >mount. If the reader could signal the event you could set it up so that
    >it'll force-unmount the disk, but you risk data loss and even filesystem
    >corruption that way.


    How can it corrupt a filesystem on a device that's been removed? Or
    did you mean corrupt some other filesystem (surely not).

    Removing a device will probably result in data loss and perhaps
    corruption of its filesystem, but panic()ing rather than forced
    unmounting does nothing not fix that.

    So I guess you must just mean that it would be bad to get in the habit
    of unplugging devices because you can get away with it, but I don't
    see panic()ing as an appropriate way to discourage that.

    -- Richard
    --
    "Consideration shall be given to the need for as many as 32 characters
    in some alphabets" - X3.4, 1963.

  9. Re: USB stick access freezing up system?

    Begin
    On Wed, 5 Dec 2007 13:52:32 +0000 (UTC),
    Michel Talon wrote:
    > jpd wrote:
    >>
    >> > I'm also confused why FreeBSD panics when the stick is removed. I
    >> > would expect the application to fail, but the stick is just a mount in
    >> > the app's directory tree, nothing essential...

    >>
    >> You know that but your system does not know that. FreeBSD does not make
    >> a distinction between disks and flash, or cds, or whatever. A mount is a
    >> mount. If the reader could signal the event you could set it up so that
    >> it'll force-unmount the disk, but you risk data loss and even filesystem
    >> corruption that way.

    >
    > Yes, but Linux does that. [...] Thinking that a panic is a better
    > solution seems to me quite strange, you have the same loss of data
    > plus big inconvenience.


    So you're saying silent loss of data is to be preferrable? I think
    plenty of people will strongly disagree, even if it does have merits in
    some cases. It does not in many other cases.

    I already said that you could setup FreeBSD to do the same. The point
    was to explain that the current behaviour is a design decision.

    You could decide to silently mount a disk read only if it starts to
    fail. No guarantee the disk won't silently deteriorate further anyway
    because of sectorshuffling when it notices certain sectors have become
    unreadable. Do I want that to happen? Not in all cases, no.


    The problem includes that mounting something doesn't just mean ``you
    can access the data now'', but that it means quite a lot in terms of
    promises to the system. The easiest way around the problem is to not
    mount at all. Cue mtools.


    --
    j p d (at) d s b (dot) t u d e l f t (dot) n l .
    This message was originally posted on Usenet in plain text.
    Any other representation, additions, or changes do not have my
    consent and may be a violation of international copyright law.

  10. Re: USB stick access freezing up system?

    Begin
    On 5 Dec 2007 14:17:57 GMT, Richard Tobin wrote:
    > In article ,
    > jpd wrote:
    >
    >>> I'm also confused why FreeBSD panics when the stick is removed. I
    >>> would expect the application to fail, but the stick is just a mount in
    >>> the app's directory tree, nothing essential...

    >
    >>You know that but your system does not know that. FreeBSD does not make
    >>a distinction between disks and flash, or cds, or whatever. A mount is a
    >>mount. If the reader could signal the event you could set it up so that
    >>it'll force-unmount the disk, but you risk data loss and even filesystem
    >>corruption that way.

    >
    > How can it corrupt a filesystem on a device that's been removed? Or
    > did you mean corrupt some other filesystem (surely not).
    >
    > Removing a device will probably result in data loss and perhaps
    > corruption of its filesystem, but panic()ing rather than forced
    > unmounting does nothing not fix that.


    I was talking about losing partial writes and metadata problems. Of
    course, the medium itself won't be available for further scribbling, but
    what if your medium happens to need signals from the hardware to finish
    up a session? This is not unheard of. Drives might need parking[1], CDs
    might need their session closed[2], etc.

    As to the rest of the storage, probably not directly, but ``surely not''
    is an assumption that you can't prove to be true for all cases. I could
    conceive of side effects, programs concluding to go on and scribble
    elsewhere, causing inconsistencies and whatnot. Far-fetched but again
    not inconceivable. Reparability of that I won't speculate on.


    > So I guess you must just mean that it would be bad to get in the habit
    > of unplugging devices because you can get away with it, but I don't
    > see panic()ing as an appropriate way to discourage that.


    You'd be guessing wrong. As currently setup, removing storage without
    unmounting is a very clear no-no, but it's not done just to discourage
    people from doing it.


    [1] I have transported an MFM disk that had not been parked with the
    proper utility before powering down. Damage had resulted.
    This was on an old XT running DOS.
    [2] CD burners usually lock their tray while busy, but that is besides
    the point. They need that peace and quiet to do their thing.

    --
    j p d (at) d s b (dot) t u d e l f t (dot) n l .
    This message was originally posted on Usenet in plain text.
    Any other representation, additions, or changes do not have my
    consent and may be a violation of international copyright law.

  11. Re: USB stick access freezing up system?

    jpd wrote:
    > >
    > > Yes, but Linux does that. [...] Thinking that a panic is a better
    > > solution seems to me quite strange, you have the same loss of data
    > > plus big inconvenience.

    >
    > So you're saying silent loss of data is to be preferrable? I think


    Yes, far, far preferable than panicking. Panicking is the worst solution
    possible in all the possibles. Even DragonFlyBSD has "solved" this
    problem in the same way as Linux, same way as Windows, basically same
    way as anyone else. Clearly if you remove an USB key without unmounting
    it, you will lose the buffers that were waiting to be written. Everybody
    expects that, and nobody will complain that he has lost such buffers
    when removing the key. Panicking not only loses these buffers, but
    moreover endangers other data and programs which may have been of great
    value and were totally irrelevant to the USB transfer. This is
    completely crazy.

    > plenty of people will strongly disagree, even if it does have merits in
    > some cases. It does not in many other cases.
    >


    In my opinion, many "design" solutions have the only merit that people are
    used to them, but have been chosen initially in a totally arbitrary way.

    --

    Michel TALON


  12. Re: USB stick access freezing up system?

    Begin
    On Wed, 5 Dec 2007 15:33:39 +0000 (UTC),
    Michel Talon wrote:
    > jpd wrote:
    >> >
    >> > Yes, but Linux does that. [...] Thinking that a panic is a better
    >> > solution seems to me quite strange, you have the same loss of data
    >> > plus big inconvenience.

    >>
    >> So you're saying silent loss of data is to be preferrable? I think

    >
    > Yes, far, far preferable than panicking. Panicking is the worst solution
    > possible in all the possibles.


    Show how.

    You could start with showing how ripping out the rootdisk of a booted
    system and not panicking makes sense.


    Now suppose it also contained, oh, medical records. Maybe those medical
    records are used to steer an X-ray machine? Oh, no worries, it will just
    substitute default or maybe random values and truck on.

    Show that killing the patient by radiation overdose is better than just
    shutting down so that the operating nurse knows the machine doesn't
    work.


    Sure, it's far fetched. But it is a possible and valid scenario. The
    point is that different situations have different needs, contrary
    to your declaration above. Silent data loss is not always the best
    solution. Panicking is not always, by definition, the worst solution in
    all cases. Even if there are better solutions possible it is not the
    worst in this example. This invalidates your claim.


    > Even DragonFlyBSD has "solved" this problem in the same way as Linux,
    > same way as Windows, basically same way as anyone else.


    ``everybody else uses windows or linux, both do this, therefore everybody
    else must be right''

    I think I'm detecting a wee bit of rhetoric here.


    > Clearly if you remove an USB key without unmounting it, you will lose
    > the buffers that were waiting to be written.


    We weren't just talking about USB keys. Right at the start I said that
    mounting USB sticks uses the same mechanism as mounting disks.


    > Everybody expects that, and nobody will complain that he has lost such
    > buffers when removing the key.


    Go talk to anybody who holds users' hands for a living. Picture this:

    ``I deleted everything on my disk, then I emptied the bin. BUT I NEED
    MY DATA! YOU MUST BRING IT BACK! YOU MUST YOU MUST YOU MUST!''


    Ergo, more rhetoric. It also blithely ignores reality. I can easily show
    this by pointing to macintosh floppy drives. They did not have an eject
    button[1] for a reason.


    > Panicking not only loses these buffers, but moreover endangers other
    > data and programs which may have been of great value and were totally
    > irrelevant to the USB transfer. This is completely crazy.


    You're arguing a point that I already said was a design decision, made
    oh, three and a half decades ago. There may have been other options and
    they will have had different tradeoffs. If you look at it in context of
    the design parameters back then, it makes perfect sense.

    You, instead, are pretending them people back then must have been stark
    hopping mad because in the meantime the world has changed.


    > In my opinion, [...]


    That's nice. But I wasn't discussing that. Or mine, for that matter.


    [1] They did have an emergency eject hole and lever. Your point?

    --
    j p d (at) d s b (dot) t u d e l f t (dot) n l .
    This message was originally posted on Usenet in plain text.
    Any other representation, additions, or changes do not have my
    consent and may be a violation of international copyright law.

  13. Re: USB stick access freezing up system?

    On Dec 6, 1:06 am, jpd wrote:
    > Begin <4fa6365f-8c58-4e75-9cee-27a83f865...@a39g2000pre.googlegroups.com>
    > On Wed, 5 Dec 2007 05:13:15 -0800 (PST),
    >
    > googlegro...@sensation.net.au wrote:
    > > I still don't understand why this is considered a "give up and
    > > restart" situation. Does this mean that if a non RAIDed drive fails a
    > > FreeBSD box will reboot (possibly repeatedly and endlessly), or is it
    > > only applicable to umass devices?

    >
    > Suppose you yanked the root disk. Could it run?
    >
    > It might reboot then panic on failing to mount root. If that causes it
    > to reboot, it'll keep on cycling.
    >
    > The system expects mounted things to be *there* and neither it nor
    > applications have an useful response if mounts vanish. As mentioned,
    > it doesn't differentiate between types or grades of mount. Compare NFS
    > mounts when the server suddenly hangs.
    >
    > It's a design decision, and an old one. Doing anything else is trying
    > to handle errors that cannot be meaningfully handled, at minimum with
    > regards to the data, in the majority of cases. The assumption is also
    > very widespread in the system. Unix is designed to be always on, not to
    > be turned off or have parts ripped out of it on a whim.
    >
    > Think about it. How would you handle it? Does that solution work for all
    > cases? What does it mean in extra error codes and handling in programs?


    I would say "discard all related buffers, syslog a fatal device
    problem, then return an error on any further I/O attempts" but I
    suspect you'll have some reason not to do that. Windows XP did this
    when I had problems with a flaky SATA cable, since it was a RAID0
    array there was no possible way it could continue reading or writing
    from the "drive" so it popped up a balloon and said that further
    access was not possible. No crashing involved though.

  14. Re: USB stick access freezing up system?

    In article ,
    jpd wrote:
    >Begin
    >On Wed, 5 Dec 2007 15:33:39 +0000 (UTC),
    >Michel Talon wrote:
    >> jpd wrote:
    >>> >
    >>> > Yes, but Linux does that. [...] Thinking that a panic is a better
    >>> > solution seems to me quite strange, you have the same loss of data
    >>> > plus big inconvenience.
    >>>
    >>> So you're saying silent loss of data is to be preferrable? I think

    >>
    >> Yes, far, far preferable than panicking. Panicking is the worst solution
    >> possible in all the possibles.

    >
    >Show how.
    >
    >You could start with showing how ripping out the rootdisk of a booted
    >system and not panicking makes sense.
    >
    >
    >Now suppose it also contained, oh, medical records. Maybe those medical
    >records are used to steer an X-ray machine? Oh, no worries, it will just
    >substitute default or maybe random values and truck on.
    >
    >Show that killing the patient by radiation overdose is better than just
    >shutting down so that the operating nurse knows the machine doesn't
    >work.
    >
    >
    >Sure, it's far fetched. But it is a possible and valid scenario. The
    >point is that different situations have different needs, contrary
    >to your declaration above. Silent data loss is not always the best
    >solution. Panicking is not always, by definition, the worst solution in
    >all cases. Even if there are better solutions possible it is not the
    >worst in this example. This invalidates your claim.
    >
    >
    >> Even DragonFlyBSD has "solved" this problem in the same way as Linux,
    >> same way as Windows, basically same way as anyone else.

    >
    >``everybody else uses windows or linux, both do this, therefore everybody
    > else must be right''
    >
    >I think I'm detecting a wee bit of rhetoric here.
    >
    >
    >> Clearly if you remove an USB key without unmounting it, you will lose
    >> the buffers that were waiting to be written.

    >
    >We weren't just talking about USB keys. Right at the start I said that
    >mounting USB sticks uses the same mechanism as mounting disks.
    >
    >
    >> Everybody expects that, and nobody will complain that he has lost such
    >> buffers when removing the key.

    >
    >Go talk to anybody who holds users' hands for a living. Picture this:
    >
    >``I deleted everything on my disk, then I emptied the bin. BUT I NEED
    > MY DATA! YOU MUST BRING IT BACK! YOU MUST YOU MUST YOU MUST!''
    >
    >
    >Ergo, more rhetoric. It also blithely ignores reality. I can easily show
    >this by pointing to macintosh floppy drives. They did not have an eject
    >button[1] for a reason.
    >
    >
    >> Panicking not only loses these buffers, but moreover endangers other
    >> data and programs which may have been of great value and were totally
    >> irrelevant to the USB transfer. This is completely crazy.

    >
    >You're arguing a point that I already said was a design decision, made
    >oh, three and a half decades ago. There may have been other options and
    >they will have had different tradeoffs. If you look at it in context of
    >the design parameters back then, it makes perfect sense.
    >
    >You, instead, are pretending them people back then must have been stark
    >hopping mad because in the meantime the world has changed.


    >> In my opinion, [...]


    >That's nice. But I wasn't discussing that. Or mine, for that matter.


    >[1] They did have an emergency eject hole and lever. Your point?


    And in the old days my 8" drives could not be opened [at least some
    of them] when there was any activity at all. Those being
    8" floppy drives.

    But as things needed to be made cheaper all the protections were
    slowly deleted.

    In the 5.25" floppy world you put on a TAB to write protect the
    disk. The cheaper tabs often had glue/adhesive that would dry so
    the tab would fall off and you could accidentally write to them.

    3M [AKA Scotch] made plastic tabs that stuck on nicely. But some
    of the drive manufacturers found that it was cheaper to use an
    optical sensor instead of a mechanical one to detect the
    write-protect tab.

    The optical sensors were infra-red and the 3M tabs were transparent
    to the light - voila - no write protection at all.

    OTOH the 8" floppies use a write-enable tab so a missing tab
    would mean that you could not write to the disk. This was
    a 'fail-safe' approach, something that is missing in so many
    products today.

    You could buy the 8" floppies with or without notches - and it was
    easy to just take a hole punch to convert a write-enabled disk
    to write-protected.

    Then there is the old 'legend' about the computer admin who wanted
    to make sure that NO ONE could put on a write-enable ring on
    the reel-reel tape being used then.

    So he filled the slot where the rubber ring would go with expoxy
    so you could not put the ring in place. Of course the mechanical
    sensors saw that the tape was write enabled. Unintended
    consequences because of lack of understanding.

    I do >>NOT<< miss the old days at all.

    Bill

    --
    Bill Vermillion - bv @ wjv . com

  15. Re: USB stick access freezing up system?

    Michel Talon wrote:
    > jpd wrote:
    >>> Yes, but Linux does that. [...] Thinking that a panic is a better
    >>> solution seems to me quite strange, you have the same loss of data
    >>> plus big inconvenience.

    >> So you're saying silent loss of data is to be preferrable? I think

    >
    > Yes, far, far preferable than panicking. Panicking is the worst solution
    > possible in all the possibles. Even DragonFlyBSD has "solved" this
    > problem in the same way as Linux, same way as Windows, basically same
    > way as anyone else. Clearly if you remove an USB key without unmounting
    > it, you will lose the buffers that were waiting to be written. Everybody
    > expects that, and nobody will complain that he has lost such buffers
    > when removing the key. Panicking not only loses these buffers, but
    > moreover endangers other data and programs which may have been of great
    > value and were totally irrelevant to the USB transfer. This is
    > completely crazy.
    >
    >> plenty of people will strongly disagree, even if it does have merits in
    >> some cases. It does not in many other cases.
    >>

    >
    > In my opinion, many "design" solutions have the only merit that people are
    > used to them, but have been chosen initially in a totally arbitrary way.
    >


    OpenVMS puts the disk into mount verification and suspends _ALL_ I/O to
    the disk until it can be verified. This involves "pinging" the disk,
    then reading the super-block to verify its the same volume.

    After mount verification completes, all suspended I/O is restarted to
    the disk.

    Works EVEN ON THE SYSTEM (ROOT) DISK.

    One can turn off a disk, then turn it back on. Of course, if the disk
    maintains its own internal write-cache, one could lose data.

    There is no system panic, there is no lost data due to unwritten buffers
    - with the exception of the disk's write-cache. But who enables the
    disk write-cache for production, mission critical disks?

    Chuck

  16. Re: USB stick access freezing up system?

    In article ,
    jpd wrote:

    >> How can it corrupt a filesystem on a device that's been removed? Or
    >> did you mean corrupt some other filesystem (surely not).
    >>
    >> Removing a device will probably result in data loss and perhaps
    >> corruption of its filesystem, but panic()ing rather than forced
    >> unmounting does nothing not fix that.


    >I was talking about losing partial writes and metadata problems. Of
    >course, the medium itself won't be available for further scribbling, but
    >what if your medium happens to need signals from the hardware to finish
    >up a session? This is not unheard of. Drives might need parking[1], CDs
    >might need their session closed[2], etc.


    But how does panic()ing help this at all?

    >As to the rest of the storage, probably not directly, but ``surely not''
    >is an assumption that you can't prove to be true for all cases. I could
    >conceive of side effects, programs concluding to go on and scribble
    >elsewhere, causing inconsistencies and whatnot.


    Inconsistencies at the user level, yes. But not filesystem
    inconsistencies. And again, panic()ing is very likely to result in
    user-level problems in completely unrelated programs.

    >You'd be guessing wrong. As currently setup, removing storage without
    >unmounting is a very clear no-no, but it's not done just to discourage
    >people from doing it.


    I'm not suggesting that it shouldn't be a "no-no", just that screwing
    everything else on the system is pointless. No other operating system
    I use does it, and the result is that FreeBSD is less usable.

    Should FreeBSD panic on disk errors?

    -- Richard
    --
    :wq

  17. Re: USB stick access freezing up system?

    Begin
    On 7 Dec 2007 13:09:08 GMT, Richard Tobin wrote:
    > In article ,
    > jpd wrote:
    >>As to the rest of the storage, probably not directly, but ``surely not''
    >>is an assumption that you can't prove to be true for all cases. I could
    >>conceive of side effects, programs concluding to go on and scribble
    >>elsewhere, causing inconsistencies and whatnot.

    >
    > Inconsistencies at the user level, yes. But not filesystem
    > inconsistencies. And again, panic()ing is very likely to result in
    > user-level problems in completely unrelated programs.


    Assuming you can meaningfully differentiate ``related'' and
    ``unrelated'' programs. Can you? Can you while guaranteeing no
    unforeseen side effects? We'd love to hear about it.


    >>You'd be guessing wrong. As currently setup, removing storage without
    >>unmounting is a very clear no-no, but it's not done just to discourage
    >>people from doing it.

    >
    > I'm not suggesting that it shouldn't be a "no-no", just that screwing
    > everything else on the system is pointless.


    Your assumption of the existence of an ``everything else'' needs fixing.
    It may exist, or equally well may not.


    > No other operating system I use does it, and the result is that
    > FreeBSD is less usable.


    Well, if you believe that, don't use it, then. Your call.

    FreeBSD certainly is not the only system setup this way; it has been
    with us since the early days of unix at least.

    Linux did it just as much, until someone decided mounting r/o was
    ``better''. I know of people who have been nastily surprised by that
    behaviour that would not have happened had the system paniced instead.

    The point is that either may be preferrable in some situation. It's good
    to have a choice in the matter. The default for FreeBSD is to panic.


    > Should FreeBSD panic on disk errors?


    I think that to panic is the appropriate _default_ response, if nothing
    else because that's what experienced people expect it to do, and it really
    is the only general response appropriate to the breach of assumptions
    that happens when a mount vanishes. Not just USB sticks, any mount.

    You can of course make exceptions, but that is on your own judgement.


    --
    j p d (at) d s b (dot) t u d e l f t (dot) n l .
    This message was originally posted on Usenet in plain text.
    Any other representation, additions, or changes do not have my
    consent and may be a violation of international copyright law.

  18. Re: USB stick access freezing up system?

    In article ,
    Richard Tobin wrote:
    >In article ,
    >jpd wrote:
    >
    >>> How can it corrupt a filesystem on a device that's been removed? Or
    >>> did you mean corrupt some other filesystem (surely not).
    >>>
    >>> Removing a device will probably result in data loss and perhaps
    >>> corruption of its filesystem, but panic()ing rather than forced
    >>> unmounting does nothing not fix that.

    >
    >>I was talking about losing partial writes and metadata problems. Of
    >>course, the medium itself won't be available for further scribbling, but
    >>what if your medium happens to need signals from the hardware to finish
    >>up a session? This is not unheard of. Drives might need parking[1], CDs
    >>might need their session closed[2], etc.

    >
    >But how does panic()ing help this at all?
    >
    >>As to the rest of the storage, probably not directly, but ``surely not''
    >>is an assumption that you can't prove to be true for all cases. I could
    >>conceive of side effects, programs concluding to go on and scribble
    >>elsewhere, causing inconsistencies and whatnot.

    >
    >Inconsistencies at the user level, yes. But not filesystem
    >inconsistencies. And again, panic()ing is very likely to result in
    >user-level problems in completely unrelated programs.
    >
    >>You'd be guessing wrong. As currently setup, removing storage without
    >>unmounting is a very clear no-no, but it's not done just to discourage
    >>people from doing it.


    >I'm not suggesting that it shouldn't be a "no-no", just that screwing
    >everything else on the system is pointless. No other operating system
    >I use does it, and the result is that FreeBSD is less usable.


    Then you have missed a few commercail Unix systems :-)

    >Should FreeBSD panic on disk errors?


    There are disk errors, and then there are disk errors.

    I think a panic on trying to write to a disk that has been
    inadvertantly removed is acceptable. That may be far better
    than having you think something completed successfully and then
    find the end-of-month accounting runs, payroll, some-financials,
    etc. in error. Of course on systems that do things like that
    you often have a commit/rollback utility to make sure that
    all things are in a given state for a given period of time.

    But for other disk errors a panic probably should not occur, but
    problems should be written to some logging facility - and hope that
    isn't on a drive with an error - or perhaps performing your error
    logging to another machine/system.

    Bill
    >
    >-- Richard
    >--
    >:wq



    --
    Bill Vermillion - bv @ wjv . com

  19. Re: USB stick access freezing up system?

    In article , Bill Vermillion wrote:

    >I think a panic on trying to write to a disk that has been
    >inadvertantly removed is acceptable. That may be far better
    >than having you think something completed successfully and then
    >find the end-of-month accounting runs, payroll, some-financials,
    >etc. in error.


    Surely payroll systems can report errors in some way other than a
    kernel panic! Don't they check the result of fwrite() and fclose()?

    But of course most of us are not running payroll systems anyway.
    I wouldn't mind if there was a switch to enable panics on disk
    removal, but I think you'd find the vast majority would leave
    it in the "off" position.

    -- Richard
    --
    :wq

  20. Re: USB stick access freezing up system?

    Begin
    On 9 Dec 2007 12:34:41 GMT, Richard Tobin wrote:
    > In article , Bill Vermillion wrote:
    >>I think a panic on trying to write to a disk that has been
    >>inadvertantly removed is acceptable. That may be far better
    >>than having you think something completed successfully and then
    >>find the end-of-month accounting runs, payroll, some-financials,
    >>etc. in error.

    >
    > Surely payroll systems can report errors in some way other than a
    > kernel panic! Don't they check the result of fwrite() and fclose()?


    Do you think you can afford the cost should your assumptions prove
    false in the real world? Are you prepared to shell out for it?

    Your argument has merit insofar that it is possible in theory, but what
    about in practice? An example.

    Suppose write() suddenly returns ENXIO (``device not configured''). What
    would you do? Log an error. Suppose that returns ENXIO? Write a mail.
    And if THAT return ENXIO (mail program is on a mount that vanished)? Or
    maybe the program loads fine, but its spool was on the disk that is now
    gone. Suppose the mail program successfully tells you it gave up. Now
    what? Print it on stderr? It's a daemon and it doesn't have a controlling
    tty. Print it on the console, then? Say it's a server, and is headless?

    Do you think the message will come through? Does the program have any
    assurance of the message getting through? Can it tell?


    Payroll systems are notorious for being all-important, but obscure
    chunks of software running somewhere in the corner and nobody cares
    until someone reports it's been down for a while and the financial
    people have been twiddling their thumbs all morning/week/month/year.
    Without telling anybody. It happened to me, and this was a small
    company where they knew me and could just have told me, or written an
    email--email was on a different, functional, system. Then getting the
    message out it isn't a technology problem, but clearly a user problem.

    Sometimes you really do have to intentionally break things to make sure
    the people involved will finally get off their lazy arses and do the
    thing they were supposed to do already.

    Say you are the sysadmin responsible and thus will get to clean up the
    mess. In this case a misconfiguration had the log fill up the root. Do
    you want it trucking on in the meantime?

    For me, the answer is a clear no. I don't even know what it does when
    it does what it is supposed to do. Figuring out what it did instead and
    trying to reverse that? No chance.


    Do I want to risk your ``surely'' here? Heck no. For all its faults, if
    the rules of the universe start to come apart, collapsing the universe
    is the simplest proper response. What is more, it does not rely on
    the competence of the programmer of the application to make it do the
    expected thing.

    The alternative is called ``graceful degradation'', which for some
    things is not just desirable, but mandatory, and rightfully so. But
    its cost can easily be horrendous. Because you have to deal with all
    possibilities and some include sitations where normal operations have
    just become not merely tricky, but flat out impossible.

    In short, that easy ``surely'' of yours, comes with a steep price tag.
    Are you prepared to pay the cost?


    --
    j p d (at) d s b (dot) t u d e l f t (dot) n l .
    This message was originally posted on Usenet in plain text.
    Any other representation, additions, or changes do not have my
    consent and may be a violation of international copyright law.

+ Reply to Thread
Page 1 of 2 1 2 LastLast