Printable View
Help! As I write, kmail is generating lots of unkown subject, unknown date,
unknown everything mails and lots of my old mails seem to be corrupted,
mainly headers gone.
I can't find anything on the internet. Anybody come across this? Suggestions?
Please copy any answer to schmidt at umwelteinsatz.ch
Theo Schmidt
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
Bearjadat, čakčamánu 23. b. 2005 10:16, Theo Schmidt čálii:[color=blue]
> Help! As I write, kmail is generating lots of unkown subject, unknown date,
> unknown everything mails and lots of my old mails seem to be corrupted,
> mainly headers gone.
>
> I can't find anything on the internet. Anybody come across this?
> Suggestions?
>
> Please copy any answer to schmidt at umwelteinsatz.ch
>
> Theo Schmidt[/color]
I experienced this behavior when I used two antispam filters (spamassassin and
bogofilter). Look into the filter rules and remove the spamassassin or the
bogofilter rules. Hope it works :-)
--
Børre Gaup
Would two filters, one on the client and one on the server, have the same
effect? I've had this problem as well, and AFAIK I'm not using bogofilter
anywhere but I do have spamassassin on my server as well as on my KMail
client.
On Friday 23 September 2005 03:55 am, Børre Gaup wrote:[color=blue]
> Bearjadat, čakčamánu 23. b. 2005 10:16, Theo Schmidt čálii:[color=green]
> > Help! As I write, kmail is generating lots of unkown subject, unknown
> > date, unknown everything mails and lots of my old mails seem to be
> > corrupted, mainly headers gone.
> >
> > I can't find anything on the internet. Anybody come across this?
> > Suggestions?
> >
> > Please copy any answer to schmidt at umwelteinsatz.ch
> >
> > Theo Schmidt[/color]
>
> I experienced this behavior when I used two antispam filters (spamassassin
> and bogofilter). Look into the filter rules and remove the spamassassin or
> the bogofilter rules. Hope it works :-)[/color]
--
Larry Garfield AIM: LOLG42
[email]larry@garfieldtech.com[/email] ICQ: 6817012
"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson
On Friday 23 September 2005 04:16 am, Theo Schmidt wrote:[color=blue]
> Help! As I write, kmail is generating lots of unkown subject, unknown date,
> unknown everything mails and lots of my old mails seem to be corrupted,
> mainly headers gone.[/color]
Well, a little bit like the 2nd problem. Re the first, is kmail generating
those emails automatically, or are you generating them and they get sent out
with unknown subject, ...?
Anyway, the problem I had/have stemmed from using kmail on two different
machines with the mail stored on one machine. The bottom line, if I wasn't
careful, I'd try to access (old) mails with the indexes from the wrong
machine.
My solution:
* I'm very careful about using kmail from both machines, in fact, normally
I make sure I only run kmail from one machine at a time
* the solution to my problem is typically to regenerate the indexes. To do
that, I go into the mail directory (.Mail, iirc, although you may have it
somewhere else), delete the existing indexes, and let kmail regenerate them.
It's been a little while since I've had to do that, can'd recall if I shut
kmail down while I did that or not. In any case, the index for a particular
folder will be regenerated when you try to access that folder (from within
kmail)--there will be a (fairly short, iirc) delay while the index is
recreated.
Here, from directory ~/.Mail, is the ls -al listing for one (mbox) folder
(named tldp) with the three indexes. To start, you might want to experiment
with just one mail folder. Delete all three of the .index files, then
restart kmail (assuming you shut it down) and try to access the mail in that
folder. After a short delay, I'm hopeful that it will be ok.
-rw------- 1 rhk rhk 868526 Sep 19 16:05 tldp
-rw------- 1 rhk rhk 99043 Sep 19 16:05 .tldp.index
-rw-r--r-- 1 rhk rhk 937 Sep 19 16:05 .tldp.index.ids
-rw-r--r-- 1 rhk rhk 5425 Aug 24 07:28 .tldp.index.sorted
hope this helps,
Randy Kramer
[color=blue]
> I can't find anything on the internet. Anybody come across this?
> Suggestions?
>
> Please copy any answer to schmidt at umwelteinsatz.ch
>
> Theo Schmidt[/color]
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
Randy Kramer schrieb:
[color=blue]
> Well, a little bit like the 2nd problem. Re the first, is kmail
> generating
>
>those emails automatically, or are you generating them and they get sent out
>with unknown subject, ...?
>
>[/color]
No, not sending them, but I see there was also a problem with the server
of my provider, which may have set this off.
[color=blue]
>...
> * the solution to my problem is typically to regenerate the indexes. To do
>that, I go into the mail directory (.Mail, iirc, although you may have it
>somewhere else), delete the existing indexes, and let kmail regenerate them.
>
>...
>Here, from directory ~/.Mail, is the ls -al listing for one (mbox) folder
>(named tldp) with the three indexes. To start, you might want to experiment
>with just one mail folder. Delete all three of the .index files, then
>restart kmail (assuming you shut it down) and try to access the mail in that
>folder. After a short delay, I'm hopeful that it will be ok.
>
>-rw------- 1 rhk rhk 868526 Sep 19 16:05 tldp
>-rw------- 1 rhk rhk 99043 Sep 19 16:05 .tldp.index
>-rw-r--r-- 1 rhk rhk 937 Sep 19 16:05 .tldp.index.ids
>-rw-r--r-- 1 rhk rhk 5425 Aug 24 07:28 .tldp.index.sorted
>
>[/color]
This has solved it, thanks! Kmail used to compact mailboxes on closing;
it looks like it no longer does so.
Theo Schmidt
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
On Friday 23 September 2005 01:29 pm, Theo Schmidt wrote:[color=blue]
> This has solved it, thanks! Kmail used to compact mailboxes on closing;
> it looks like it no longer does so.[/color]
You're welcome! But there is something else I should have mentioned--hope you
haven't unindexed your inbox yet--
on my kmail system, compaction was totally disabled "for safety reasons"--when
I removed the index, all of a sudden I got thousands of old emails that had
been "marked for deletion" (my words) (and no longer visible) but never
actually deleted.
The note below (from my offline TWiki-like thing) tells how to enable
compaction on your inbox. I now try to remember to do that several times a
day.
regards,
Randy Kramer
---++ kmail: compact inbox disabled
* [[[url]http://mail.kde.org/pipermail/kmail-devel/2005-June/019581.html][/url][]]
<blockquote><blockquote><pre>
</pre>On Wednesday 29 June 2005 14:43, Edwin Schepers wrote:[color=blue]
> Hi,
> When I try to compact my inbox, I get the message that for safety reasons,
> compaction has been disabled for inbox.
> But is there some method to do the compaction ? I have an inbox of 140M now
> containing zero messages. I couldn't find any option. I guess `>inbox` is[/color]
not[color=blue]
> the proper way to do this.[/color]
Quit kmail (including the systray icon if you use it), open
~/.kde/share/config/kmailrc,
look for lines containing "Compactable=false", remove those lines, restart
kmail.
The previous discussions on how to improve this issue didn't lead to a
solution yet, it seems.
</blockquote></blockquote>
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Am Freitag, 23. September 2005 20:30 schrieb Randy Kramer:[color=blue]
> On Friday 23 September 2005 01:29 pm, Theo Schmidt wrote:[color=green]
> > This has solved it, thanks! Kmail used to compact mailboxes on closing;
> > it looks like it no longer does so.[/color]
>
> You're welcome! But there is something else I should have mentioned--hope
> you haven't unindexed your inbox yet--
>
> on my kmail system, compaction was totally disabled "for safety
> reasons"--when I removed the index, all of a sudden I got thousands of old
> emails that had been "marked for deletion" (my words) (and no longer
> visible) but never actually deleted.[/color]
When kmail detects a corrupted mbox file for the inbox (and for any other
folder), compacting it might eat all your mail, that's the safety reasons.
Without compacting, kmail can simply ignore the corrupted parts of the mbox
file, and it still works for everything else. But a compaction will mess this
all up.
Cheers,
Reinhold
- --
- ------------------------------------------------------------------
Reinhold Kainhofer, Vienna, Austria
email: [email]reinhold@kainhofer.com[/email], [url]http://reinhold.kainhofer.com/[/url]
* Financial and Actuarial Mathematics, TU Wien, [url]http://www.fam.tuwien.ac.at[/url]
* K Desktop Environment, [url]http://www.kde.org/[/url], KOrganizer maintainer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iD8DBQFDN6NNTqjEwhXvPN0RAgfYAJ9ducWlxeMhWlGh+UMl1r/xakifsACeI0x5
q/gct8MYPCJkJ6HVdwcHWjc=
=eujL
-----END PGP SIGNATURE-----
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
On Monday 26 September 2005 03:29 am, Reinhold Kainhofer wrote:[color=blue]
> Am Freitag, 23. September 2005 20:30 schrieb Randy Kramer:[color=green]
> > On Friday 23 September 2005 01:29 pm, Theo Schmidt wrote:[color=darkred]
> > > This has solved it, thanks! Kmail used to compact mailboxes on
> > > closing; it looks like it no longer does so.[/color]
> >
> > You're welcome! But there is something else I should have
> > mentioned--hope you haven't unindexed your inbox yet--
> >
> > on my kmail system, compaction was totally disabled "for safety
> > reasons"--when I removed the index, all of a sudden I got thousands of
> > old emails that had been "marked for deletion" (my words) (and no longer
> > visible) but never actually deleted.[/color]
>
> When kmail detects a corrupted mbox file for the inbox (and for any other
> folder), compacting it might eat all your mail, that's the safety reasons.
> Without compacting, kmail can simply ignore the corrupted parts of the mbox
> file, and it still works for everything else. But a compaction will mess
> this all up.[/color]
Reinhold,
Thanks for the information!
I don't know if you are the right person to ask, but I'll ask here anyway,
maybe someone will additional information.
Is there / will there ever be a fix for this? I mean, I certainly don't want
to lose emails (from my inbox or anywhere else) but also I don't want my
inbox to:
* grow without bounds filled with the text of removed emails
* bring back those deleted emails when I have to reindex the inbox to fix
some other problem (as discussed in this thread)
What do other email clients do? Do they:
* have a system of compaction that never fails, despite a perhaps corrupted
mbox file
* have a method to detect mbox corruption, perhaps when compaction is
invoked, warn the operator, stop/prevent the compaction, and suggest
corrective action to the user
* have the same potential problem (of potentially losing emails on
compaction of a corrupted mbox file), but just let the user learn the hard
way
* other?
regards,
Randy Kramer
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
Am Montag, 26. September 2005 14:07 schrieb Randy Kramer:[color=blue]
> Is there / will there ever be a fix for this?[/color]
Sure, rename your inbox to some other folder and let kmail recreate it as
maildir. maildir does not has such problems because every e-mail is a
seperate file. Speed then depends on the used filesystem (e.g. for lots of
small files if you only have short text mails). Maildir is usually faster for
mailboxes with lots of (big) attachments.
HS
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
On Monday 26 September 2005 12:30 pm, Hendrik Sattler wrote:[color=blue]
> Am Montag, 26. September 2005 14:07 schrieb Randy Kramer:[color=green]
> > Is there / will there ever be a fix for this?[/color]
>
> Sure, rename your inbox to some other folder and let kmail recreate it as
> maildir. maildir does not has such problems because every e-mail is a
> seperate file. Speed then depends on the used filesystem (e.g. for lots of
> small files if you only have short text mails). Maildir is usually faster
> for mailboxes with lots of (big) attachments.[/color]
Thanks!
I suppose maildir will be OK (and maybe even better) for my inbox, which I
generally keep "trimmed" (not too many emails).
I don't think I want to do that for my mail folders which often have a lot
(thousands) of archived emails (usually short).
Is it the general consensus that mbox is more subject to corruption than
maildir?
I guess I'll want to dig into that so more--not so much for email, but because
I'm building a plain text database with multiple records per file.
I use a text record separator to separate records (currently "\n---++ "). My
largest file at the moment is about 3000 records with 3 M characters. I
haven't seen the need to create an index yet, but that may be coming.
I guess if I get some corruption without the index, I'll eventually notice it
in some record that I look at it, and may be able to fix it, as everything is
visible plain text, at least when viewed in a (plain) text editor.
Well, anyway, I guess there's something for me to think about here.
regards,
Randy Kramer
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Am Montag, 26. September 2005 21:23 schrieb Randy Kramer:[color=blue]
> I suppose maildir will be OK (and maybe even better) for my inbox, which I
> generally keep "trimmed" (not too many emails).[/color]
Maildir is the recommended format. It's even the default for a new kmail
installation.
[color=blue]
> I don't think I want to do that for my mail folders which often have a lot
> (thousands) of archived emails (usually short).[/color]
So what? Our department's mail server uses maildir, and there are far more
than only thousands of messages.
On the other hand, for lots of messages, the mbox file needs to be searched /
loaded in whole, while in maildir only one file needs to be loaded.
[color=blue]
> Is it the general consensus that mbox is more subject to corruption than
> maildir?[/color]
In mbox, all messages are just concatenated together, so if there is even only
one broken byte (e.g. a null byte), it might possibly mess up all messages
that come afterwards. In maildir, each message is one file, so at most this
one message can be corrupted.
[color=blue]
> I use a text record separator to separate records (currently "\n---++ ").
> My largest file at the moment is about 3000 records with 3 M characters. I
> haven't seen the need to create an index yet, but that may be coming.
>
> I guess if I get some corruption without the index, I'll eventually notice
> it in some record that I look at it, and may be able to fix it, as
> everything is visible plain text, at least when viewed in a (plain) text
> editor.[/color]
Sure, but if you have a broken sector or something, some sequence might be
filled with random data, and while you might be able to manually spot this
and guess whether it's between two mails (i.e. a mail end/begin corrupted) or
halfway through a message, it's impossible for a computer.
This means that the index (some message states need to be stored in a separate
file, since mbox doesn't store them) of all later messages might possibly be
off by one... In particular, for all messages after that you can't be really
sure if that message is deleted or the one after it.
KMail uses the offset into the file in the index file, so this problem can be
worked around, but compacting will possibly remove too many / too less
messages.
Reinhold
- --
- ------------------------------------------------------------------
Reinhold Kainhofer, Vienna, Austria
email: [email]reinhold@kainhofer.com[/email], [url]http://reinhold.kainhofer.com/[/url]
* Financial and Actuarial Mathematics, TU Wien, [url]http://www.fam.tuwien.ac.at[/url]
* K Desktop Environment, [url]http://www.kde.org/[/url], KOrganizer maintainer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iD8DBQFDOGIaTqjEwhXvPN0RApkaAJ9KoVHE0iuANKr/LKyMfptfYoTY5ACgqauT
Cw5aMdYP6s49jTO5EhmEFpY=
=KbQr
-----END PGP SIGNATURE-----
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Am Montag, 26. September 2005 23:03 schrieb Reinhold Kainhofer:[color=blue]
> Am Montag, 26. September 2005 21:23 schrieb Randy Kramer:[color=green]
> > I suppose maildir will be OK (and maybe even better) for my inbox, which
> > I generally keep "trimmed" (not too many emails).[/color]
>
> Maildir is the recommended format. It's even the default for a new kmail
> installation.
>[color=green]
> > I don't think I want to do that for my mail folders which often have a
> > lot (thousands) of archived emails (usually short).[/color]
>
> So what? Our department's mail server uses maildir, and there are far more
> than only thousands of messages.
>
> On the other hand, for lots of messages, the mbox file needs to be searched
> / loaded in whole, while in maildir only one file needs to be loaded.[/color]
Oh I forgot: You said that you accessed the same mbox file from two different
machines. mbox format is known to cause massive mbox file corruption in this
case, since if one app adds/changes a message, the mbox file needs to be
changed (rewritten). If the second app changes the folder at the same time,
it looses either that message, or has wrong offsets in the mbox file. The
latter is really critical, since many mail applications don't rewrite the
whole mbox file (which would be madness for large folders, when you often
change message flags like read, answered, etc.) but rather use the binary
index of the message start into the message.
With Maildir, concurrent access is possible, since each message is its own
file. At most, the index might be out of sync, and in that case the mail
application can simply re-generate the index.
Cheers,
Reinhold
- --
- ------------------------------------------------------------------
Reinhold Kainhofer, Vienna, Austria
email: [email]reinhold@kainhofer.com[/email], [url]http://reinhold.kainhofer.com/[/url]
* Financial and Actuarial Mathematics, TU Wien, [url]http://www.fam.tuwien.ac.at[/url]
* K Desktop Environment, [url]http://www.kde.org/[/url], KOrganizer maintainer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iD8DBQFDOGRQTqjEwhXvPN0RAoofAJwI21s0a2y+Vc/J+GHWsRJFEFw+CACfdUe0
sL8RLGI8yUUuUiLu7dZhyZE=
=77jU
-----END PGP SIGNATURE-----
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
On Monday 26 September 2005 11:30 am, Hendrik Sattler wrote:[color=blue]
> Am Montag, 26. September 2005 14:07 schrieb Randy Kramer:[color=green]
> > Is there / will there ever be a fix for this?[/color]
>
> Sure, rename your inbox to some other folder and let kmail recreate it as
> maildir. maildir does not has such problems because every e-mail is a
> seperate file. Speed then depends on the used filesystem (e.g. for lots of
> small files if you only have short text mails). Maildir is usually faster
> for mailboxes with lots of (big) attachments.
>
> HS[/color]
I'm having/had a similar problem (hasn't happened recently, but I'm still
concerned about it), but I use Maildir locally in KMail. It's an IMAP
account, and the IMAP server (which I also run) is also using Maildir.
There's no mbox anywhere, AFAIK. Why would I be having similar "nullified"
email problems, if mbox is the culprit?
--
Larry Garfield AIM: LOLG42
[email]larry@garfieldtech.com[/email] ICQ: 6817012
"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
Randy Kramer wrote:
[color=blue]
> I suppose maildir will be OK (and maybe even better) for my inbox, which I
> generally keep "trimmed" (not too many emails).
>
> I don't think I want to do that for my mail folders which often have a lot
> (thousands) of archived emails (usually short).
>
> Is it the general consensus that mbox is more subject to corruption than
> maildir?[/color]
By definition.
Why do you think Maildir would perform worse for folders with thousands of
emails? Everything I've read suggests it will perform better - and more
reliably.
--
derek
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
On Tuesday 27 September 2005 15:27, Derek Broughton wrote:[color=blue]
> Randy Kramer wrote:[color=green]
> > I suppose maildir will be OK (and maybe even better) for my inbox,
> > which I generally keep "trimmed" (not too many emails).
> >
> > I don't think I want to do that for my mail folders which often
> > have a lot (thousands) of archived emails (usually short).
> >
> > Is it the general consensus that mbox is more subject to corruption
> > than maildir?[/color]
>
> By definition.
>
> Why do you think Maildir would perform worse for folders with
> thousands of emails? Everything I've read suggests it will perform
> better - and more reliably.[/color]
Because you need a file system which handles small files efficiently
(i.e. ReiseFS) otherwise you waste much space on your hard drive.
Cheers,
Andr
On Tuesday 27 September 2005 09:27 am, Derek Broughton wrote:[color=blue]
> Randy Kramer wrote:[color=green]
> > I suppose maildir will be OK (and maybe even better) for my inbox, which
> > I generally keep "trimmed" (not too many emails).
> >
> > I don't think I want to do that for my mail folders which often have a
> > lot (thousands) of archived emails (usually short).
> >
> > Is it the general consensus that mbox is more subject to corruption than
> > maildir?[/color]
>
> By definition.
>
> Why do you think Maildir would perform worse for folders with thousands of
> emails? Everything I've read suggests it will perform better - and more
> reliably.[/color]
Thanks for asking!
I may have to exorcise some old MS Dos/Windows demons from my thinking.
First a quick (but dumb, I should look it up) question. Does Linux do the
thing that Dos/Windows does (used to do?) of each file requiring a minimum
space (one cluster?), or does it vary by filesystem?
Attempting to answer my own question: Presumably (V)FAT(16,32) must be the
same as MS for compatibility. I don't have any idea, though, about ext2 and
ext3, and I'm guessing (and may have read) that one of the ways Reiser's
achieves its greater efficiency for small files is by not doing that?
Then, I can remember (again, back in my dos/Windows days) two problems that I
may mix up a little bit. I guess the first was the limitation on the number
of files on a disk based on the size of the (primary?) FAT, which was, iirc,
overcome by allowing secondary or virtual FATs (or something along those
lines). I'm sure that's not a problem in Linux.
The 2nd problem, referenced above--I did run into applications where the
number of files in a directory was so large that the access time for a file
became unacceptable because (I guess) of the time required to search the FAT?
(Aside: I can't remember when that number became a problem in dos/Windows, but
I'm sure it was before 10's of thousands of files in a directory.)
I don't know if Linux can run into the same problem. It seems that an indexed
file (e.g., mbox with index) is an alternative to that, ahh, but I guess only
if the index can be searched very efficiently.
How is a search for a file name done in Linux--is it a linear type thing?
(Without being very conversant in big O notation, I'm trying to ask if it's
proportional to the number of entries (file names) in the directory (would
that be O[n]?), or does it do something more clever (would that be O[1}?).
Or, again, does it depend on the file system, (and maybe Reiser's has that
(potential) problem licked?)
Finally (for now ;-), without fully understanding inodes, I guess I'd be
worried about running out (or needing to allocate an excessive number in
advance) to allow for super large quantities of files.
Or, I guess one might say all of these problems might be controlled by proper
administration/tuning of the system, which might include picking an
appropriate type of filesystem for the requirements. Still, I was hoping to
get a little more insight by mentioning these points for discussion.
Thanks!
Randy Kramer
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
Am Dienstag, 27. September 2005 16:48 schrieb Randy Kramer:[color=blue]
> First a quick (but dumb, I should look it up) question. Does Linux do the
> thing that Dos/Windows does (used to do?) of each file requiring a minimum
> space (one cluster?), or does it vary by filesystem?[/color]
That never had anything to do with DOS or Linux, it's always a matter of the
filesystem. ReiserFS can handle that, all others (e.g. XFS, ext3) don't.
However, that's not reason for me to use ReiserFS, hard drives are big enough
these days (this may be different for REALLY lots of small files like a nntp
spool).
[color=blue]
> Then, I can remember (again, back in my dos/Windows days) two problems that
> I may mix up a little bit. I guess the first was the limitation on the
> number of files on a disk based on the size of the (primary?) FAT, which
> was, iirc, overcome by allowing secondary or virtual FATs (or something
> along those lines). I'm sure that's not a problem in Linux.[/color]
Again, do not stick to OS thinkings but stay with the filesystem, please!
ext2/ext3 do have an inode limit (you can define that at FS creation time),
XFS and ReiserFS don't.
[color=blue]
> The 2nd problem, referenced above--I did run into applications where the
> number of files in a directory was so large that the access time for a file
> became unacceptable because (I guess) of the time required to search the
> FAT?[/color]
Maybe, again you win XFS and ReiserFS, ext[23] also need quite some time for
large directories (there is caching support in ext3 with later kernels, IIRC,
that speeds that up).
[color=blue]
> How is a search for a file name done in Linux--is it a linear type thing?
> (Without being very conversant in big O notation, I'm trying to ask if it's
> proportional to the number of entries (file names) in the directory (would
> that be O[n]?), or does it do something more clever (would that be O[1}?).
> Or, again, does it depend on the file system, (and maybe Reiser's has that
> (potential) problem licked?)[/color]
It's again dependent on the filesystem.
HS
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
Randy Kramer wrote:
[color=blue]
> On Tuesday 27 September 2005 09:27 am, Derek Broughton wrote:[color=green]
>>
>> Why do you think Maildir would perform worse for folders with thousands
>> of
>> emails? Everything I've read suggests it will perform better - and more
>> reliably.[/color]
>
> First a quick (but dumb, I should look it up) question. Does Linux do the
> thing that Dos/Windows does (used to do?) of each file requiring a minimum
> space (one cluster?), or does it vary by filesystem?[/color]
It varies decidedly between the different filesystems.[color=blue]
>
> Attempting to answer my own question: Presumably (V)FAT(16,32) must be
> the same as MS for compatibility.[/color]
Yes.
[color=blue]
> I don't have any idea, though, about ext2 and ext3,[/color]
Ext3 is just ext2 with journalling. You can even mount an ext3 partition as
ext2. That said, I've never pried into the internals but it does store
files more efficiently than a FAT system.
[color=blue]
> Then, I can remember (again, back in my dos/Windows days) two problems
> that I
> may mix up a little bit. I guess the first was the limitation on the
> number of files on a disk based on the size of the (primary?) FAT, which
> was, iirc, overcome by allowing secondary or virtual FATs (or something
> along those
> lines). I'm sure that's not a problem in Linux.[/color]
Not in a long time.[color=blue]
>
> The 2nd problem, referenced above--I did run into applications where the
> number of files in a directory was so large that the access time for a
> file became unacceptable because (I guess) of the time required to search
> the FAT?
>
> I don't know if Linux can run into the same problem.[/color]
Again, I haven't looked into the internals but by all reports it isn't an
issue where the number of files is on the order of "thousands". Actually,
since I _do_ know how FAT works, I don't see why a properly written
application should have that much difficulty finding a file in a FAT
directory either.
[color=blue]
> It seems that an
> indexed file (e.g., mbox with index) is an alternative to that, ahh, but I
> guess only if the index can be searched very efficiently.[/color]
And only if you can guarantee that the index is in sync with the file - per
Reinhold's email.
--
derek
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
Andr Wbbeking wrote:
[color=blue]
> On Tuesday 27 September 2005 15:27, Derek Broughton wrote:[color=green]
>> Randy Kramer wrote:[color=darkred]
>> > I suppose maildir will be OK (and maybe even better) for my inbox,
>> > which I generally keep "trimmed" (not too many emails).
>> >
>> > I don't think I want to do that for my mail folders which often
>> > have a lot (thousands) of archived emails (usually short).
>> >
>> > Is it the general consensus that mbox is more subject to corruption
>> > than maildir?[/color]
>>
>> By definition.
>>
>> Why do you think Maildir would perform worse for folders with
>> thousands of emails? Everything I've read suggests it will perform
>> better - and more reliably.[/color]
>
> Because you need a file system which handles small files efficiently
> (i.e. ReiseFS) otherwise you waste much space on your hard drive.[/color]
Space is cheap :-) It's still going to be _faster_ to use maildir than
mbox. Of course, all my partitions are Reiser, anyway.
--
derek
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]
On Tuesday 27 September 2005 01:44 pm, Pete Jewell wrote:[color=blue]
> However, ReiserFS is *much* more efficient when you have thousands of
> files in one directory, because it uses a hashing algorithm to determine
> where the required file is (or starts) in the filesystem. This is
> something I know about (hashing) based on my experience with Pick
> database systems, which also use hashing and are incredibly fast at
> keyed record retrieval (as well as entire file/table traversal).
>
> I've used ReiserFS in the past mainly for it's journalling capability,
> which at the time was more complete than ext3's (this was on a RH6.2
> system with the 2.4.x series kernels). As my customers at the time were
> very likely to simply switch the system off (for any reason, including
> not knowing how a particular application works that they'd wandered
> into), this feature saw a lot of (successful) use![/color]
Thanks to Derek, Hendrik, and Pete for the replies!
Is there a chance that the hash for a ReiserFS can become corrupted like the
index for a mbox file can be? Or maybe I should ask it differently, because
presumably something can happen to make it corrupted--does Reisers have some
better error detection / correction / recovery for the hash than is typical
of an index for an mbox file?
(Maybe I need to go read up on Reiser, and join a Reiser list. ;-)
Randy Kramer
--
To UNSUBSCRIBE, email to [email]debian-kde-REQUEST@lists.debian.org[/email]
with a subject of "unsubscribe". Trouble? Contact [email]listmaster@lists.debian.org[/email]