slow newsgroup access - Hardware

This is a discussion on slow newsgroup access - Hardware ; I run my own news server with leafnode. As Usenet has gotten busier, I now have over 74K files in /var/spool/news/alt/fan/cecil-adams . leafnode doesn't deal with this well at all; a.f.c-a takes ~1:50 to open, during which time leafnode uses ...

+ Reply to Thread
Results 1 to 19 of 19

Thread: slow newsgroup access

  1. slow newsgroup access

    I run my own news server with leafnode. As Usenet has gotten busier, I
    now have over 74K files in /var/spool/news/alt/fan/cecil-adams .
    leafnode doesn't deal with this well at all; a.f.c-a takes ~1:50 to
    open, during which time leafnode uses 97%+ (of one CPU, it's
    single-threaded) and trn just sits there; the progress meter then takes
    a second or two to go from 0 to 100%. I already have the news spool
    directory (ext3) mounted noatime. Is there a faster filesystem, or some
    option I can give to leafnode or put in fstab, or is it that leafnode
    just doesn't scale well and I should bite the bullet and lower the
    retention time?

    --
    The powers in charge keep us in a continuous stampede of patriotic
    fervor with the cry of national emergency. Always there has been some
    terrible evil to gobble us up if we did not furnish the sums demanded.
    Yet these disasters seem never to have been quite real. -- D. MacArthur

  2. Re: slow newsgroup access

    I demand that Hactar may or may not have written...

    > I run my own news server with leafnode. As Usenet has gotten busier, I now
    > have over 74K files in /var/spool/news/alt/fan/cecil-adams . leafnode
    > doesn't deal with this well at all; a.f.c-a takes ~1:50 to open, during
    > which time leafnode uses 97%+ (of one CPU, it's single-threaded) and trn
    > just sits there; the progress meter then takes a second or two to go from 0
    > to 100%. I already have the news spool directory (ext3) mounted noatime.


    Does that file system have the "dir_index" flag set?

    If not, unmount it, set the flag (tune2fs -O dir_index) then run "e2fsck -D"
    on it (the directories need to be re-indexed before you do anything else with
    that partition); it'll now be safe to re-mount.

    If it already does have that flag set or you don't notice much difference,
    then...

    > Is there a faster filesystem, or some option I can give to leafnode or put
    > in fstab, or is it that leafnode just doesn't scale well and I should bite
    > the bullet and lower the retention time?


    .... you could *try* ext4, I suppose, but AFAIK it uses the same b-tree scheme
    as ext3 (with dir_index). Also, it's still somewhat work-in-progress, or at
    least marked "experimental". As for the rest, I'll leave that to others. :-)

    --
    | Darren Salt | linux or ds at | nr. Ashington, | Toon
    | RISC OS, Linux | youmustbejoking,demon,co,uk | Northumberland | Army
    | + Generate power using sun, wind, water, nuclear. FORGET COAL AND OIL.

    Clones are people two.

  3. Re: slow newsgroup access

    In article <4F73082A88%news@youmustbejoking.demon.cu.invalid>,
    Darren Salt wrote:
    > I demand that Hactar may or may not have written...
    >
    > > I run my own news server with leafnode. As Usenet has gotten busier, I now
    > > have over 74K files in /var/spool/news/alt/fan/cecil-adams . leafnode
    > > doesn't deal with this well at all; a.f.c-a takes ~1:50 to open, during
    > > which time leafnode uses 97%+ (of one CPU, it's single-threaded) and trn
    > > just sits there; the progress meter then takes a second or two to go from 0
    > > to 100%. I already have the news spool directory (ext3) mounted noatime.

    >
    > Does that file system have the "dir_index" flag set?


    How do I tell?

    > > Is there a faster filesystem, or some option I can give to leafnode or put
    > > in fstab, or is it that leafnode just doesn't scale well and I should bite
    > > the bullet and lower the retention time?

    >
    > ... you could *try* ext4, I suppose, but AFAIK it uses the same b-tree scheme
    > as ext3 (with dir_index). Also, it's still somewhat work-in-progress, or at
    > least marked "experimental". As for the rest, I'll leave that to others. :-)


    Someone recommended reiserfs, but I'm wary of reiserfs after reading
    http://linuxmafia.com/faq/Filesystems/reiserfs.html or something
    similar.

    --
    -eben QebWenE01R@vTerYizUonI.nOetP http://royalty.mine.nu:81

    Hanlon's Razor: "Never attribute to malice that which can be
    adequately explained by stupidity." Derived from Robert Heinlein

  4. Re: slow newsgroup access

    On 2008-02-07, Hactar wrote:

    > I run my own news server with leafnode. As Usenet has gotten busier, I
    > now have over 74K files in /var/spool/news/alt/fan/cecil-adams .
    > leafnode doesn't deal with this well at all; a.f.c-a takes ~1:50 to
    > open, during which time leafnode uses 97%+ (of one CPU, it's
    > single-threaded) and trn just sits there; the progress meter then takes
    > a second or two to go from 0 to 100%. I already have the news spool
    > directory (ext3) mounted noatime. Is there a faster filesystem, or some
    > option I can give to leafnode or put in fstab, or is it that leafnode
    > just doesn't scale well and I should bite the bullet and lower the
    > retention time?


    I have a 72K group on an office news server, without any noticeable
    sluggyness to open the list of new posts.

    Even my home server (a lowly P-II 350 MHz) has no problem with such numbers.


    Are you sure you're not running a very expensive scoring, in your newsreader?



    --
    There is an art, it says, or rather, a knack to flying.
    The knack lies in learning how to throw yourself at the ground and miss.
    Douglas Adams

  5. Re: slow newsgroup access

    Hactar staggered into the Black Sun and said:
    > Darren Salt wrote:
    >> I demand that Hactar may or may not have written...
    >> > I now have over 74K files in
    >> > /var/spool/news/alt/fan/cecil-adams .


    Time to run "leafnode --expire"?

    >> > this well at all; a.f.c-a takes ~1:50 to open, during which time
    >> > leafnode uses 97%+ (of one CPU, it's single-threaded) and trn just
    >> > sits there

    >> Does that file system have the "dir_index" flag set?

    > How do I tell?


    "dumpe2fs -h /dev/whatever | grep index". It's set by default on most
    things that have been made in the last few years.

    >> > Is there a faster filesystem, or some option I can give to leafnode
    >> > or put in fstab, or is it that leafnode just doesn't scale well and
    >> > I should bite the bullet and lower the retention time?

    >> ... you could *try* ext4, I suppose, but AFAIK it uses the same
    >> b-tree scheme as ext3 (with dir_index). Also, it's still somewhat
    >> work-in-progress, or at least marked "experimental".

    > Someone recommended reiserfs, but I'm wary of reiserfs


    I've never had any problems with ReiserFS, but others have. Using ext4
    probably wouldn't work well until it's out of beta. Yeah, lowering the
    retention time would probably be the best bet here. Here, slrn gets a
    bit slow on col.misc when I haven't run slrnpull --expire for 2 or 3
    months, so it isn't really a client-specific thing.

    --
    I think I'll have to put on 500 pounds of subwoofers, amps, and other
    delicious herbs. --MegaHAL, trained on ASR
    Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

  6. Re: slow newsgroup access

    On 2008-02-08, Rikishi 42 claimed:
    > On 2008-02-07, Hactar wrote:
    >
    >> I run my own news server with leafnode. As Usenet has gotten busier, I
    >> now have over 74K files in /var/spool/news/alt/fan/cecil-adams .
    >> leafnode doesn't deal with this well at all; a.f.c-a takes ~1:50 to
    >> open, during which time leafnode uses 97%+ (of one CPU, it's
    >> single-threaded) and trn just sits there; the progress meter then takes
    >> a second or two to go from 0 to 100%. I already have the news spool
    >> directory (ext3) mounted noatime. Is there a faster filesystem, or some
    >> option I can give to leafnode or put in fstab, or is it that leafnode
    >> just doesn't scale well and I should bite the bullet and lower the
    >> retention time?

    >
    > I have a 72K group on an office news server, without any noticeable
    > sluggyness to open the list of new posts.
    >
    > Even my home server (a lowly P-II 350 MHz) has no problem with such numbers.
    >
    >
    > Are you sure you're not running a very expensive scoring, in your newsreader?


    I have a leafnode spool on this machine (2.8G) and another on one in
    the basement (500MHz).

    The spool on this machine is over 12M. It doesn't go through a lot of
    filtering.

    The spool in the basement is 398M. It goes through a tremendous set of
    filters after it travels over the network to this machine.

    I don't have delays that slow from either spool. Even going back for an
    entire newsgroup on the bigger spool (around 370M of that 398M) doesn't
    take much longer than what he's seeing. In neither case does the CPU
    ever get that high just from reading any group. Max I've ever seen was
    less than 50%,

    --
    Conformity constipates creativity!

  7. Re: slow newsgroup access

    In article ,
    Dances With Crows wrote:
    > Hactar staggered into the Black Sun and said:
    > > Darren Salt wrote:
    > >> I demand that Hactar may or may not have written...
    > >> > I now have over 74K files in
    > >> > /var/spool/news/alt/fan/cecil-adams .

    >
    > Time to run "leafnode --expire"?


    Tried that, no change. I must have it cronned.

    > >> > this well at all; a.f.c-a takes ~1:50 to open, during which time
    > >> > leafnode uses 97%+ (of one CPU, it's single-threaded) and trn just
    > >> > sits there
    > >> Does that file system have the "dir_index" flag set?

    > > How do I tell?

    >
    > "dumpe2fs -h /dev/whatever | grep index". It's set by default on most
    > things that have been made in the last few years.


    Hmm, sure about that? That gives me nothing, as does "... | grep -i ...".
    The full dumpe2fs output is:

    dumpe2fs 1.38 (30-Jun-2005)
    Filesystem volume name: news_spool
    Last mounted on:
    Filesystem UUID: 2ec53801-197a-49af-aa91-d5742809a91e
    Filesystem magic number: 0xEF53
    Filesystem revision #: 1 (dynamic)
    Filesystem features: has_journal filetype sparse_super
    Default mount options: (none)
    Filesystem state: clean
    Errors behavior: Continue
    Filesystem OS type: Linux
    Inode count: 1004896
    Block count: 1004062
    Reserved block count: 20081
    Free blocks: 821125
    Free inodes: 898331
    First block: 0
    Block size: 4096
    Fragment size: 4096
    Blocks per group: 32768
    Fragments per group: 32768
    Inodes per group: 32416
    Inode blocks per group: 1013
    Filesystem created: Wed Feb 21 21:20:38 2007
    Last mount time: Thu Feb 7 23:49:48 2008
    Last write time: Thu Feb 7 23:50:47 2008
    Mount count: 9
    Maximum mount count: 27
    Last checked: Fri Sep 21 18:42:27 2007
    Check interval: 15552000 (6 months)
    Next check after: Wed Mar 19 17:42:27 2008
    Reserved blocks uid: 0 (user root)
    Reserved blocks gid: 0 (group root)
    First inode: 11
    Inode size: 128
    Journal inode: 8
    Default directory hash: tea
    Directory Hash Seed: 7e90a81b-2f5a-4710-badd-cfb377ca82de
    Journal backup: inode blocks

    Unless ... you're talking about the "features" line, in which case "no
    it's not set".

    I'm not crazy (at least in this instance), it's ext3:

    /dev/hda10 on /var/spool/news type ext3 (rw,noatime)

    --
    -eben QebWenE01R@vTerYizUonI.nOetP royalty.mine.nu:81

    Unix is user-friendly; it's just picky
    about who it makes friends with.

  8. Re: slow newsgroup access

    In article ,
    Hactar wrote:
    > In article ,
    > Dances With Crows wrote:
    > > Hactar staggered into the Black Sun and said:
    > > > Darren Salt wrote:
    > > >> I demand that Hactar may or may not have written...

    >
    > > >> > this well at all; a.f.c-a takes ~1:50 to open, during which
    > > >> > time leafnode uses 97%+ (of one CPU, it's single-threaded) and trn
    > > >> > just sits there
    > > >> Does that file system have the "dir_index" flag set?
    > > > How do I tell?

    > >
    > > "dumpe2fs -h /dev/whatever | grep index". It's set by default on
    > > most things that have been made in the last few years.


    ....

    > Unless ... you're talking about the "features" line, in which case "no
    > it's not set".


    OK, did this:

    root@pc:/var/spool# umount news && tune2fs -O dir_index /dev/hda10 &&
    e2fsck -D /dev/hda10 && mount news
    tune2fs 1.38 (30-Jun-2005)
    e2fsck 1.38 (30-Jun-2005)
    news_spool: clean, 106603/1004896 files, 182982/1004062 blocks

    Took less than a second to run. No significant change in newsgroup-opening
    time.

    Hmm, now dumpe2fs says:

    dumpe2fs 1.38 (30-Jun-2005)
    Filesystem features: has_journal dir_index filetype needs_recovery sparse_super

    How did that happen? "needs_recovery" is new.

    --
    -eben QebWenE01R@vTerYizUonI.nOetP royalty.mine.nu:81

    Only two things are infinite, the universe and human stupidity,
    and I'm not sure about the former." -- Albert Einstein

  9. Re: slow newsgroup access

    Rikishi 42 wrote:
    > Even my home server (a lowly P-II 350 MHz) has no problem with such numbers.


    Yeah no problems here either (I'm using a Pentium 120Mhz).

    Regards,

    Mark.

    --
    Mark Hobley,
    393 Quinton Road West,
    Quinton, BIRMINGHAM.
    B32 1QE.

  10. Re: slow newsgroup access

    Hactar wrote:
    > I run my own news server with leafnode. As Usenet has gotten busier, I
    > now have over 74K files in /var/spool/news/alt/fan/cecil-adams .
    > leafnode doesn't deal with this well at all [...]


    If you're still using version 1 of leafnode, you might want to consider
    upgrading. (TBH I'm not sure whether it will help, as I'm clearly in
    a different situation - my largest spool directory contains only 2200
    files.)

    Chris

  11. Re: slow newsgroup access

    Hactar staggered into the Black Sun and said:
    > Hactar wrote:
    >> Dances With Crows wrote:
    >> > "dumpe2fs -h /dev/whatever | grep index". It's set by default on
    >> > most things that have been made in the last few years.

    > root@pc:/var/spool# umount news && tune2fs -O dir_index /dev/hda10 &&
    > e2fsck -D /dev/hda10 && mount news
    >
    > Took less than a second to run. No significant change in
    > newsgroup-opening time.


    Hm. I'd have to look at the internals, but it may be that the index
    is only created when a file is created in a dir. So previously existing
    files will have no index, and new files will have an index based on the
    dir hash. I suppose you could test this by cp'ing the files to
    somewhere else, removing the old files in the spool, then cp'ing them
    back again.

    > dumpe2fs 1.38 (30-Jun-2005)
    > Filesystem features: has_journal dir_index filetype
    > needs_recovery sparse_super
    > How did that happen? "needs_recovery" is new.


    If a filesystem hasn't been umounted or mounted ro, it'll have its
    needs_recovery flag set. This is a Feature, so filesystems can have
    their journals replayed (or get fscked) if they're not cleanly umounted.

    --
    Kimono dragon: found mainly on Pacific islands and atolls, this is the
    more dangerous version of the somewhat placid bathrobe gecko.
    --jghanc
    Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

  12. Re: slow newsgroup access

    On 2008-02-08, Chris Davies wrote:
    >
    >
    > Hactar wrote:
    >> I run my own news server with leafnode. As Usenet has gotten busier, I
    >> now have over 74K files in /var/spool/news/alt/fan/cecil-adams .
    >> leafnode doesn't deal with this well at all [...]

    >
    > If you're still using version 1 of leafnode, you might want to consider
    > upgrading. (TBH I'm not sure whether it will help, as I'm clearly in
    > a different situation - my largest spool directory contains only 2200
    > files.)


    My server runs 1.x, to. No problem in speed.

    --
    There is an art, it says, or rather, a knack to flying.
    The knack lies in learning how to throw yourself at the ground and miss.
    Douglas Adams

  13. Re: slow newsgroup access

    In article ,
    Dances With Crows wrote:
    > Hactar staggered into the Black Sun and said:
    > > Hactar wrote:
    > >> Dances With Crows wrote:
    > >> > "dumpe2fs -h /dev/whatever | grep index". It's set by default on
    > >> > most things that have been made in the last few years.

    > > root@pc:/var/spool# umount news && tune2fs -O dir_index /dev/hda10 &&
    > > e2fsck -D /dev/hda10 && mount news
    > >
    > > Took less than a second to run. No significant change in
    > > newsgroup-opening time.

    >
    > Hm. I'd have to look at the internals, but it may be that the index
    > is only created when a file is created in a dir. So previously existing
    > files will have no index, and new files will have an index based on the
    > dir hash. I suppose you could test this by cp'ing the files to
    > somewhere else, removing the old files in the spool, then cp'ing them
    > back again.


    I did:

    root@pc:/etc/news/leafnode# tar cf /tmp/v.s.n.tar
    root@pc:/var/spool/news# tar xf /tmp/v.s.n.tar

    I should have checked whether they got new inodes. Hang on:

    eben@pc:/tmp$ echo x > foo
    eben@pc:/tmp$ ls -i foo
    243003 4.0K foo
    eben@pc:/tmp$ cp foo bar
    eben@pc:/tmp$ tar cf baz.tar bar
    eben@pc:/tmp$ tar xf baz.tar
    eben@pc:/tmp$ ls -i foo
    243003 4.0K foo

    Nope, doesn't appear so. Probably the easiest way ito force a new inode is
    to throw in a mkfs between the two tars. I'll do it at night when Usenet is
    less busy.

    > > dumpe2fs 1.38 (30-Jun-2005)
    > > Filesystem features: has_journal dir_index filetype
    > > needs_recovery sparse_super
    > > How did that happen? "needs_recovery" is new.

    >
    > If a filesystem hasn't been umounted or mounted ro, it'll have its
    > needs_recovery flag set. This is a Feature, so filesystems can have
    > their journals replayed (or get fscked) if they're not cleanly umounted.


    Odd, I didn't unmount it uncleanly. Ah well, a restart has happened
    between now and then. Lemme see:

    dumpe2fs 1.38 (30-Jun-2005)
    Filesystem features: has_journal dir_index filetype needs_recovery sparse_super

    Nope, same.

    --
    -eben QebWenE01R@vTerYizUonI.nOetP http://royalty.mine.nu:81
    An ASCII character walks into a bar and orders a double. "Having a bad
    day?" asks the barman. "Yeah, I have a parity error," replies the ASCII
    chrcter. The barman says, "Yeah, I thought you looked a bit off." - Skud

  14. Re: slow newsgroup access

    I demand that Hactar may or may not have written...

    [snip]
    > OK, did this:


    > root@pc:/var/spool# umount news && tune2fs -O dir_index /dev/hda10 &&
    > e2fsck -D /dev/hda10 && mount news
    > tune2fs 1.38 (30-Jun-2005)
    > e2fsck 1.38 (30-Jun-2005)
    > news_spool: clean, 106603/1004896 files, 182982/1004062 blocks


    > Took less than a second to run. No significant change in newsgroup-opening
    > time.


    That *was* a bit quick...
    # umount /dev/hda10 && e2fsck -f -D /dev/hda10 && mount /dev/hda10

    (tune2fs's man page says that "e2fsck -D" can be run to convert the
    directories. It doesn't say "must be run" or "may need the 'force' flag".)

    > Hmm, now dumpe2fs says:


    > dumpe2fs 1.38 (30-Jun-2005)
    > Filesystem features: has_journal dir_index filetype needs_recovery

    sparse_super

    > How did that happen? "needs_recovery" is new.


    Was the partition mounted at the time? If so, and no problems occur, that
    flag will be cleared when the partition is unmounted.

    --
    | Darren Salt | linux or ds at | nr. Ashington, | Toon
    | RISC OS, Linux | youmustbejoking,demon,co,uk | Northumberland | Army
    | + Output less CO2 => avoid boiling weather. TIME IS RUNNING OUT *FAST*.

    Confucius say: He who speak with forked tongue, not need chopsticks.

  15. Re: slow newsgroup access

    In article ,
    Hactar wrote:
    > In article ,
    > Hactar wrote:
    > > In article ,
    > > Dances With Crows wrote:
    > > > Hactar staggered into the Black Sun and said:
    > > > > Darren Salt wrote:
    > > > >> I demand that Hactar may or may not have written...

    > >
    > > > >> > this well at all; a.f.c-a takes ~1:50 to open, during which
    > > > >> > time leafnode uses 97%+ (of one CPU, it's single-threaded) and trn
    > > > >> > just sits there
    > > > >> Does that file system have the "dir_index" flag set?
    > > > > How do I tell?
    > > >
    > > > "dumpe2fs -h /dev/whatever | grep index". It's set by default on
    > > > most things that have been made in the last few years.

    >
    > ...
    >
    > > Unless ... you're talking about the "features" line, in which case "no
    > > it's not set".

    >
    > OK, did this:
    >
    > root@pc:/var/spool# umount news && tune2fs -O dir_index /dev/hda10 &&
    > e2fsck -D /dev/hda10 && mount news
    > tune2fs 1.38 (30-Jun-2005)
    > e2fsck 1.38 (30-Jun-2005)
    > news_spool: clean, 106603/1004896 files, 182982/1004062 blocks
    >
    > Took less than a second to run. No significant change in newsgroup-opening
    > time.


    Something changed. Don't know whether it was a cache exiring or the reboot
    that did the trick, but now it takes a second or two to open that
    newsgroup. But it was probably related to dir_index.

    --
    -eben QebWenE01R@vTerYizUonI.nOetP http://royalty.mine.nu:81
    Are you confident that you appear to be professional in your electronic
    communication? Consider this: A: No
    Q: Can I top post? from nick@xx.co.uk

  16. Re: slow newsgroup access

    In article ,
    Hactar wrote:
    > In article ,
    > Hactar wrote:
    > > In article ,
    > > Hactar wrote:
    > > >
    > > > Unless ... you're talking about the "features" line, in which case "no
    > > > it's not set".

    > >
    > > OK, did this:
    > >
    > > root@pc:/var/spool# umount news && tune2fs -O dir_index /dev/hda10 &&
    > > e2fsck -D /dev/hda10 && mount news
    > > tune2fs 1.38 (30-Jun-2005)
    > > e2fsck 1.38 (30-Jun-2005)
    > > news_spool: clean, 106603/1004896 files, 182982/1004062 blocks
    > >
    > > Took less than a second to run. No significant change in newsgroup-opening
    > > time.

    >
    > Something changed. Don't know whether it was a cache exiring or the reboot
    > that did the trick, but now it takes a second or two to open that
    > newsgroup. But it was probably related to dir_index.


    I take that back. It's always fast when "suck" hasn't run between
    invocations of leafnode (at least, I think that's the criterion). Bah.
    I guess I'll drop retention.

    --
    The powers in charge keep us in a continuous stampede of patriotic
    fervor with the cry of national emergency. Always there has been some
    terrible evil to gobble us up if we did not furnish the sums demanded.
    Yet these disasters seem never to have been quite real. -- D. MacArthur

  17. Re: slow newsgroup access

    Hactar wrote:
    > In article ,
    > Hactar wrote:
    >> In article ,
    >> Hactar wrote:
    >>> In article ,
    >>> Hactar wrote:
    >>>> Unless ... you're talking about the "features" line, in which case "no
    >>>> it's not set".
    >>> OK, did this:
    >>>
    >>> root@pc:/var/spool# umount news && tune2fs -O dir_index /dev/hda10 &&
    >>> e2fsck -D /dev/hda10 && mount news
    >>> tune2fs 1.38 (30-Jun-2005)
    >>> e2fsck 1.38 (30-Jun-2005)
    >>> news_spool: clean, 106603/1004896 files, 182982/1004062 blocks
    >>>
    >>> Took less than a second to run. No significant change in newsgroup-opening
    >>> time.

    >> Something changed. Don't know whether it was a cache exiring or the reboot
    >> that did the trick, but now it takes a second or two to open that
    >> newsgroup. But it was probably related to dir_index.

    >
    > I take that back. It's always fast when "suck" hasn't run between
    > invocations of leafnode (at least, I think that's the criterion). Bah.
    > I guess I'll drop retention.
    >


    Which makes it more likely to be some sort of misconfiguration between
    your news client and leafnode.
    Fwiw, I'm running leafnode on a P166mmx with about 20 newsgroups and a
    lot of filtering which results in a 37 Mb spool and no delay at all.
    Fetchnews runs every ten minutes as a cron job, texpire once a day.
    It makes me wonder, how did you configure leafnode?
    You're not restarting it every time are you?
    I'm not sure but I think that could cause leafnode to renumber the
    messages and that might cause your news client to reload all messages.

    Michel.

  18. Re: slow newsgroup access

    In article ,
    dillinger wrote:
    > Hactar wrote:
    > > In article ,
    > > Hactar wrote:
    > >> In article ,
    > >> Hactar wrote:
    > >>> In article ,
    > >>> Hactar wrote:
    > >>>> Unless ... you're talking about the "features" line, in which case "no
    > >>>> it's not set".
    > >>> OK, did this:
    > >>>
    > >>> root@pc:/var/spool# umount news && tune2fs -O dir_index /dev/hda10 &&
    > >>> e2fsck -D /dev/hda10 && mount news
    > >>> tune2fs 1.38 (30-Jun-2005)
    > >>> e2fsck 1.38 (30-Jun-2005)
    > >>> news_spool: clean, 106603/1004896 files, 182982/1004062 blocks
    > >>>
    > >>> Took less than a second to run. No significant change in newsgroup-opening
    > >>> time.
    > >> Something changed. Don't know whether it was a cache exiring or the reboot
    > >> that did the trick, but now it takes a second or two to open that
    > >> newsgroup. But it was probably related to dir_index.

    > >
    > > I take that back. It's always fast when "suck"


    Not suck, fetchnews.

    > > hasn't run between invocations of leafnode (at least, I think that's the
    > > criterion). Bah. I guess I'll drop retention.

    >
    > Which makes it more likely to be some sort of misconfiguration between
    > your news client and leafnode.
    > Fwiw, I'm running leafnode on a P166mmx with about 20 newsgroups and a
    > lot of filtering which results in a 37 Mb spool and no delay at all.
    > Fetchnews runs every ten minutes as a cron job, texpire once a day.
    > It makes me wonder, how did you configure leafnode?


    root@pc:~# grep -v -e '^$' -e '^#' /etc/news/leafnode/config
    expire = 90
    server = news.$ISP
    initialfetch = 500
    debugmode = 0
    username = $USER
    password = $PASS
    groupexpire comp.risks = 200
    maxcrosspost = 5
    hostname = royalty.mine.nu

    > You're not restarting it every time are you?


    It's running from (x)inetd, so I guess so. I did find 38K articles not
    in comp.risks and older than 90 days, (the oldest from August 2006) and 34K
    of those are in a.f.c-a (45% of that newsgroup's count). Can I just delete
    them, or is there more work to do?

    > I'm not sure but I think that could cause leafnode to renumber the
    > messages and that might cause your news client to reload all messages.


    In that case I would expect some progress from trn's bar graph during
    that operation, and the CPU load to come from trn, not leafnode.

    --
    -eben QebWenE01R@vTerYizUonI.nOetP http://royalty.mine.nu:81
    AQUARIUS: There's travel in your future when your tongue freezes to the
    back of a speeding bus. Fill the void in your pathetic life by playing
    Whack-a-Mole 17 hours a day. -- Weird Al, _Your Horoscope for Today_

  19. Re: slow newsgroup access

    Hactar wrote:
    > In article ,
    > dillinger wrote:
    >> Hactar wrote:
    >>> In article ,
    >>> Hactar wrote:
    >>>> In article ,
    >>>> Hactar wrote:
    >>>>> In article ,
    >>>>> Hactar wrote:
    >>>>>> Unless ... you're talking about the "features" line, in which case "no
    >>>>>> it's not set".
    >>>>> OK, did this:
    >>>>>
    >>>>> root@pc:/var/spool# umount news && tune2fs -O dir_index /dev/hda10 &&
    >>>>> e2fsck -D /dev/hda10 && mount news
    >>>>> tune2fs 1.38 (30-Jun-2005)
    >>>>> e2fsck 1.38 (30-Jun-2005)
    >>>>> news_spool: clean, 106603/1004896 files, 182982/1004062 blocks
    >>>>>
    >>>>> Took less than a second to run. No significant change in newsgroup-opening
    >>>>> time.
    >>>> Something changed. Don't know whether it was a cache exiring or the reboot
    >>>> that did the trick, but now it takes a second or two to open that
    >>>> newsgroup. But it was probably related to dir_index.
    >>> I take that back. It's always fast when "suck"

    >
    > Not suck, fetchnews.
    >
    >>> hasn't run between invocations of leafnode (at least, I think that's the
    >>> criterion). Bah. I guess I'll drop retention.

    >> Which makes it more likely to be some sort of misconfiguration between
    >> your news client and leafnode.
    >> Fwiw, I'm running leafnode on a P166mmx with about 20 newsgroups and a
    >> lot of filtering which results in a 37 Mb spool and no delay at all.
    >> Fetchnews runs every ten minutes as a cron job, texpire once a day.
    >> It makes me wonder, how did you configure leafnode?

    >
    > root@pc:~# grep -v -e '^$' -e '^#' /etc/news/leafnode/config
    > expire = 90
    > server = news.$ISP
    > initialfetch = 500
    > debugmode = 0
    > username = $USER
    > password = $PASS
    > groupexpire comp.risks = 200
    > maxcrosspost = 5
    > hostname = royalty.mine.nu
    >
    >> You're not restarting it every time are you?

    >
    > It's running from (x)inetd, so I guess so.


    Right, stupid question, really, leafnode only runs from inetd.

    > I did find 38K articles not in comp.risks and older than 90 days,
    > (the oldest from August 2006) and 34Kof those are in a.f.c-a
    > (45% of that newsgroup's count). Can I just deletethem, or is there
    > more work to do?


    texpire should take care of those.

    >> I'm not sure but I think that could cause leafnode to renumber the
    >> messages and that might cause your news client to reload all messages.

    >
    > In that case I would expect some progress from trn's bar graph during
    > that operation, and the CPU load to come from trn, not leafnode.


    Seems logical.

    There is one more thing I can think of:
    Did you configure trn to fetch the headers of the messages first, and
    the rest only when you read them? That should save some time on opening
    a group with a lot of new messages.

    Michel

+ Reply to Thread