Hi,

On Fri, Jun 20, 2008 at 03:34:26PM +0300, Krassimir Slavchev wrote:
> # ls -l
> total 1239702
> - -rw------- 1 vscan vscan 4398199488512 Jun 20 15:18 auto-whitelist
> - -rw------- 1 vscan vscan 22 Jun 20 15:18 bayes.lock
> - -rw------- 1 vscan vscan 102168 Jun 20 15:18 bayes_journal
> - -rw------- 1 vscan vscan 1099639861248 Jun 20 15:18 bayes_seen
> - -rw------- 1 vscan vscan 21057536 Jun 20 15:18 bayes_toks
>
> but:
> # df -h
> Filesystem Size Used Avail Capacity Mounted on
> /dev/ar0s1a 19G 3.9G 14G 22% /
> ...
>
> # du
> 1239758 .
>
> # mount
> /dev/ar0s1a on / (ufs, local)
> ...

This is most likely due to the use of 'sparse files': sequential file
data that consists of only zeros doesn't need have actual storage
associated to it. This is quite normal in UNIX environments, and quite
harmless.

--
Rink P.W. Springer - http://rink.nu
"Anyway boys, this is America. Just because you get more votes doesn't
mean you win." - Fox Mulder
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

Rink Springer wrote:
> Hi,
>
> On Fri, Jun 20, 2008 at 03:34:26PM +0300, Krassimir Slavchev wrote:
>> # ls -l
>> total 1239702
>> - -rw------- 1 vscan vscan 4398199488512 Jun 20 15:18 auto-whitelist
>> - -rw------- 1 vscan vscan 22 Jun 20 15:18 bayes.lock
>> - -rw------- 1 vscan vscan 102168 Jun 20 15:18 bayes_journal
>> - -rw------- 1 vscan vscan 1099639861248 Jun 20 15:18 bayes_seen
>> - -rw------- 1 vscan vscan 21057536 Jun 20 15:18 bayes_toks
>>
>> but:
>> # df -h
>> Filesystem Size Used Avail Capacity Mounted on
>> /dev/ar0s1a 19G 3.9G 14G 22% /
>> ...
>>
>> # du
>> 1239758 .
>>
>> # mount
>> /dev/ar0s1a on / (ufs, local)
>> ...
>
> This is most likely due to the use of 'sparse files': sequential file
> data that consists of only zeros doesn't need have actual storage
> associated to it. This is quite normal in UNIX environments, and quite
> harmless.

Except that the file in question should be, judging by the filename, a
simple text file. I don't really see how a whitelist could grow to such
monstrous sizes Most likely it's a file system corruption - fsck
should be the first thing to try.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIW7aOldnAQVacBcgRAouWAKCLxfKAx9767zfE+lIgFn 6rFZ4cawCeLstv
Bkqut78WbRLWkrnMbUUq394=
=lEs9
-----END PGP SIGNATURE-----

On Fri, Jun 20, 2008 at 03:54:22PM +0200, Ivan Voras wrote:
> Except that the file in question should be, judging by the filename, a
> simple text file. I don't really see how a whitelist could grow to such
> monstrous sizes Most likely it's a file system corruption - fsck
> should be the first thing to try.

The 'vscan' user leads me assume this is SpamAssassin - I've seen this
behaviour at work, where our scripts were trying to backup a 1TB file
(which actually was ~vscan/.spamassassin/auto-whitelist). The result was
that the backup script died due to lack of disk space on the backup
server (as we don't use compression).

When I was investigating why the file could be so large it, it turned
out the file was only a few hunderd 'real' MB's, so that is why I assume
this person is having the same issue as we do. The file is a Berkeley DB
file, by the way, so there's nothing textfile about it ;-)

But still, when in doubt, fsck(8) is sure to aid you.

--
Rink P.W. Springer - http://rink.nu
"Anyway boys, this is America. Just because you get more votes doesn't
mean you win." - Fox Mulder
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

Rink Springer wrote:
> On Fri, Jun 20, 2008 at 03:54:22PM +0200, Ivan Voras wrote:
>> Except that the file in question should be, judging by the filename, a
>> simple text file. I don't really see how a whitelist could grow to such
>> monstrous sizes Most likely it's a file system corruption - fsck
>> should be the first thing to try.
>
> The 'vscan' user leads me assume this is SpamAssassin - I've seen this
> behaviour at work, where our scripts were trying to backup a 1TB file
> (which actually was ~vscan/.spamassassin/auto-whitelist). The result was
> that the backup script died due to lack of disk space on the backup
> server (as we don't use compression).
>
> When I was investigating why the file could be so large it, it turned
> out the file was only a few hunderd 'real' MB's, so that is why I assume
> this person is having the same issue as we do. The file is a Berkeley DB
> file, by the way, so there's nothing textfile about it ;-)

I learn something every day
Didn't know BDB was smart enough to create sparse files.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIW73BldnAQVacBcgRAueCAKDvHyIIKXlCeubk7H4Ddy hA76j9SACeLGUc
BdGXkCi8uujS2fHqsRUtajg=
=y1PG
-----END PGP SIGNATURE-----

Ivan Voras wrote:
> Rink Springer wrote:
> > The 'vscan' user leads me assume this is SpamAssassin - I've seen this
> > behaviour at work, where our scripts were trying to backup a 1TB file
> > (which actually was ~vscan/.spamassassin/auto-whitelist). The result was
> > that the backup script died due to lack of disk space on the backup
> > server (as we don't use compression).
> >
> > When I was investigating why the file could be so large it, it turned
> > out the file was only a few hunderd 'real' MB's, so that is why I assume
> > this person is having the same issue as we do. The file is a Berkeley DB
> > file, by the way, so there's nothing textfile about it ;-)
>
> I learn something every day
> Didn't know BDB was smart enough to create sparse files.

BTW, you can use "ls -ls" to display the number of physical
blocks allocated to the file, so you can easily see whether
a file is sparse or not:

$ dd if=/dev/zero of=foo1 bs=1m count=1
$ truncate -s 1m foo2
$ ls -ls foo1 foo2
1040 -rw------- 1 olli olli 1048576 Jun 20 22:43 foo1
32 -rw------- 1 olli olli 1048576 Jun 20 22:43 foo2

As you can see, the file size is the same, but the block
counts are different (I have BLOCKSIZE=K in my environment,
so the blocks are displayed in 1KB units).

I've written a small script that can be used to detect
sparse files (it even displays the "sparseness" percentage):

http://www.secnetix.de/olli/scripts/sparsecheck

Best regards
Oliver

PS: Of course it is still possible that a file system is
corrupt and needs fsck, no matter whether those files are
sparse or not.

--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd

"C++ is over-complicated nonsense. And Bjorn Shoestrap's book
a danger to public health. I tried reading it once, I was in
recovery for months."
-- Cliff Sarginson
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Fri, Jun 20, 2008 at 04:25:05PM +0200, Ivan Voras wrote:

> > When I was investigating why the file could be so large it, it turned
> > out the file was only a few hunderd 'real' MB's, so that is why I assume
> > this person is having the same issue as we do. The file is a Berkeley DB
> > file, by the way, so there's nothing textfile about it ;-)
> I learn something every day
> Didn't know BDB was smart enough to create sparse files.

Even base system uses such files :-)

$ ls -lsk /etc/pwd.db
140 -rw-r--r-- 1 root wheel 159744 21 ÉÀÎ 12:26 /etc/pwd.db
$ file /etc/pwd.db
/etc/pwd.db: Berkeley DB 1.85 (Hash, version 2, native byte-order)

File occupies 140K = 143360 bytes and its length is 159744 bytes,
so it has to be sparse. FreeBSD 4.11-STABLE.

Eugene
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Oliver Fromme wrote:
> Ivan Voras wrote:
> > Rink Springer wrote:
> > > The 'vscan' user leads me assume this is SpamAssassin - I've seen this
> > > behaviour at work, where our scripts were trying to backup a 1TB file
> > > (which actually was ~vscan/.spamassassin/auto-whitelist). The result was
> > > that the backup script died due to lack of disk space on the backup
> > > server (as we don't use compression).
> > >
> > > When I was investigating why the file could be so large it, it turned
> > > out the file was only a few hunderd 'real' MB's, so that is why I assume
> > > this person is having the same issue as we do. The file is a Berkeley DB
> > > file, by the way, so there's nothing textfile about it ;-)
> >
> > I learn something every day
> > Didn't know BDB was smart enough to create sparse files.
>
> BTW, you can use "ls -ls" to display the number of physical
> blocks allocated to the file, so you can easily see whether
> a file is sparse or not:
>
> $ dd if=/dev/zero of=foo1 bs=1m count=1
> $ truncate -s 1m foo2
> $ ls -ls foo1 foo2
> 1040 -rw------- 1 olli olli 1048576 Jun 20 22:43 foo1
> 32 -rw------- 1 olli olli 1048576 Jun 20 22:43 foo2

# ls -lsk
total 1247288
664064 -rw------- 1 vscan vscan 4398199488512 Jun 23 09:39
auto-whitelist
88 -rw------- 1 vscan vscan 89976 Jun 23 09:39 bayes_journal
566704 -rw------- 1 vscan vscan 1099639861248 Jun 23 09:39 bayes_seen
16432 -rw------- 1 vscan vscan 21454848 Jun 23 09:39 bayes_toks

>
> As you can see, the file size is the same, but the block
> counts are different (I have BLOCKSIZE=K in my environment,
> so the blocks are displayed in 1KB units).
>
> I've written a small script that can be used to detect
> sparse files (it even displays the "sparseness" percentage):
>
> http://www.secnetix.de/olli/scripts/sparsecheck
>
> Best regards
> Oliver
>
> PS: Of course it is still possible that a file system is
> corrupt and needs fsck, no matter whether those files are
> sparse or not.
>


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFIX0W+xJBWvpalMpkRAqrdAJ47eLQ+WMp6zBrme5gNyC SvzBtdUwCffYwT
+37ul1gPqmk7rVKXRrha7fU=
=uSGe
-----END PGP SIGNATURE-----
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Ivan Voras wrote:
> Rink Springer wrote:
>> On Fri, Jun 20, 2008 at 03:54:22PM +0200, Ivan Voras wrote:
>>> Except that the file in question should be, judging by the filename, a
>>> simple text file. I don't really see how a whitelist could grow to such
>>> monstrous sizes Most likely it's a file system corruption - fsck
>>> should be the first thing to try.
>> The 'vscan' user leads me assume this is SpamAssassin - I've seen this
>> behaviour at work, where our scripts were trying to backup a 1TB file
>> (which actually was ~vscan/.spamassassin/auto-whitelist). The result was
>> that the backup script died due to lack of disk space on the backup
>> server (as we don't use compression).

Yes, it is SpamAssassin and I have the same problem with the backups.
The Amanda complains with:
xxxxxxxxxxx / lev 1 FAILED [dump larger than available tape space,
307256178 KB, skipping incremental]
yyyyyyyyyyyy / lev 4 FAILED [dump larger than tape, 354005617 KB,
skipping incremental]

I don't think this is a file system corruption because there are no
reasons for this. Also I have seen this on different machines.

Any ideas how to fix this?

>>
>> When I was investigating why the file could be so large it, it turned
>> out the file was only a few hunderd 'real' MB's, so that is why I assume
>> this person is having the same issue as we do. The file is a Berkeley DB
>> file, by the way, so there's nothing textfile about it ;-)
>
> I learn something every day
> Didn't know BDB was smart enough to create sparse files.
>



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFIX0kwxJBWvpalMpkRAjkaAKCbF7sylsDyI2umCAoneq BqAYCNqwCgnQjA
9FkgJiVhqNI2NGCAlWpqfbg=
=PQZ1
-----END PGP SIGNATURE-----
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Mon, Jun 23, 2008 at 09:56:48AM +0300, Krassimir Slavchev wrote:

> I don't think this is a file system corruption because there are no
> reasons for this. Also I have seen this on different machines.
>
> Any ideas how to fix this?

Either use backup software that is aware of sparse files
or eliminate sofrware that uses such files :-)

Eugene
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Mon, Jun 23, 2008 at 09:56:48AM +0300, Krassimir Slavchev wrote:
> Ivan Voras wrote:
> > Rink Springer wrote:
> >> On Fri, Jun 20, 2008 at 03:54:22PM +0200, Ivan Voras wrote:
> >>> Except that the file in question should be, judging by the filename, a
> >>> simple text file. I don't really see how a whitelist could grow to such
> >>> monstrous sizes Most likely it's a file system corruption - fsck
> >>> should be the first thing to try.
> >> The 'vscan' user leads me assume this is SpamAssassin - I've seen this
> >> behaviour at work, where our scripts were trying to backup a 1TB file
> >> (which actually was ~vscan/.spamassassin/auto-whitelist). The result was
> >> that the backup script died due to lack of disk space on the backup
> >> server (as we don't use compression).

I've used SpamAssassin for a few years now, and I've never seen this
happen (including during migration from RELENG_6 to RELENG_7, and
between SpamAssassin versions (from 3.0 to 3.2.x). I cannot even begin
to imagine how that file reached such a size, which is why I believe the
issue may be filesystem corruption.

I will take a moment to point out something I do whenever SpamAssassin
gets upgraded, though: I always delete bayes_* and auto-whitelist files
from .spamassassin, on every account. I do this because there have been
cases in the past where the data format has changed in the DB, and SA
hasn't done a good job of seamlessly migrating them.

> Yes, it is SpamAssassin and I have the same problem with the backups.
> The Amanda complains with:
> xxxxxxxxxxx / lev 1 FAILED [dump larger than available tape space,
> 307256178 KB, skipping incremental]
> yyyyyyyyyyyy / lev 4 FAILED [dump larger than tape, 354005617 KB,
> skipping incremental]

There isn't any way to solve this problem. The file, as far as dump is
concerned, is indeed 3TB.

> I don't think this is a file system corruption because there are no
> reasons for this. Also I have seen this on different machines.

I can point you to all of the threads in the past year where users have
had mysterious problems with their filesystems, which have been fixed by
booting single-user and fsck'ing. That may not be the problem here,
but it doesn't hurt for you to do it "just in case" now does it?

If it happens on multiple machines, I've only two recommendations:

1) Try what I said above (delete the bayes_* and auto-whitelist files).
If SpamAssassin recreates them as 3TB, then there is indeed a problem
with your system(s), because mine do not do that.

2) Consider migrating to dspam instead, which supposedly has a much
greater success rate of blocking spam.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Mon, Jun 23, 2008 at 12:19:28AM -0700, Jeremy Chadwick wrote:
> I've used SpamAssassin for a few years now, and I've never seen this
> happen (including during migration from RELENG_6 to RELENG_7, and
> between SpamAssassin versions (from 3.0 to 3.2.x). I cannot even begin
> to imagine how that file reached such a size, which is why I believe the
> issue may be filesystem corruption.

I've seen spamassassin creating large sparse file regurally and do
other strang things like get the database into a state where it
spins if it trys to clean it out. That's even starting with a clean
database.

> There isn't any way to solve this problem. The file, as far as dump is
> concerned, is indeed 3TB.

Dump does understand sparse files, but doesn't store them in the
most efficient way possible. Dumping a filesystem with a 1TB file
seems to use about 2GB.

David.

# truncate -s 1T /var/tmp/bigfile
# dump 0Lf - /var | wc -c
DUMP: Date of this level 0 dump: Mon Jun 23 09:50:50 2008
DUMP: Date of last level 0 dump: the epoch
DUMP: Dumping snapshot of /dev/ad4s3d (/var) to standard output
DUMP: mapping (Pass I) [regular files]
DUMP: mapping (Pass II) [directories]
DUMP: estimated 2212849 tape blocks.
DUMP: dumping (Pass III) [directories]
DUMP: dumping (Pass IV) [regular files]
DUMP: DUMP: 2212776 tape blocks
DUMP: finished in 59 seconds, throughput 37504 KBytes/sec
DUMP: DUMP IS DONE
2265876480

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"