On Tue, 2008-01-29 at 22:37 +0100, Pierre Habouzit wrote:
> On Tue, Jan 29, 2008 at 09:16:24PM +0000, Moritz Muehlenhoff wrote:
> > Fortify Source
> > ==============
> >
> > This feature adds validation for internal C functions such as strcpy
> > for buffer sizes known during compile time. While vulnerabilities in
> > the functions it protects have become uncommon in high-profile apps,
> > it will be useful for fringe packages we have in the archive.
> >
> > This feature is present in glibc since version 2.5, and is enabled
> > through the use of "-D_FORTIFY_SOURCE=2" and "-O2" or higher.
> >
>
> Well, -D_FORTIFY_SOURCE=2 is a severe performance loss in many
> applications, and I wouldn't recommend activating it by default. =1 has
> not the drawback with that regard though, but is less useful security
> wise (though it catch many programmatic issues, and full archive rebuild
> with -D_FORTIFY_SOURCE=1 would be worthwile independently of this).
>

Out of curiosity, what applications in particular does
-D_FORTIFY_SOURCE=2 cause issues in? It may be worthwhile to profile
this feature and correct it's behaviour if the performance loss is that
big of a deal.

William

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHn587oB+26npOQg4RAprzAJ9PF8ZZqWIK4yrbrRh68R 9RyVkusQCeJf9t
Sk6AsqS2hery6E793rFzStI=
=jKL7
-----END PGP SIGNATURE-----

On Tue, Jan 29, 2008 at 09:48:43PM +0000, William Pit**** wrote:
> On Tue, 2008-01-29 at 22:37 +0100, Pierre Habouzit wrote:
> > On Tue, Jan 29, 2008 at 09:16:24PM +0000, Moritz Muehlenhoff wrote:
> > > Fortify Source
> > > ==============
> > >
> > > This feature adds validation for internal C functions such as strcpy
> > > for buffer sizes known during compile time. While vulnerabilities in
> > > the functions it protects have become uncommon in high-profile apps,
> > > it will be useful for fringe packages we have in the archive.
> > >
> > > This feature is present in glibc since version 2.5, and is enabled
> > > through the use of "-D_FORTIFY_SOURCE=2" and "-O2" or higher.
> > >
> >
> > Well, -D_FORTIFY_SOURCE=2 is a severe performance loss in many
> > applications, and I wouldn't recommend activating it by default. =1 has
> > not the drawback with that regard though, but is less useful security
> > wise (though it catch many programmatic issues, and full archive rebuild
> > with -D_FORTIFY_SOURCE=1 would be worthwile independently of this).
> >
>
> Out of curiosity, what applications in particular does
> -D_FORTIFY_SOURCE=2 cause issues in? It may be worthwhile to profile
> this feature and correct it's behaviour if the performance loss is that
> big of a deal.

Basically any application that uses memcpy/memmove and some other
common functions heavily.

--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHn6EQvGr7W6HudhwRAocJAJ9ATbMqUpuqPH3ZRdP7l+ zFo1K7OQCgjauU
0D76i+Bea7bffq+UzgXpVds=
=Wo4N
-----END PGP SIGNATURE-----

Pierre Habouzit wrote:
>> Fortify Source
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>=20
>> This feature adds validation for internal C functions such as strcpy
>> for buffer sizes known during compile time. While vulnerabilities in
>> the functions it protects have become uncommon in high-profile apps,
>> it will be useful for fringe packages we have in the archive.
>>=20
>> This feature is present in glibc since version 2.5, and is enabled
>> through the use of "-D_FORTIFY_SOURCE=3D2" and "-O2" or higher.
>>=20
>
> Well, -D_FORTIFY_SOURCE=3D2 is a severe performance loss in many
> applications, and I wouldn't recommend activating it by default. =3D1 has
> not the drawback with that regard though, but is less useful security
> wise (though it catch many programmatic issues, and full archive rebuild
> with -D_FORTIFY_SOURCE=3D1 would be worthwile independently of this).

There are certainly performance trade-offs involved and the final
selection of features will depend on the testing of the respective
maintainers (testing should be eased by hardening-wrapper).

hardening-wrapper makes it simple to enable/disable selective single
features, if anyone wants to run specific benchmarks on the overhead,
please post them to the Wiki.

We're mostly trying to bootstrap a discussion here, the details on
how to put this into effect archive-wide will depend heavily on the
toolchain configuration proposal by Matthias Klose. Maybe "classes"
of security-sensitivity of applications can be defined, which specify
a set of selected options.

Cheers,
Moritz


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Tue, Jan 29, 2008 at 10:31:48PM +0000, Moritz Muehlenhoff wrote:
> There are certainly performance trade-offs involved and the final
> selection of features will depend on the testing of the respective
> maintainers (testing should be eased by hardening-wrapper).

I understand. To be fair, I'm worried in the implications of the SSP,
FORTITY_SOURCES and PIE proposals. Others looks fine, but those three
may have very important performance issues embedded.

* SSP has a cost proportional to the number of calls an application
performs (If I'm correct), which in CPU intensive tasks may become an
issue.
* FORTITY_SOURCES=2 checks memcpy and memmove, though other functions it
checks should just not be used and applications beeing too slow
because of them should just be shot down.
* PIE is just IMHO not an option on x86 :/

Though probably someone should come up with some benchmarks. The usual
culprits (multimedia libraries, html renderers, xml processors, …) all
provide easily deployed bench, and before we go any further I'd like to
see some numbers.

If it's say less than a percent, okay I'm all for it. If we have more
than 10% performance losses because of that, then we implicitely ask our
users to sometimes buy faster machines (I know many people having
installations where their multimedia player eats 80% CPU while decoding
a film because they run it on old hardware, we may just kill this kind
of use, and I would be sorry).

--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHn6yMvGr7W6HudhwRAld6AJ0f6VVfSI2LgWsAw5WIZg 33/+anigCfRriC
0GqcGScqIe4DQV52zbgE/ok=
=PfCQ
-----END PGP SIGNATURE-----

On Tue, Jan 29, 2008 at 11:45:32PM +0100, Pierre Habouzit wrote:
>On Tue, Jan 29, 2008 at 10:31:48PM +0000, Moritz Muehlenhoff wrote:
>> There are certainly performance trade-offs involved and the final
>> selection of features will depend on the testing of the respective
>> maintainers (testing should be eased by hardening-wrapper).
>
> I understand. To be fair, I'm worried in the implications of the SSP,
>FORTITY_SOURCES and PIE proposals. Others looks fine, but those three
>may have very important performance issues embedded.
>
>* PIE is just IMHO not an option on x86 :/

If you use shared libraries, you already have this performance hit, and
if you want to use %ebx in non-PIC code, you have to save and restore it
around each shared-library function call. Also note that x86 has other
problems (namely the 387) that make it unsuitable as a serious
architecture.

> Though probably someone should come up with some benchmarks. The usual
>culprits (multimedia libraries, html renderers, xml processors, …)all
>provide easily deployed bench, and before we go any further I'd like to
>see some numbers.

And now for some numbers, on an amd64 machine (/proc/cpuinfo at [0]),
crypto code with heavy optimization in a shared library. All object
files (including for the executable) are compiled with -fPIC, because
I'm lazy.

Performance without protection:

MD4: 335544320 bytes in 0.745s (429.667 MiB/s)
MD5: 335544320 bytes in 1.055s (303.452 MiB/s)
RMD160: 335544320 bytes in 1.757s (182.169 MiB/s)
SHA1: 335544320 bytes in 1.801s (177.693 MiB/s)
SHA256: 335544320 bytes in 3.519s (90.928 MiB/s)

With -fstack-protector only:

MD4: 335544320 bytes in 0.748s (427.999 MiB/s)
MD5: 335544320 bytes in 1.056s (302.979 MiB/s)
RMD160: 335544320 bytes in 1.751s (182.744 MiB/s)
SHA1: 335544320 bytes in 1.804s (177.348 MiB/s)
SHA256: 335544320 bytes in 3.515s (91.028 MiB/s)

With -D_FORTIFY_SOURCE=2 only:

MD4: 335544320 bytes in 0.745s (429.670 MiB/s)
MD5: 335544320 bytes in 1.053s (303.846 MiB/s)
RMD160: 335544320 bytes in 1.756s (182.225 MiB/s)
SHA1: 335544320 bytes in 1.794s (178.330 MiB/s)
SHA256: 335544320 bytes in 3.514s (91.057 MiB/s)

With -fPIE -pie only:

MD4: 335544320 bytes in 0.741s (431.703 MiB/s)
MD5: 335544320 bytes in 1.052s (304.173 MiB/s)
RMD160: 335544320 bytes in 1.755s (182.381 MiB/s)
SHA1: 335544320 bytes in 1.797s (178.104 MiB/s)
SHA256: 335544320 bytes in 3.517s (90.996 MiB/s)

With all three features:

MD4: 335544320 bytes in 0.744s (429.978 MiB/s)
MD5: 335544320 bytes in 1.058s (302.479 MiB/s)
RMD160: 335544320 bytes in 1.758s (182.071 MiB/s)
SHA1: 335544320 bytes in 1.793s (178.471 MiB/s)
SHA256: 335544320 bytes in 3.520s (90.896 MiB/s)

In conclusion, there is no appreciable performance hit on any algorithm.
Note that these are all hash algorithms, but they all make heavy use of
memcpy, and are extremely CPU-intensive.

Code available upon request.

[0] https://crustytoothpaste.ath.cx/machines/lakeview

--
brian m. carlson / brian with sandals: Houston, Texas, US
+1 713 440 7475 | http://crustytoothpaste.ath.cx/~bmc | My opinion only
troff on top of XML: http://crustytoothpaste.ath.cx/~bmc/code/thwack
OpenPGP: RSA v4 4096b 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iQIVAwUBR5+7LL9TXYEfUvaLAQLSug/8Ciy5kCD+DRYD/+qVQXcMu6zkoB2mT2Ga
lFnofdgMPatsvloq1lP8PT88d7OplvO8qCn3u8aRPFriisvH7n BUyVE+FhzXqqP2
Ik61zcD+NU5bb4Nbi7fP/CDanvsh0DalS6K55JHzmgJseZ310oGIQbmRfeAqDx4s
kzxY4fdaYtI2CiT8tiSLazUdomAxvAbcjFxqB2gLgVoKzWscXj hGzFSGPHaj/k5+
5eoctDTU5yqMPj706N96ulLGNOCBAaYbAKKGRSuWU0/0ulnkROXfC5VrNVz0CKsO
yWzSOOO/5uLTOJoPrUNCaj3ctntrFfGFy9RagnPXd74n3bdgV101cANKRD ThJEaK
+ub4K8AM3kw7QB5dDV3M21aivoBCHjp4QotOAxuvxMcDyQuGKX h6Ol0DpQCulkGa
pNgvEvgxdjCx/m+Ut7deTqJdKDJhKF4I+r6q9e7Rp2fwROKY7RUBfei3IPDebTA V
1u82XYy5SHTEjWpx9aIWTH48DHT6LqwO6UwMNZx23xa2kxYMtf HNsLkQDNvXlW8M
hn/Zjt04YcihQ3cE7DeVTyKtXqld564XrXO8xGBrt+Z+PZJ4Mf0Mx hfyudeBOk+W
2vydNZrSwLLa9QMOVwbnc2N3FFrIZgIA/LLWVkLKcOS2ErBz9js8FIwDdXCLdLYi
IXfMTuVmAuw=
=MszC
-----END PGP SIGNATURE-----

On Tue, Jan 29, 2008 at 11:45:32PM +0100, Pierre Habouzit wrote:
> * SSP has a cost proportional to the number of calls an application
> performs (If I'm correct), which in CPU intensive tasks may become an
> issue.

In testing I've done, this is trivial overhead. Note that the extra code
is only generated for functions with >8 byte of stack.
(-fstack-protector-all is for doing _all_ functions, which doesn't make
any sense.)

> * FORTITY_SOURCES=2 checks memcpy and memmove, though other functions it
> checks should just not be used and applications beeing too slow
> because of them should just be shot down.

I still haven't found a comprehensive list of what -D_FORTIFY_SOURCE
does, and at which levels various features are enabled. I've dug up
various bits and pieces, but I'd love to see a pointer to good
documentation. Without that, I suspect attempting to develop a
reasonable test workload is bound to miss certain things.

> * PIE is just IMHO not an option on x86 :/

I disagree here -- the bulk of applications tend to use shared libraries,
and those are all PIC. I have a hard time believing that adding the
same relocation overhead for the main program is an issue. Of course,
without numbers, we're all just waving our arms.

> Though probably someone should come up with some benchmarks. The usual
> culprits (multimedia libraries, html renderers, xml processors, …) all
> provide easily deployed bench, and before we go any further I'd like to
> see some numbers.

Does anyone have any good test harnesses we can try this on? I'd be
more than happy to run them on some modern hardware.

-Kees

--
Kees Cook


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Tue, Jan 29, 2008 at 11:47:57PM +0000, brian m. carlson wrote:
> In conclusion, there is no appreciable performance hit on any algorithm.
> Note that these are all hash algorithms, but they all make heavy use of
> memcpy, and are extremely CPU-intensive.

OTOH the memcpy they use are statically checkable, so it's not a good
test. the _chk versions of memcpy that are used are builtins in gcc that
reduce to a simple memcpy if gcc was able to say statically if the bound
was OK. Hash algorithms are not that good for checks for those features
because their stack depth is usually thin, and boundaries easily
checked.

That's why I suggested testing encoding/decoding multimedia streams
instead, or heavy text processors (xslt, docbook, ...) come to mind.


--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHoDkovGr7W6HudhwRAssIAJwNKu5Ex+cO+6gzaoVKHB x5yUrjfACfWdiJ
5crp5yDOVJ8ilV8iVb2DmZg=
=QSCO
-----END PGP SIGNATURE-----

On Tue, Jan 29, 2008 at 11:47:57PM +0000, brian m. carlson wrote:
> And now for some numbers, on an amd64 machine (/proc/cpuinfo at [0]),

Oh and I missed that at the first read, but …

> With -fPIE -pie only:
>
> MD4: 335544320 bytes in 0.741s (431.703 MiB/s)
> MD5: 335544320 bytes in 1.052s (304.173 MiB/s)
> RMD160: 335544320 bytes in 1.755s (182.381 MiB/s)
> SHA1: 335544320 bytes in 1.797s (178.104 MiB/s)
> SHA256: 335544320 bytes in 3.517s (90.996 MiB/s)

Of course it has no hit on amd64, it has all the relative operations
needed to support PIE properly, and is not register starved. Benches
have to be performed on x86 to have any valuable sense.

With everything enabled, I expect on many applications on x86 to get
on a regular basis 10% to 15% performances hit. Of course, on an
architecture like amd64, I rather expect it to rarely be over a few
percent. And as despise-able x86 is, it's still by a definite margin one
of the most used architecture in Debian.

--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHoEJIvGr7W6HudhwRAi3WAJ9tU7mIsIFAX+MH4swnb8 AIjNlNeACeJH8m
x3Uwv8wIWyMmT+DTes/6qi8=
=v6BG
-----END PGP SIGNATURE-----

Kees Cook wrote:
> Does anyone have any good test harnesses we can try this on? I'd be
> more than happy to run them on some modern hardware.

Video:
mplayer with the -benchmark option in conjunction with -nosound and -vo.

HTML rendering:
Mike Hommey once blogged about benchmarking the ACID test:
http://web.glandium.org/blog/?cat=17

Nexuiz:
| To run the benchmark: start Nexuiz & open the console (`) issuing:
| timedemo demos/demo1.dem The results will be stored in:
| ~/.nexuiz/data/benchmark.log

Not sure about XML benchmarks.

Cheers,
Moritz


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Wed, Jan 30, 2008 at 07:46:55PM +0000, Moritz Muehlenhoff wrote:
> Kees Cook wrote:
> > Does anyone have any good test harnesses we can try this on? I'd be
> > more than happy to run them on some modern hardware.
>
> Video:
> mplayer with the -benchmark option in conjunction with -nosound and -vo.
>
> HTML rendering:
> Mike Hommey once blogged about benchmarking the ACID test:
> http://web.glandium.org/blog/?cat=17
>
> Nexuiz:
> | To run the benchmark: start Nexuiz & open the console (`) issuing:
> | timedemo demos/demo1.dem The results will be stored in:
> | ~/.nexuiz/data/benchmark.log
>
> Not sure about XML benchmarks.

About XML benchmarks you can use any big-enough xslt thingy. I'm sure
people know docbook things that take long time to build, and that can
give good numbers
--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHoNufvGr7W6HudhwRAvYtAKCDSeoMkiM3RDc3sVQ/iYAb91ocdQCfaIhR
L0RcKSARfWlwGDfZcMWqN5A=
=q/xO
-----END PGP SIGNATURE-----

Hi,

I finally got some time to run some benchmarks. I checked the results[1]
into the "hardening" svn tree, in case other people want to contribute
more stuff.

On Wed, Jan 30, 2008 at 08:46:55PM +0100, Moritz Muehlenhoff wrote:
> Video:
> mplayer with the -benchmark option in conjunction with -nosound and -vo.

mplayer doesn't compile with PIE due to the various ASM routines. (I've
noted this failure mode in the wiki[2] now.) However, with everything
else enabled (including FORTIFY_SOURCE), there was no measurable
difference (it was below the percentage difference between runs):

runtime in seconds
Mplayer Normal Hardened
1 10.87 10.807
2 10.873 10.824
3 10.854 10.963
4 10.809 10.84
5 10.877 10.838
avg 10.8566 10.8544 diff: -0.02%
error 0.19% 1.00%

> HTML rendering:
> Mike Hommey once blogged about benchmarking the ACID test:
> http://web.glandium.org/blog/?cat=17

I followed that to: http://celtickane.com/webdesign/jsspeed2007.php
The differences between runs were too high for me to use, so I skipped
this for now.

> Nexuiz:
> | To run the benchmark: start Nexuiz & open the console (`) issuing:
> | timedemo demos/demo1.dem The results will be stored in:
> | ~/.nexuiz/data/benchmark.log

This one showed a possible difference:

nexuiz Normal Hardened
1 66.68 68.113
2 66.802 66.93
3 66.758 67.03
4 66.728 67.051
5 66.859 67.037
avg 66.7654 67.2322 diff: 0.70%
error 0.14% 1.31%

So, for nexuiz, with all hardening enabled in i386, there was a
less-than-1-percent reduction in speed. Though the error margin for the
hardened runs were still larger than the measured slow-down.

> Not sure about XML benchmarks.

I did parse/render tests with inkscape on i386. Some of that is XML, but
I figured it was heavy CPU, which might be worth something. Note that
inkscape already compiles with all hardening options (excepting PIE),
so the "hardened" time differences are entirely due to PIE. This one
turned out similar to nexuiz, but with less error. Again, less than 1
percent slow-down was seen.

inkscape Normal Hardened
1 48.163 48.503
2 48.227 48.535
3 48.267 48.647
4 48.335 48.431
5 48.199 48.587
avg 48.2382 48.5406 diff: 0.63%
error 0.20% 0.22%

I also ran inkscape and nexuiz tests on x86_64, and there was no
measurable difference. I'm unclear if this was due to the extra
registers, or just that that CPU was much faster and the difference
vanished into the noise.

-Kees


[1] http://svn.debian.org/wsvn/hardening/benchmarks/
[2] http://wiki.debian.org/Hardening

--
Kees Cook @outflux.net


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Wed, Mar 05, 2008 at 06:16:33AM +0000, Kees Cook wrote:
> Hi,
>
> I finally got some time to run some benchmarks. I checked the results[1]
> into the "hardening" svn tree, in case other people want to contribute
> more stuff.
>
> On Wed, Jan 30, 2008 at 08:46:55PM +0100, Moritz Muehlenhoff wrote:
> > Video:
> > mplayer with the -benchmark option in conjunction with -nosound and -vo.
>
> mplayer doesn't compile with PIE due to the various ASM routines. (I've
> noted this failure mode in the wiki[2] now.) However, with everything
> else enabled (including FORTIFY_SOURCE), there was no measurable
> difference (it was below the percentage difference between runs):
>
> runtime in seconds
> Mplayer Normal Hardened
> 1 10.87 10.807
> 2 10.873 10.824
> 3 10.854 10.963
> 4 10.809 10.84
> 5 10.877 10.838
> avg 10.8566 10.8544 diff: -0.02%
> error 0.19% 1.00%
>
> > HTML rendering:
> > Mike Hommey once blogged about benchmarking the ACID test:
> > http://web.glandium.org/blog/?cat=17
>
> I followed that to: http://celtickane.com/webdesign/jsspeed2007.php
> The differences between runs were too high for me to use, so I skipped
> this for now.
>
> > Nexuiz:
> > | To run the benchmark: start Nexuiz & open the console (`) issuing:
> > | timedemo demos/demo1.dem The results will be stored in:
> > | ~/.nexuiz/data/benchmark.log
>
> This one showed a possible difference:
>
> nexuiz Normal Hardened
> 1 66.68 68.113
> 2 66.802 66.93
> 3 66.758 67.03
> 4 66.728 67.051
> 5 66.859 67.037
> avg 66.7654 67.2322 diff: 0.70%
> error 0.14% 1.31%
>
> So, for nexuiz, with all hardening enabled in i386, there was a
> less-than-1-percent reduction in speed. Though the error margin for the
> hardened runs were still larger than the measured slow-down.
>
> > Not sure about XML benchmarks.
>
> I did parse/render tests with inkscape on i386. Some of that is XML, but
> I figured it was heavy CPU, which might be worth something. Note that
> inkscape already compiles with all hardening options (excepting PIE),
> so the "hardened" time differences are entirely due to PIE. This one
> turned out similar to nexuiz, but with less error. Again, less than 1
> percent slow-down was seen.
>
> inkscape Normal Hardened
> 1 48.163 48.503
> 2 48.227 48.535
> 3 48.267 48.647
> 4 48.335 48.431
> 5 48.199 48.587
> avg 48.2382 48.5406 diff: 0.63%
> error 0.20% 0.22%
>
> I also ran inkscape and nexuiz tests on x86_64, and there was no
> measurable difference. I'm unclear if this was due to the extra
> registers, or just that that CPU was much faster and the difference
> vanished into the noise.

Thank you very much for those. Though what did you built using -fPIE
FORTIFY_SOURCES and so on ? only the tested applications ? or their
build-deps as well ? Because I don't expect mplayer to be slowed a lot
if you don't rebuild its ogg/mp3/mpg/... as well Same goes for
inkscape.

--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHzmUEvGr7W6HudhwRAkGQAJ43VmfHdE1dCzaGtTmSSE lsSUif3QCcCdYa
O3cne3QlLvlrdNwJiVaWig0=
=Kfsa
-----END PGP SIGNATURE-----

On Tue, 04 Mar 2008, Kees Cook wrote:
> mplayer doesn't compile with PIE due to the various ASM routines. (I've
> noted this failure mode in the wiki[2] now.) However, with everything
> else enabled (including FORTIFY_SOURCE), there was no measurable
> difference (it was below the percentage difference between runs):
>
> runtime in seconds
> Mplayer Normal Hardened
> 1 10.87 10.807
> 2 10.873 10.824
> 3 10.854 10.963
> 4 10.809 10.84
> 5 10.877 10.838


Just for future reference, it'd probably be better to run more than 5
tests of each population in the future, as 5 tests means you'll only
detect very large differences in performance at any reasonable level
of signifigance.

FE:

> t.test(x=c(10.87,10.873,10.854,10.809,10.877),y=c( 10.807,10.824,10.963,10.84,10.838))

Welch Two Sample t-test

data: c(10.87, 10.873, 10.854, 10.809, 10.877) and c(10.807, 10.824, 10.963, 10.84, 10.838)
t = 0.0722, df = 5.561, p-value = 0.945
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.07382831 0.07822831
sample estimates:
mean of x mean of y
10.8566 10.8544


> This one showed a possible difference:
>
> nexuiz Normal Hardened
> 1 66.68 68.113
> 2 66.802 66.93
> 3 66.758 67.03
> 4 66.728 67.051
> 5 66.859 67.037

While there may be a possible difference here, it's not significant,
given the sample size:

> t.test(x=c(66.68,66.802,66.758,66.728,66.859),y=c( 68.113,66.93,67.03,67.051,67.037))

Welch Two Sample t-test

data: c(66.68, 66.802, 66.758, 66.728, 66.859) and c(68.113, 66.93, 67.03, 67.051, 67.037)
t = -2.0899, df = 4.154, p-value = 0.1023
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.0779888 0.1443888
sample estimates:
mean of x mean of y
66.7654 67.2322

But useful data nevertheless.[1]


Don Armstrong

1: I won't even begin to discuss how many times I see benchmarks
without SEM or sd reported.
--
I'd sign up in a hot second for any cellular company whose motto was:
"We're less horrible than a root canal with a cold chisel."
-- Cory Doctorow

http://www.donarmstrong.com http://rzlab.ucr.edu


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Wed, Mar 05, 2008 at 01:29:01AM -0800, Don Armstrong wrote:
> Just for future reference, it'd probably be better to run more than 5
> tests of each population in the future, as 5 tests means you'll only
> detect very large differences in performance at any reasonable level
> of signifigance.

I suspected. I'm not much of a statistician, so I saved the
raw data, hoping other folks would hop in to help out on this one.
(Which you have!)

Getting larger data sets will be rather time-consuming -- especially for
nexuiz which I didn't figure out how to automate.

> > t.test(x=c(10.87,10.873,10.854,10.809,10.877),y=c( 10.807,10.824,10.963,10.84,10.838))

What tool is this you're using?

> Welch Two Sample t-test
>
> data: c(10.87, 10.873, 10.854, 10.809, 10.877) and c(10.807, 10.824, 10.963, 10.84, 10.838)
> t = 0.0722, df = 5.561, p-value = 0.945
> alternative hypothesis: true difference in means is not equal to 0
> 95 percent confidence interval:
> -0.07382831 0.07822831
> sample estimates:
> mean of x mean of y
> 10.8566 10.8544

Which of these outputs should be paid attention to?

> But useful data nevertheless.[1]
>
> 1: I won't even begin to discuss how many times I see benchmarks
> without SEM or sd reported.

Heh, well I know of the ideas, but haven't had any practice actually
calculating them.

Thanks!

-Kees

--
Kees Cook @outflux.net


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Wed, Mar 05, 2008 at 10:16:52AM +0100, Pierre Habouzit wrote:
> On Wed, Mar 05, 2008 at 06:16:33AM +0000, Kees Cook wrote:
> > I finally got some time to run some benchmarks. I checked the results[1]
> > into the "hardening" svn tree, in case other people want to contribute
> > more stuff.
>
> Thank you very much for those. Though what did you built using -fPIE
> FORTIFY_SOURCES and so on ? only the tested applications ? or their
> build-deps as well ? Because I don't expect mplayer to be slowed a lot
> if you don't rebuild its ogg/mp3/mpg/... as well Same goes for
> inkscape.

Well, libraries are already -fPIC so there's no need to recompile those.
As for FORTIFY_SOURCE, that's true, I didn't rebuild the libraries with
it for these tests. Getting all libs rebuilt may take a lot longer.

--
Kees Cook @outflux.net


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Wed, Mar 05, 2008 at 09:55:37AM -0800, Kees Cook wrote:
>>> t.test(x=c(10.87,10.873,10.854,10.809,10.877),y=c( 10.807,10.824,10.963,10.84,10.838))
> What tool is this you're using?

GNU R. Takes a while to get into, but hard to beat for statistics.

>> data: c(10.87, 10.873, 10.854, 10.809, 10.877) and c(10.807, 10.824, 10.963, 10.84, 10.838)
>> t = 0.0722, df = 5.561, p-value = 0.945
>> alternative hypothesis: true difference in means is not equal to 0
>> 95 percent confidence interval:
>> -0.07382831 0.07822831
>> sample estimates:
>> mean of x mean of y
>> 10.8566 10.8544
> Which of these outputs should be paid attention to?

The p-value is a good idea. If it's about 0.05, people tend to say the result
is statistically significant. In your case it's 0.945, which means that the
result is for all practical purposes worthless.

/* Steinar */
--
Homepage: http://www.sesse.net/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

While these benchmarks should show any differences in raw processing
performance, there's also the question of what differences the hardening
measures make to application start-up times. PIE in particular should cause
some slowdown when the executables are first run, but it would take some
other sort of benchmark to determine the impact. PIE also makes prelinking
ineffective, so it would probably be worth testing the difference between
prelinked and PIE executables.

On Wednesday 05 March 2008, Kees Cook wrote:
> I also ran inkscape and nexuiz tests on x86_64, and there was no
> measurable difference. I'm unclear if this was due to the extra
> registers, or just that that CPU was much faster and the difference
> vanished into the noise.

The x86_64 arch has special hardware to better support PIE, so the lack of any
noticeable difference in performance is likely due to that.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHzvMMLE8yW/+QbWIRAiK4AKCIAzEffL+IfAG/SJi/gZdhFBcdIACgvMjf
PGV8PmmglwTEawveVSwgGgE=
=SItv
-----END PGP SIGNATURE-----

On Wed, 05 Mar 2008, Kees Cook wrote:
> On Wed, Mar 05, 2008 at 01:29:01AM -0800, Don Armstrong wrote:
> > Just for future reference, it'd probably be better to run more than 5
> > tests of each population in the future
>
> Getting larger data sets will be rather time-consuming -- especially
> for nexuiz which I didn't figure out how to automate.

Yeah; the converse is that since you're really interested in major
differences, 5 samples should be able to detect those.[1]

> > t = 0.0722, df = 5.561, p-value = 0.945
> > 95 percent confidence interval:
> > -0.07382831 0.07822831
> > sample estimates:
> > mean of x mean of y
> > 10.8566 10.8544
>
> Which of these outputs should be paid attention to?

The p value and the convidence interval are the two things that are
really useful. The first tells you that assuming the true difference
between the means is zero, you would expect to see a difference like
this or larger 94.5% of the time. The second tells you that with 95%
confidence the true difference between the means is between -0.07 and
0.07.


Don Armstrong

1: There are sample size calculators where given power (or 1-beta),
alpha, and the difference you wish to detect will give you the number
of samples required.
--
When bad men combine, the good must associate; else they will fall one
by one, an unpitied sacrifice in a contemptible struggle.
-- Edmund Burke "Thoughts on the Cause of Present Discoontents"

http://www.donarmstrong.com http://rzlab.ucr.edu


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Wed, Mar 05, 2008 at 05:48:57PM +0000, Kees Cook wrote:
> On Wed, Mar 05, 2008 at 10:16:52AM +0100, Pierre Habouzit wrote:
> > On Wed, Mar 05, 2008 at 06:16:33AM +0000, Kees Cook wrote:
> > > I finally got some time to run some benchmarks. I checked the results[1]
> > > into the "hardening" svn tree, in case other people want to contribute
> > > more stuff.
> >
> > Thank you very much for those. Though what did you built using -fPIE
> > FORTIFY_SOURCES and so on ? only the tested applications ? or their
> > build-deps as well ? Because I don't expect mplayer to be slowed a lot
> > if you don't rebuild its ogg/mp3/mpg/... as well Same goes for
> > inkscape.
>
> Well, libraries are already -fPIC so there's no need to recompile those.
> As for FORTIFY_SOURCE, that's true, I didn't rebuild the libraries with
> it for these tests. Getting all libs rebuilt may take a lot longer.

Well then sadly it makes the benchmarks a lot less interesting :/

--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHzzWivGr7W6HudhwRAtJLAJ456lxuNoEX5wfnWamnuI 2ImoQArACgmwgN
NKT0uq9dYYmUdG2ADgocGj0=
=oapd
-----END PGP SIGNATURE-----