ports/126853: ports-mgmt/portaudit: speed up audit of installedpackages - FreeBSD

This is a discussion on ports/126853: ports-mgmt/portaudit: speed up audit of installedpackages - FreeBSD ; Good day. A while ago I had created the new utility that serves as VuXML filter for the installed packages: http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/126853 My primary intention was to speed up the process of auditing the vulnerable ports: I needed to run portaudit ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: ports/126853: ports-mgmt/portaudit: speed up audit of installedpackages

  1. ports/126853: ports-mgmt/portaudit: speed up audit of installedpackages

    Good day.

    A while ago I had created the new utility that serves as VuXML
    filter for the installed packages:
    http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/126853

    My primary intention was to speed up the process of auditing the
    vulnerable ports: I needed to run portaudit checks with Nagios and to
    avoid large timeouts.

    The new utility is called pkg_audit and it serves as a simple text
    filter: on input it takes the full VuXML feed and on output it puts
    VuXML entries that matches ports that are installed in the system with
    port version specification substituted with the actual port versions.

    No harm is done to the actual poartudit -- if pkg_audit is missing, old
    code path is activated.

    If someone is interested and will be able to test -- I am all ears.

    Thanks!
    --
    Eygene
    _ ___ _.--. #
    \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard
    / ' ` , __.--' # to read the on-line manual
    )/' _/ \ `-_, / # while single-stepping the kernel.
    `-'" `"\_ ,_.-;_.-\_ ', fsc/as #
    _.-'_./ {_.' ; / # -- FreeBSD Developers handbook
    {_.-``-' {_/ #

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2.0.9 (FreeBSD)

    iEYEARECAAYFAkjY784ACgkQthUKNsbL7YhcBACgkxuLO+Hm23 w+yRsbP07VfmM+
    1poAn0kvc5fWx4a9OTcfO203lDawKRXn
    =BZFN
    -----END PGP SIGNATURE-----


  2. Re: ports/126853: ports-mgmt/portaudit: speed up audit ofinstalled packages

    Roman, good day.

    Sat, Sep 27, 2008 at 08:18:08PM +0400, Roman Kurakin wrote:
    > Have you also posted this to ports@?


    No, forgot to do it. CC'ing ports@

    Thanks!

    The original posting to hackers@ goes below. It will be double-posted
    to the bug-followup@ -- sorry for this.

    > Eygene Ryabinkin wrote:
    > > Good day.
    > >
    > > A while ago I had created the new utility that serves as VuXML
    > > filter for the installed packages:
    > > http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/126853
    > >
    > > My primary intention was to speed up the process of auditing the
    > > vulnerable ports: I needed to run portaudit checks with Nagios and to
    > > avoid large timeouts.
    > >
    > > The new utility is called pkg_audit and it serves as a simple text
    > > filter: on input it takes the full VuXML feed and on output it puts
    > > VuXML entries that matches ports that are installed in the system with
    > > port version specification substituted with the actual port versions.
    > >
    > > No harm is done to the actual poartudit -- if pkg_audit is missing, old
    > > code path is activated.
    > >
    > > If someone is interested and will be able to test -- I am all ears.


    Additional clarifications inspired by the off-line talk with rik@:
    I could take another route and add this functionality to the pkg_info.
    I took another approach for the following reasons.

    1. pkg_info's option list is already quite big -- around 32 options
    and switches.

    2. It is easier to test for the presence of the new tool (pkg_audit)
    and use it, instead of checking the support for the new option in
    pkg_info.

    3. I see no options in pkg_info that can be naturally extended to
    absorbe the new functionality. The closest is '-E', but pkg_audit
    needs to read VuXML entries, choose ones that are present in the system
    and output the found VuXML entries with version templates substituted
    with the real entries, so pkg_audit is filter-like utility. In my
    opinion, such extension of pkg_info's "-E" will be very unnatural.

    4. I feel that it is Unix-way to do the things: create small utilities
    that do their (small) job in a proper fashion. Moreover, since the
    majority of a code sits in the pkg_install's library, there is a very
    slight code duplication, if any.

    Thanks for you time.
    --
    Eygene
    _ ___ _.--. #
    \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard
    / ' ` , __.--' # to read the on-line manual
    )/' _/ \ `-_, / # while single-stepping the kernel.
    `-'" `"\_ ,_.-;_.-\_ ', fsc/as #
    _.-'_./ {_.' ; / # -- FreeBSD Developers handbook
    {_.-``-' {_/ #

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2.0.9 (FreeBSD)

    iEYEARECAAYFAkjfUx4ACgkQthUKNsbL7YhBbwCfSA2OhrF4uP 6vfYW/eQ2kQ7/p
    LbgAmwTYjAJor/5W5Tc5lzeVW3RQXgQx
    =xsRg
    -----END PGP SIGNATURE-----


  3. Re: ports/126853: ports-mgmt/portaudit: speed up audit ofinstalled packages

    Miroslav, good day.

    Mon, Oct 06, 2008 at 12:41:05AM +0200, Miroslav Lachman wrote:
    > I am busy these days, but it is nice to read about your progress. I hope
    > I will get some time to test all of these large patches in a few days
    > and I will report back my experiences!


    Fine, thank you! I am re-CC'ing bug-followup@ to track this letter,
    since it contains some useful information that should go into GNATS.

    > One note before tests... do -n flag always download new INDEX file, or
    > is it possible to use one already existing in /usr/ports?


    Currently, it is downloads bzipped INDEX file to /var/db/portaudit every
    time, but it uses mirror mode, so if remote file hadn't changed at all,
    all network expences are just the HTTP's HEAD request and reply.

    I can add another variable to the portaudit to force the usage of the
    existing INDEX file, if it is needed. By the way, how are you keeping
    your INDEX file up to date (your proposed usage of 'pkg_version -I'
    implies that you're always rely on it)? I am just curious -- my INDEX
    files are almost always stay unupdated, even if I am using portupgrade.

    And there can be another way if one keeps ports tree updated: utility
    can use 'make' to determine the version that is currently available on
    the examined host.

    But downloading the INDEX file from the central server seemed to be the
    best way, since it almost always gives one the latest port versions, so
    I had implemented this in a first place.

    Don't know, however, how the badly the load to the central HTTP server
    will be raised. I am using just two first fields from the INDEX file,
    so I can use such a stripped file. For me, the reduction was about
    6x: SIZE(INDEX-7.bz2) = 1126189, SIZE(INDEX-7.stripped.bz2) = 184345.

    I am CC'ing the portmgr team. Guys, could you quickly glance over these
    patches and determine if they are useful to the project in large? If
    yes, then may be such a stripped INDEX can be created on the FreeBSD
    servers (via cut -f1-2 -d'|' INDEX-N)?

    Thanks!
    --
    Eygene
    _ ___ _.--. #
    \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard
    / ' ` , __.--' # to read the on-line manual
    )/' _/ \ `-_, / # while single-stepping the kernel.
    `-'" `"\_ ,_.-;_.-\_ ', fsc/as #
    _.-'_./ {_.' ; / # -- FreeBSD Developers handbook
    {_.-``-' {_/ #

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2.0.9 (FreeBSD)

    iEYEARECAAYFAkjpoNkACgkQthUKNsbL7YhirACeIewIwCM0wr/UBHnMp7hieJya
    eUoAoIAwCdHLh38sZwl+bvw65cB9OvgW
    =ANyt
    -----END PGP SIGNATURE-----


  4. Re: ports/126853: ports-mgmt/portaudit: speed up audit of installedpackages

    Hello,

    On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote:

    > But downloading the INDEX file from the central server seemed to be the
    > best way, since it almost always gives one the latest port versions, so
    > I had implemented this in a first place.


    I've been following this, but I don't agree that (port|pkg_)audit should do
    this, from the very perspective you're writing this program from:

    On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote:
    > 4. I feel that it is Unix-way to do the things: create small utilities
    > that do their (small) job in a proper fashion.


    Instead, it can provide installed-pkgnamepkgorigin output. Then,
    any utility can check whether a new version is available, using what ever
    source it finds relevant.
    For example, it is completely irrelevant if a new version is available on the
    FreeBSD servers, when your machine uses a buildserver in a local network. For
    those machines it's relevant whether their build server has a new version and
    one can automatically upgrade if one so desires.
    Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're
    again reporting false information.

    It's also quite trivial to provide this availibility information in a daily
    security script, for the "majority of cases" and it's better to have tunables
    like _use_remote_portindex, _use_portsdir=/bigdisk/usr/ports in a script.
    --
    Mel
    _______________________________________________
    freebsd-hackers@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...reebsd-hackers
    To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"


+ Reply to Thread