[RFC] No longer create full set of static devices during install - Debian

This is a discussion on [RFC] No longer create full set of static devices during install - Debian ; The attached set of patches allow us to no longer need/create the current huge set of static devices during installation. Instead /dev from the D-I environment is bind mounted to /target/dev after debootstrap has been run. This also ensures that ...

+ Reply to Thread
Results 1 to 14 of 14

Thread: [RFC] No longer create full set of static devices during install

  1. [RFC] No longer create full set of static devices during install

    The attached set of patches allow us to no longer need/create the current
    huge set of static devices during installation. Instead /dev from the D-I
    environment is bind mounted to /target/dev after debootstrap has been run.

    This also ensures that we will no longer "miss" any devices in /target/dev
    during installations. (We have had bug reports in the past where we were
    missing e.g. /dev/hdi.)

    debootstrap is modified to include in the udeb only the minimal tarball of
    devices currently also included in the regular debootstrap binary package.
    We still need the udeb to be "arch any" because of the utility pkgdetails
    and because /usr/lib/debootstrap/arch needs to be included.

    A minor change in rootskel creates the /dev/.static/dev directory because
    some packages may try to create static devices during installation. If any
    static devices are created, these are copied to /target/dev during
    finish-install (after unmounting the bind mount).
    I've also removed the creation of /.dev in the D-I environment as that seems
    obsolete.

    The create_devices function in base-installer has been simplified a lot
    because it no longer actually has to create any device nodes anymore.
    It now takes care of the bind mount to /target/dev and does some apt-install
    calls for RAID, LVM and crypto installs.
    (Suggestions for a better name for this function welcome!)

    I've done successful LVM and LVM-crypto installs using the new udebs.

    This proofed to be a lot simpler than I had expected, so the question is:
    what have I missed?

    Are there still any arches that don't use udev for default installs?
    If there are, they could be accommodated by calling MAKEDEV during
    base-installer (using the same set of devices that used to be created by
    debootstrap).

    Cheers,
    FJP


    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD4DBQBHNPAigm/Kwh6ICoQRAve1AJ0d8tSWYxiSjMbuy8i42mdPKPLrEACXbARu
    olSYItHD4AMQkIgMK4W+ww==
    =gGqG
    -----END PGP SIGNATURE-----


  2. Re: [RFC] No longer create full set of static devices during install

    Frans Pop wrote:
    > The create_devices function in base-installer has been simplified a lot
    > because it no longer actually has to create any device nodes anymore.
    > It now takes care of the bind mount to /target/dev and does some apt-install
    > calls for RAID, LVM and crypto installs.
    > (Suggestions for a better name for this function welcome!)


    setup_dev

    > Are there still any arches that don't use udev for default installs?


    On arm, ads_cf doesn't currently use udev in d-i, but I'm probably going
    to change that, and ads_cf shouldn't hold up any ongoing progress in d-i
    anyway. AFAIK that's the last user of userdevfs.

    > + mkdir -p /dev/.static/dev


    /dev/static is supposed to be mode 700. udev will currently fix up
    the existing permissions when it's installed, but to avoid subtly
    breaking if udev's postinst changed, I'd make it mode 700 to start with.

    I wonder if we need to include this directory in /dev at all. udev
    should create it, and migrate devices to it, if it doesn't already
    exist.

    --
    see shy jo

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD8DBQFHNjh9d8HHehbQuO8RAgD3AKCm4HgT4wY9833uVKjPfo CBMq61jgCgnJ6f
    VbUiOO+clUBLgXkIdTIk7JM=
    =8pra
    -----END PGP SIGNATURE-----


  3. Re: [RFC] No longer create full set of static devices during install

    On Sunday 11 November 2007, Joey Hess wrote:
    > Frans Pop wrote:
    > > The create_devices function in base-installer has been simplified a lot
    > > because it no longer actually has to create any device nodes anymore.
    > > It now takes care of the bind mount to /target/dev and does some
    > > apt-install calls for RAID, LVM and crypto installs.
    > > (Suggestions for a better name for this function welcome!)

    >
    > setup_dev


    Done. Still does not really cover the apt-install of dmraid and such...

    > > Are there still any arches that don't use udev for default installs?

    >
    > On arm, ads_cf doesn't currently use udev in d-i, but I'm probably going
    > to change that, and ads_cf shouldn't hold up any ongoing progress in d-i
    > anyway. AFAIK that's the last user of userdevfs.


    OK.

    > > + mkdir -p /dev/.static/dev

    >
    > /dev/static is supposed to be mode 700. udev will currently fix up
    > the existing permissions when it's installed, but to avoid subtly
    > breaking if udev's postinst changed, I'd make it mode 700 to start with.
    >
    > I wonder if we need to include this directory in /dev at all. udev
    > should create it, and migrate devices to it, if it doesn't already
    > exist.


    For an installed system it's created by /etc/init.d/udev on boot. Thing is
    that because I now bind mount /dev/ on top of /target/dev/, anything
    created in /target/dev/ during installation would get lost on the reboot.

    So I think we should mimic /dev/.static in the D-I environment to preserve
    any static devices created during installation; unless we decide we don't
    really care of course, which is probably valid too.

    I did just see that my implementation was a bit too simplistic.
    /dev/.static needs to be a bind mount in its own right (see /dev/MAKEDEV).
    I can probably come up with a cleaner implementation. I'll also have to add
    some additional code to ensure things remain idempotent.

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD8DBQBHNrI0gm/Kwh6ICoQRAqD4AJ9vgwOn/fHvU54UBKg6seRTVJVhfgCeP+z4
    zJgeB2OGnIOHiujHMQMqgBI=
    =opQ1
    -----END PGP SIGNATURE-----


  4. Re: [RFC] No longer create full set of static devices during install

    On Saturday 10 November 2007, Frans Pop wrote:
    > The attached set of patches allow us to no longer need/create the current
    > huge set of static devices during installation. Instead /dev from the D-I
    > environment is bind mounted to /target/dev after debootstrap has been
    > run.


    This has been committed now. Will upload tomorrow if there are no further
    comments.

    > debootstrap is modified to include in the udeb only the minimal tarball
    > of devices currently also included in the regular debootstrap binary
    > package. We still need the udeb to be "arch any" because of the utility
    > pkgdetails and because /usr/lib/debootstrap/arch needs to be included.


    We could probably make the debootstrap udeb "arch all" by moving pkgdetails
    to base-installer and passing '--arch $(udpkg --print-architecture)', but
    I'm not sure that's worth it.

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD8DBQBHODtRgm/Kwh6ICoQRApT9AKCUdo6w5UjUFksSJcC4wPgzRvZDSACfWyH9
    0Rph283DjWTN0LJ+Us4MZkw=
    =OuB+
    -----END PGP SIGNATURE-----


  5. Re: [RFC] No longer create full set of static devices during install

    Frans Pop wrote:
    > We could probably make the debootstrap udeb "arch all" by moving pkgdetails
    > to base-installer and passing '--arch $(udpkg --print-architecture)', but
    > I'm not sure that's worth it.


    It _would_ be handy if debootstrap could quickly be updated for breakage
    w/o needing it to be autobuilt..

    --
    see shy jo

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD8DBQFHOLXVd8HHehbQuO8RApQxAJ4gNFUegnMX0rG0MMJLh1 6MP+XzYQCggaYn
    3UOk5+fejK0wcS5bEXLOYCE=
    =GWOn
    -----END PGP SIGNATURE-----


  6. Re: [RFC] No longer create full set of static devices during install

    On Mon, Nov 12, 2007 at 12:38:49PM +0100, Frans Pop wrote:
    > On Saturday 10 November 2007, Frans Pop wrote:
    > > The attached set of patches allow us to no longer need/create the current
    > > huge set of static devices during installation. Instead /dev from the D-I
    > > environment is bind mounted to /target/dev after debootstrap has been
    > > run.

    >
    > This has been committed now. Will upload tomorrow if there are no further
    > comments.


    +1; the previous situation has been bothersome.

    > > debootstrap is modified to include in the udeb only the minimal tarball
    > > of devices currently also included in the regular debootstrap binary
    > > package. We still need the udeb to be "arch any" because of the utility
    > > pkgdetails and because /usr/lib/debootstrap/arch needs to be included.

    >
    > We could probably make the debootstrap udeb "arch all" by moving pkgdetails
    > to base-installer and passing '--arch $(udpkg --print-architecture)', but
    > I'm not sure that's worth it.


    +1 also, except I'd do it by adding:

    elif type udpkg >/dev/null 2>&1 && udpkg --print-architecture >/dev/null 2>&1; then
    ARCH=`udpkg --print-architecture`

    .... to debootstrap rather than passing the architecture in from
    base-installer.

    --
    Colin Watson [cjwatson@debian.org]


    --
    To UNSUBSCRIBE, email to debian-boot-REQUEST@lists.debian.org
    with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

  7. Re: [RFC] No longer create full set of static devices during install

    On Tuesday 13 November 2007, Colin Watson wrote:
    > > We could probably make the debootstrap udeb "arch all" by moving
    > > pkgdetails to base-installer and passing '--arch $(udpkg
    > > --print-architecture)', but I'm not sure that's worth it.

    >
    > +1 also,


    Cool. I've just committed the changes I had prepared (and tested).
    I'd very much appreciate a review of these changes.

    One thing that is not completely nice is that bootstrap-base now installs a
    file in /usr/lib/debootstrap/. I thought about putting it elsewhere, but
    think the current location is defensible as well with the rationale that it
    provides a missing component of debootstrap.

    I also have a patch prepared that saves 3kB by using tab indentation in
    debootstrap and function. Will commit that next.

    > except I'd do it by adding:
    > elif type udpkg >/dev/null 2>&1 && udpkg --print-architecture
    > >/dev/null 2>&1; then ARCH=`udpkg --print-architecture`

    >
    > ... to debootstrap rather than passing the architecture in from
    > base-installer.


    Hmmm. Why?
    We also control other (IMO similar) parameters from base-installer and your
    suggestion adds an (admittedly light) dependency on D-I internals to
    debootstrap.

    Cheers,
    FJP

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD8DBQBHOY/8gm/Kwh6ICoQRAkpYAKCl+mqZwUVl0vB47Gu26XJrc/L1GACcCnwD
    Fyu+wLo5sKTZrH3vBOF4Lrs=
    =Pzuv
    -----END PGP SIGNATURE-----


  8. Re: [RFC] No longer create full set of static devices during install

    On Tue, Nov 13, 2007 at 12:52:18PM +0100, Frans Pop wrote:
    > On Tuesday 13 November 2007, Colin Watson wrote:
    > > +1 also,

    >
    > Cool. I've just committed the changes I had prepared (and tested).
    > I'd very much appreciate a review of these changes.


    Looks fine to me.

    > One thing that is not completely nice is that bootstrap-base now installs a
    > file in /usr/lib/debootstrap/. I thought about putting it elsewhere, but
    > think the current location is defensible as well with the rationale that it
    > provides a missing component of debootstrap.


    It's a minor layering violation, but I think it's OK here. Perhaps add a
    comment to debootstrap that the pkgdetails API needs to be kept in sync
    with base-installer, so that we don't forget in future.

    > > except I'd do it by adding:
    > > elif type udpkg >/dev/null 2>&1 && udpkg --print-architecture
    > > >/dev/null 2>&1; then ARCH=`udpkg --print-architecture`

    > >
    > > ... to debootstrap rather than passing the architecture in from
    > > base-installer.

    >
    > Hmmm. Why?


    Regularity: debootstrap already does the same thing with dpkg.

    > We also control other (IMO similar) parameters from base-installer and your
    > suggestion adds an (admittedly light) dependency on D-I internals to
    > debootstrap.


    The other parameters base-installer passes are ones that debootstrap
    can't easily figure out for itself. I think 'udpkg --print-architecture'
    is a stable enough interface that it's OK to use it if available.

    Cheers,

    --
    Colin Watson [cjwatson@debian.org]


    --
    To UNSUBSCRIBE, email to debian-boot-REQUEST@lists.debian.org
    with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

  9. Re: [RFC] No longer create full set of static devices during install

    On Tuesday 13 November 2007, Colin Watson wrote:
    > On Tue, Nov 13, 2007 at 12:52:18PM +0100, Frans Pop wrote:
    > > One thing that is not completely nice is that bootstrap-base now
    > > installs a file in /usr/lib/debootstrap/. I thought about putting it
    > > elsewhere, but think the current location is defensible as well with
    > > the rationale that it provides a missing component of debootstrap.

    >
    > It's a minor layering violation, but I think it's OK here. Perhaps add a
    > comment to debootstrap that the pkgdetails API needs to be kept in sync
    > with base-installer, so that we don't forget in future.


    Comment added.

    > > > except I'd do it by adding:
    > > > elif type udpkg >/dev/null 2>&1 && udpkg --print-architecture
    > > >
    > > > >/dev/null 2>&1; then ARCH=`udpkg --print-architecture`
    > > >
    > > > ... to debootstrap rather than passing the architecture in from
    > > > base-installer.

    > >
    > > Hmmm. Why?

    >
    > Regularity: debootstrap already does the same thing with dpkg.


    OK. Done.

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD8DBQBHOaBigm/Kwh6ICoQRAsngAJ9Wgash2NqEG2QdPal6j/mwMeOJNQCfYk7k
    yjuujDJIyt5WLPEjT1yl+7Y=
    =IGDM
    -----END PGP SIGNATURE-----


  10. Re: [RFC] No longer create full set of static devices during install

    Frans Pop writes:

    >> debootstrap is modified to include in the udeb only the minimal tarball
    >> of devices currently also included in the regular debootstrap binary
    >> package. We still need the udeb to be "arch any" because of the utility
    >> pkgdetails and because /usr/lib/debootstrap/arch needs to be included.

    >
    > We could probably make the debootstrap udeb "arch all" by moving pkgdetails
    > to base-installer and passing '--arch $(udpkg --print-architecture)', but
    > I'm not sure that's worth it.


    I support this idea since this would avoid the recompilation of it on
    all arches and would make possible to have it using /usr/share for
    functions and scripts, bug #430615.

    --
    O T A V I O S A L V A D O R
    ---------------------------------------------
    E-mail: otavio@debian.org UIN: 5906116
    GNU/Linux User: 239058 GPG ID: 49A5F855
    Home Page: http://otavio.ossystems.com.br
    ---------------------------------------------
    "Microsoft sells you Windows ... Linux gives
    you the whole house."


    --
    To UNSUBSCRIBE, email to debian-boot-REQUEST@lists.debian.org
    with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

  11. Re: [RFC] No longer create full set of static devices during install

    tags 430615 + pending
    thanks

    On Wednesday 14 November 2007, Otavio Salvador wrote:
    > Frans Pop writes:
    > > We could probably make the debootstrap udeb "arch all" by moving
    > > pkgdetails to base-installer and passing '--arch $(udpkg
    > > --print-architecture)', but I'm not sure that's worth it.

    >
    > I support this idea since this would avoid the recompilation of it on
    > all arches and would make possible to have it using /usr/share for
    > functions and scripts, bug #430615.


    I don't think this is a very high-priority reason for making the udeb
    arch:all, but now that we've done it anyway: committed.

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD8DBQBHOrqIgm/Kwh6ICoQRAvA1AKCxKpYnXoeGlQCGfY2FahS6hIZeogCgyGox
    ZAlWG+vt5OO7dDC8ZcQyMLc=
    =Jk7n
    -----END PGP SIGNATURE-----


  12. Processed: Re: [RFC] No longer create full set of static devices during install

    Processing commands for control@bugs.debian.org:

    > tags 430615 + pending

    Bug#430615: debootstrap arch:all package installs arch-independent files in /usr/lib/debootstrap/ instead of /usr/share/debootstrap/
    There were no tags set.
    Tags added: pending

    > thanks

    Stopping processing here.

    Please contact me if you need assistance.

    Debian bug tracking system administrator
    (administrator, Debian Bugs database)


    --
    To UNSUBSCRIBE, email to debian-boot-REQUEST@lists.debian.org
    with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

  13. Re: Processed: Re: [RFC] No longer create full set of static devices during install

    On Wed, Nov 14, 2007 at 09:09:07AM +0000, Debian Bug Tracking System wrote:
    > Processing commands for control@bugs.debian.org:
    >
    > > tags 430615 + pending

    > Bug#430615: debootstrap arch:all package installs arch-independent files in /usr/lib/debootstrap/ instead of /usr/share/debootstrap/
    > There were no tags set.
    > Tags added: pending


    This change puts pkgdetails in /usr/share/debootstrap. There are many
    things that should certainly be in /usr/share/debootstrap (like the
    scripts), but pkgdetails is a compiled binary and so isn't one of them.
    Please move it back to /usr/lib/debootstrap.

    --
    Colin Watson [cjwatson@debian.org]


    --
    To UNSUBSCRIBE, email to debian-boot-REQUEST@lists.debian.org
    with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

  14. Re: Processed: Re: [RFC] No longer create full set of static devices during install

    On Wednesday 14 November 2007, Colin Watson wrote:
    > On Wed, Nov 14, 2007 at 09:09:07AM +0000, Debian Bug Tracking System

    wrote:
    > > Processing commands for control@bugs.debian.org:
    > > > tags 430615 + pending

    > >
    > > Bug#430615: debootstrap arch:all package installs arch-independent
    > > files in /usr/lib/debootstrap/ instead of /usr/share/debootstrap/ There
    > > were no tags set.
    > > Tags added: pending

    >
    > This change puts pkgdetails in /usr/share/debootstrap. There are many
    > things that should certainly be in /usr/share/debootstrap (like the
    > scripts), but pkgdetails is a compiled binary and so isn't one of them.
    > Please move it back to /usr/lib/debootstrap.


    Yes, you're right. Guess just hardcoding the path is not really a problem
    here. Done and thanks.

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD8DBQBHOsU4gm/Kwh6ICoQRAp+9AJ9FeqNyRmiEz9QKNiHFfrDcVYXHYQCdEf0G
    8AtB3H6VV/QuHK6MM4Aozpc=
    =ipka
    -----END PGP SIGNATURE-----


+ Reply to Thread