[PATCH 0/16 v6] PCI: Linux kernel SR-IOV support - Kernel

This is a discussion on [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support - Kernel ; On Thu, Nov 06, 2008 at 04:40:21PM -0600, Anthony Liguori wrote: > We've been talking about avoiding hardware passthrough entirely and > just backing a virtio-net backend driver by a dedicated VF in the > host. That avoids a huge ...

+ Reply to Thread
Page 4 of 4 FirstFirst ... 2 3 4
Results 61 to 70 of 70

Thread: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

  1. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    On Thu, Nov 06, 2008 at 04:40:21PM -0600, Anthony Liguori wrote:

    > We've been talking about avoiding hardware passthrough entirely and
    > just backing a virtio-net backend driver by a dedicated VF in the
    > host. That avoids a huge amount of guest-facing complexity, let's
    > migration Just Work, and should give the same level of performance.


    I don't believe that it will, and every benchmark I've seen or have
    done so far shows a significant performance gap between virtio and
    direct assignment, even on 1G ethernet. I am willing however to
    reserve judgement until someone implements your suggestion and
    actually measures it, preferably on 10G ethernet.

    No doubt device assignment---and SR-IOV in particular---are complex,
    but I hardly think ignoring it as you seem to propose is the right
    approach.

    Cheers,
    Muli
    --
    The First Workshop on I/O Virtualization (WIOV '08)
    Dec 2008, San Diego, CA, http://www.usenix.org/wiov08/
    <->
    SYSTOR 2009---The Israeli Experimental Systems Conference
    http://www.haifa.il.ibm.com/conferences/systor2009/
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    Greg KH wrote:
    > It's that "second" part that I'm worried about. How is that going to
    > happen? Do you have any patches that show this kind of "assignment"?
    >
    >


    For kvm, this is in 2.6.28-rc.

    Note there are two ways to assign a device to a guest:

    - run the VF driver in the guest: this has the advantage of best
    performance, but requires pinning all guest memory, makes live migration
    a tricky proposition, and ties the guest to the underlying hardware.
    - run the VF driver in the host, and use virtio to connect the guest to
    the host: allows paging the guest and allows straightforward live
    migration, but reduces performance, and hides any features not exposed
    by virtio from the guest.


    --
    error compiling committee.c: too many arguments to function

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    Andi Kleen wrote:
    > Anthony Liguori writes:
    >
    >> What we would rather do in KVM, is have the VFs appear in the host as
    >> standard network devices. We would then like to back our existing PV
    >> driver to this VF directly bypassing the host networking stack. A key
    >> feature here is being able to fill the VF's receive queue with guest
    >> memory instead of host kernel memory so that you can get zero-copy
    >> receive traffic. This will perform just as well as doing passthrough
    >> (at least) and avoid all that ugliness of dealing with SR-IOV in the
    >> guest.
    >>

    >
    > But you shift a lot of ugliness into the host network stack again.
    > Not sure that is a good trade off.
    >


    The net effect will be positive. We will finally have aio networking
    from userspace (can send process memory without resorting to
    sendfile()), and we'll be able to assign a queue to a process (which
    will enable all sorts of interesting high performance things; basically
    VJ channels without kernel involvement).

    > Also it would always require context switches and I believe one
    > of the reasons for the PV/VF model is very low latency IO and having
    > heavyweight switches to the host and back would be against that.
    >


    It's true that latency would suffer (or alternatively cpu consumption
    would increase).

    --
    error compiling committee.c: too many arguments to function

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  4. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    Greg KH wrote:
    >> We've been talking about avoiding hardware passthrough entirely and
    >> just backing a virtio-net backend driver by a dedicated VF in the
    >> host. That avoids a huge amount of guest-facing complexity, let's
    >> migration Just Work, and should give the same level of performance.
    >>

    >
    > Does that involve this patch set? Or a different type of interface.
    >


    So long as the VF is exposed as a standalone PCI device, it's the same
    interface. In fact you can take a random PCI card and expose it to a
    guest this way; it doesn't have to be SR-IOV. Of course, with a
    standard PCI card you won't get much sharing (a quad port NIC will be
    good for four guests).

    We'll need other changes in the network stack, but these are orthogonal
    to SR-IOV.

    --
    error compiling committee.c: too many arguments to function

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    Matthew Wilcox wrote:
    >> What we would rather do in KVM, is have the VFs appear in the host as
    >> standard network devices. We would then like to back our existing PV
    >> driver to this VF directly bypassing the host networking stack. A key
    >> feature here is being able to fill the VF's receive queue with guest
    >> memory instead of host kernel memory so that you can get zero-copy
    >> receive traffic. This will perform just as well as doing passthrough
    >> (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
    >>

    >
    > This argues for ignoring the SR-IOV mess completely.


    It does, but VF-in-host is not the only model that we want to support.
    It's just the most appealing.

    There will definitely be people who want to run VF-in-guest.

    --
    error compiling committee.c: too many arguments to function

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    Muli Ben-Yehuda wrote:
    >> We've been talking about avoiding hardware passthrough entirely and
    >> just backing a virtio-net backend driver by a dedicated VF in the
    >> host. That avoids a huge amount of guest-facing complexity, let's
    >> migration Just Work, and should give the same level of performance.
    >>

    >
    > I don't believe that it will, and every benchmark I've seen or have
    > done so far shows a significant performance gap between virtio and
    > direct assignment, even on 1G ethernet. I am willing however to
    > reserve judgement until someone implements your suggestion and
    > actually measures it, preferably on 10G ethernet.
    >


    Right now virtio copies data, and has other inefficiencies. With a
    dedicated VF, we can eliminate the copies.

    CPU utilization and latency will be worse. If we can limit the
    slowdowns to an acceptable amount, the simplicity and other advantages
    of VF-in-host may outweigh the performance degradation.

    > No doubt device assignment---and SR-IOV in particular---are complex,
    > but I hardly think ignoring it as you seem to propose is the right
    > approach.


    I agree. We should hedge our bets and support both models.

    --
    error compiling committee.c: too many arguments to function

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  7. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    On Sun, Nov 09, 2008 at 02:44:06PM +0200, Avi Kivity wrote:
    > Greg KH wrote:
    >> It's that "second" part that I'm worried about. How is that going to
    >> happen? Do you have any patches that show this kind of "assignment"?
    >>
    >>

    >
    > For kvm, this is in 2.6.28-rc.


    Where? I just looked and couldn't find anything, but odds are I was
    looking in the wrong place

    > Note there are two ways to assign a device to a guest:
    >
    > - run the VF driver in the guest: this has the advantage of best
    > performance, but requires pinning all guest memory, makes live migration a
    > tricky proposition, and ties the guest to the underlying hardware.


    Is this what you would prefer for kvm?

    > - run the VF driver in the host, and use virtio to connect the guest to the
    > host: allows paging the guest and allows straightforward live migration,
    > but reduces performance, and hides any features not exposed by virtio from
    > the guest.


    thanks,

    greg k-h
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  8. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    Greg KH wrote:
    > On Sun, Nov 09, 2008 at 02:44:06PM +0200, Avi Kivity wrote:
    >
    >> Greg KH wrote:
    >>
    >>> It's that "second" part that I'm worried about. How is that going to
    >>> happen? Do you have any patches that show this kind of "assignment"?
    >>>
    >>>
    >>>

    >> For kvm, this is in 2.6.28-rc.
    >>

    >
    > Where? I just looked and couldn't find anything, but odds are I was
    > looking in the wrong place
    >
    >


    arch/x86/kvm/vtd.c: iommu integration (allows assigning the device's
    memory resources)
    virt/kvm/irq*: interrupt redirection (allows assigning the device's
    interrupt resources)

    the rest (pci config space, pio redirection) are in userspace.

    >> Note there are two ways to assign a device to a guest:
    >>
    >> - run the VF driver in the guest: this has the advantage of best
    >> performance, but requires pinning all guest memory, makes live migration a
    >> tricky proposition, and ties the guest to the underlying hardware.
    >>

    >
    > Is this what you would prefer for kvm?
    >
    >


    It's not my personal preference, but it is a supported configuration.
    For some use cases it is the only one that makes sense.

    Again, VF-in-guest and VF-in-host both have their places. And since
    Linux can be both guest and host, it's best if the VF driver knows
    nothing about SR-IOV; it's just a pci driver. The PF driver should
    emulate anything that SR-IOV does not provide (like missing pci config
    space).

    --
    I have a truly marvellous patch that fixes the bug which this
    signature is too narrow to contain.

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  9. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    On Sun, Nov 09, 2008 at 09:37:20PM +0200, Avi Kivity wrote:
    > Greg KH wrote:
    >> On Sun, Nov 09, 2008 at 02:44:06PM +0200, Avi Kivity wrote:
    >>
    >>> Greg KH wrote:
    >>>
    >>>> It's that "second" part that I'm worried about. How is that going to
    >>>> happen? Do you have any patches that show this kind of "assignment"?
    >>>>
    >>>>
    >>> For kvm, this is in 2.6.28-rc.
    >>>

    >>
    >> Where? I just looked and couldn't find anything, but odds are I was
    >> looking in the wrong place
    >>
    >>

    >
    > arch/x86/kvm/vtd.c: iommu integration (allows assigning the device's memory
    > resources)


    That file is not in 2.6.28-rc4


    > virt/kvm/irq*: interrupt redirection (allows assigning the device's
    > interrupt resources)


    I only see virt/kvm/irq_comm.c in 2.6.28-rc4.

    > the rest (pci config space, pio redirection) are in userspace.


    So you don't need these pci core changes at all?

    >>> Note there are two ways to assign a device to a guest:
    >>>
    >>> - run the VF driver in the guest: this has the advantage of best
    >>> performance, but requires pinning all guest memory, makes live migration
    >>> a tricky proposition, and ties the guest to the underlying hardware.

    >>
    >> Is this what you would prefer for kvm?
    >>

    >
    > It's not my personal preference, but it is a supported configuration. For
    > some use cases it is the only one that makes sense.
    >
    > Again, VF-in-guest and VF-in-host both have their places. And since Linux
    > can be both guest and host, it's best if the VF driver knows nothing about
    > SR-IOV; it's just a pci driver. The PF driver should emulate anything that
    > SR-IOV does not provide (like missing pci config space).


    Yes, we need both.

    thanks,

    greg k-h
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  10. Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support

    Greg KH wrote:



    >> arch/x86/kvm/vtd.c: iommu integration (allows assigning the device's memory
    >> resources)
    >>

    >
    > That file is not in 2.6.28-rc4
    >
    >


    Sorry, was moved to virt/kvm/ for ia64's benefit.

    >
    >> virt/kvm/irq*: interrupt redirection (allows assigning the device's
    >> interrupt resources)
    >>

    >
    > I only see virt/kvm/irq_comm.c in 2.6.28-rc4.
    >
    >


    kvm_main.c in that directory also has some related bits.

    >> the rest (pci config space, pio redirection) are in userspace.
    >>

    >
    > So you don't need these pci core changes at all?
    >
    >


    Not beyond those required for SR-IOV and iommu support.

    --
    I have a truly marvellous patch that fixes the bug which this
    signature is too narrow to contain.

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread
Page 4 of 4 FirstFirst ... 2 3 4