Re: PCIe device driver question - Kernel

This is a discussion on Re: PCIe device driver question - Kernel ; V.Radhakrishnan wrote: >>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I >>> am able to successfully dma data into any memory (dma) address > >>> 0x0000_0001_0000_0000. > > How can you DMA "successfully" into ...

+ Reply to Thread
Results 1 to 6 of 6

Thread: Re: PCIe device driver question

  1. Re: PCIe device driver question

    V.Radhakrishnan wrote:
    >>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
    >>> am able to successfully dma data into any memory (dma) address >
    >>> 0x0000_0001_0000_0000.

    >
    > How can you DMA "successfully" into this address which is > 4 GB when
    > you have only 4 GB RAM ? Or am I missing something ?


    The MMIO and other reserved memory space at the top of the 32-bit memory
    space will cause the top part of memory to be relocated above 4GB.
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. Re: PCIe device driver question

    Hi Robert,

    Thanks for the reply. I was thinking that the MMIO and reserve memory
    will be below 4 GB was only applicable for 32-bit environments, since I
    don't have much experience in 64-bit.

    However, I had an IDENTICAL problem over 2 years ago. I had used
    posix_memalign() in user space to allocate pages aligned to 4096 byte
    pages, allocated several additional memaligned pages in user space, used
    mlock() to lock all these pages, gathered the user space addresses into
    the original pages as arrays of structures, passed this array into the
    kernel using an ioctl() call, used get_user_pages() to extract the
    struct page pointers, performed a kmap() to get the kernel virtual
    addresses and then extracted the physical addresses and 'sent' this to
    the chip to perform DMA.

    This situation is almost identical to what has been reported and hence
    my interest.

    However, I had a PCI access problem. The DMA was just NOT happening on
    any machine which had highmem, i.e over 896 MB.

    I "solved" the problem since I didn't have much time to do R&D, by
    booting with kernel command line option of mem=512M and the DMA went
    thru successfully.

    This was the linux-2.6.15 kernel then. Since the project was basically
    to test the DMA capability of the device, the actual address to where it
    was DMA-ed didn't matter, and I got paid for my work. However, this
    matter was always at the back of my head.

    What could have been the problem with the x86 32-bit PCI ?

    Thanks and regards

    V. Radhakrishnan
    www.atr-labs.com

    On Wed, 2008-07-30 at 13:21 -0600, Robert Han**** wrote:
    > V.Radhakrishnan wrote:
    > >>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
    > >>> am able to successfully dma data into any memory (dma) address >
    > >>> 0x0000_0001_0000_0000.

    > >
    > > How can you DMA "successfully" into this address which is > 4 GB when
    > > you have only 4 GB RAM ? Or am I missing something ?

    >
    > The MMIO and other reserved memory space at the top of the 32-bit memory
    > space will cause the top part of memory to be relocated above 4GB.
    > --
    > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > the body of a message to majordomo@vger.kernel.org
    > More majordomo info at http://vger.kernel.org/majordomo-info.html
    > Please read the FAQ at http://www.tux.org/lkml/


    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: PCIe device driver question

    V.Radhakrishnan wrote:
    > Hi Robert,
    >
    > Thanks for the reply. I was thinking that the MMIO and reserve memory
    > will be below 4 GB was only applicable for 32-bit environments, since I
    > don't have much experience in 64-bit.
    >
    > However, I had an IDENTICAL problem over 2 years ago. I had used
    > posix_memalign() in user space to allocate pages aligned to 4096 byte
    > pages, allocated several additional memaligned pages in user space, used
    > mlock() to lock all these pages, gathered the user space addresses into
    > the original pages as arrays of structures, passed this array into the
    > kernel using an ioctl() call, used get_user_pages() to extract the
    > struct page pointers, performed a kmap() to get the kernel virtual
    > addresses and then extracted the physical addresses and 'sent' this to
    > the chip to perform DMA.
    >
    > This situation is almost identical to what has been reported and hence
    > my interest.
    >
    > However, I had a PCI access problem. The DMA was just NOT happening on
    > any machine which had highmem, i.e over 896 MB.


    My guess there was a bug in your DMA mapping code. I don't think kmap is
    what is normally used for this. I think with get_user_pages one usually
    takes the returned page pointers to create an SG list and uses
    dma_map_sg to create a DMA mapping for them.

    >
    > I "solved" the problem since I didn't have much time to do R&D, by
    > booting with kernel command line option of mem=512M and the DMA went
    > thru successfully.
    >
    > This was the linux-2.6.15 kernel then. Since the project was basically
    > to test the DMA capability of the device, the actual address to where it
    > was DMA-ed didn't matter, and I got paid for my work. However, this
    > matter was always at the back of my head.
    >
    > What could have been the problem with the x86 32-bit PCI ?
    >
    > Thanks and regards
    >
    > V. Radhakrishnan
    > www.atr-labs.com
    >
    > On Wed, 2008-07-30 at 13:21 -0600, Robert Han**** wrote:
    >> V.Radhakrishnan wrote:
    >>>>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
    >>>>> am able to successfully dma data into any memory (dma) address >
    >>>>> 0x0000_0001_0000_0000.
    >>> How can you DMA "successfully" into this address which is > 4 GB when
    >>> you have only 4 GB RAM ? Or am I missing something ?

    >> The MMIO and other reserved memory space at the top of the 32-bit memory
    >> space will cause the top part of memory to be relocated above 4GB.
    >> --
    >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    >> the body of a message to majordomo@vger.kernel.org
    >> More majordomo info at http://vger.kernel.org/majordomo-info.html
    >> Please read the FAQ at http://www.tux.org/lkml/

    >
    >

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  4. Re: PCIe device driver question


    > My guess there was a bug in your DMA mapping code. I don't think kmap is
    > what is normally used for this. I think with get_user_pages one usually
    > takes the returned page pointers to create an SG list and uses
    > dma_map_sg to create a DMA mapping for them.


    Looking at the actual code, I see that I had used kmap() only to obtain
    kernel virtual addresses for the array of struct pages obtained from
    user space by using get_user_pages.

    Subsequently, I had used dma_map_single() and dma_unmap_single() calls
    for single buffer calls.

    The code didn't have bugs IMHO since it was used for extensive stress
    testing the initial FPGA prototype as well as the final ASIC chip ,
    sometimes running for over 4 days non-stop without breaking.

    However, using Test Access Points on the board and using a Logic
    Analyzer showed that DMA was NOT taking place when RAM > 896 MB was
    used. The hardware gurus said that PCI bus cycles just didn't seem to be
    taking place when RAM > 896 MB was used as the source OR destination
    address.

    Perhaps this was a problem in the earlier kernel(s) and has since been
    rectified ? ( I was using 2.6.15 then ... )

    I am just curious since Sanka Piyaratna reported a 'similar' kind of
    situation.

    Regards

    V. Radhakrishnan



    On Thu, 2008-07-31 at 11:37 -0600, Robert Han**** wrote:
    > V.Radhakrishnan wrote:
    > > Hi Robert,
    > >
    > > Thanks for the reply. I was thinking that the MMIO and reserve memory
    > > will be below 4 GB was only applicable for 32-bit environments, since I
    > > don't have much experience in 64-bit.
    > >
    > > However, I had an IDENTICAL problem over 2 years ago. I had used
    > > posix_memalign() in user space to allocate pages aligned to 4096 byte
    > > pages, allocated several additional memaligned pages in user space, used
    > > mlock() to lock all these pages, gathered the user space addresses into
    > > the original pages as arrays of structures, passed this array into the
    > > kernel using an ioctl() call, used get_user_pages() to extract the
    > > struct page pointers, performed a kmap() to get the kernel virtual
    > > addresses and then extracted the physical addresses and 'sent' this to
    > > the chip to perform DMA.
    > >
    > > This situation is almost identical to what has been reported and hence
    > > my interest.
    > >
    > > However, I had a PCI access problem. The DMA was just NOT happening on
    > > any machine which had highmem, i.e over 896 MB.

    >
    > My guess there was a bug in your DMA mapping code. I don't think kmap is
    > what is normally used for this. I think with get_user_pages one usually
    > takes the returned page pointers to create an SG list and uses
    > dma_map_sg to create a DMA mapping for them.
    >
    > >
    > > I "solved" the problem since I didn't have much time to do R&D, by
    > > booting with kernel command line option of mem=512M and the DMA went
    > > thru successfully.
    > >
    > > This was the linux-2.6.15 kernel then. Since the project was basically
    > > to test the DMA capability of the device, the actual address to where it
    > > was DMA-ed didn't matter, and I got paid for my work. However, this
    > > matter was always at the back of my head.
    > >
    > > What could have been the problem with the x86 32-bit PCI ?
    > >
    > > Thanks and regards
    > >
    > > V. Radhakrishnan
    > > www.atr-labs.com
    > >
    > > On Wed, 2008-07-30 at 13:21 -0600, Robert Han**** wrote:
    > >> V.Radhakrishnan wrote:
    > >>>>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
    > >>>>> am able to successfully dma data into any memory (dma) address >
    > >>>>> 0x0000_0001_0000_0000.
    > >>> How can you DMA "successfully" into this address which is > 4 GB when
    > >>> you have only 4 GB RAM ? Or am I missing something ?
    > >> The MMIO and other reserved memory space at the top of the 32-bit memory
    > >> space will cause the top part of memory to be relocated above 4GB.
    > >> --
    > >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > >> the body of a message to majordomo@vger.kernel.org
    > >> More majordomo info at http://vger.kernel.org/majordomo-info.html
    > >> Please read the FAQ at http://www.tux.org/lkml/

    > >
    > >

    > --
    > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > the body of a message to majordomo@vger.kernel.org
    > More majordomo info at http://vger.kernel.org/majordomo-info.html
    > Please read the FAQ at http://www.tux.org/lkml/


    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: PCIe device driver question

    V.Radhakrishnan wrote:
    >> My guess there was a bug in your DMA mapping code. I don't think kmap is
    >> what is normally used for this. I think with get_user_pages one usually
    >> takes the returned page pointers to create an SG list and uses
    >> dma_map_sg to create a DMA mapping for them.

    >
    > Looking at the actual code, I see that I had used kmap() only to obtain
    > kernel virtual addresses for the array of struct pages obtained from
    > user space by using get_user_pages.
    >
    > Subsequently, I had used dma_map_single() and dma_unmap_single() calls
    > for single buffer calls.


    I'm suspicious about this usage, I don't know if that will actually
    work. There is a dma_map_page call which is meant for doing a DMA
    mapping on a struct page which should likely be used instead.

    >
    > The code didn't have bugs IMHO since it was used for extensive stress
    > testing the initial FPGA prototype as well as the final ASIC chip ,
    > sometimes running for over 4 days non-stop without breaking.
    >
    > However, using Test Access Points on the board and using a Logic
    > Analyzer showed that DMA was NOT taking place when RAM > 896 MB was
    > used. The hardware gurus said that PCI bus cycles just didn't seem to be
    > taking place when RAM > 896 MB was used as the source OR destination
    > address.


    Are you sure the address being passed to the device was correct in this
    case? There should be nothing magical about 896MB from a hardware point
    of view, and the kernel in general cannot stop you from DMAing anywhere
    you like.

    >
    > Perhaps this was a problem in the earlier kernel(s) and has since been
    > rectified ? ( I was using 2.6.15 then ... )
    >
    > I am just curious since Sanka Piyaratna reported a 'similar' kind of
    > situation.
    >
    > Regards
    >
    > V. Radhakrishnan

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. Re: PCIe device driver question

    Thanks Robert !

    Unfortunately, I cannot test this with recent kernels, since this was a
    WiMAX card based on a CardBus inteface and the Toshiba laptop I have
    has only 512 MB RAM ( though I have PCs with more RAM, the PCs don't
    have a CardBus interface ).

    BRgds

    RK


    On Thu, 2008-07-31 at 12:52 -0600, Robert Han**** wrote:
    > V.Radhakrishnan wrote:
    > >> My guess there was a bug in your DMA mapping code. I don't think kmap is
    > >> what is normally used for this. I think with get_user_pages one usually
    > >> takes the returned page pointers to create an SG list and uses
    > >> dma_map_sg to create a DMA mapping for them.

    > >
    > > Looking at the actual code, I see that I had used kmap() only to obtain
    > > kernel virtual addresses for the array of struct pages obtained from
    > > user space by using get_user_pages.
    > >
    > > Subsequently, I had used dma_map_single() and dma_unmap_single() calls
    > > for single buffer calls.

    >
    > I'm suspicious about this usage, I don't know if that will actually
    > work. There is a dma_map_page call which is meant for doing a DMA
    > mapping on a struct page which should likely be used instead.
    >
    > >
    > > The code didn't have bugs IMHO since it was used for extensive stress
    > > testing the initial FPGA prototype as well as the final ASIC chip ,
    > > sometimes running for over 4 days non-stop without breaking.
    > >
    > > However, using Test Access Points on the board and using a Logic
    > > Analyzer showed that DMA was NOT taking place when RAM > 896 MB was
    > > used. The hardware gurus said that PCI bus cycles just didn't seem to be
    > > taking place when RAM > 896 MB was used as the source OR destination
    > > address.

    >
    > Are you sure the address being passed to the device was correct in this
    > case? There should be nothing magical about 896MB from a hardware point
    > of view, and the kernel in general cannot stop you from DMAing anywhere
    > you like.
    >
    > >
    > > Perhaps this was a problem in the earlier kernel(s) and has since been
    > > rectified ? ( I was using 2.6.15 then ... )
    > >
    > > I am just curious since Sanka Piyaratna reported a 'similar' kind of
    > > situation.
    > >
    > > Regards
    > >
    > > V. Radhakrishnan

    > --
    > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > the body of a message to majordomo@vger.kernel.org
    > More majordomo info at http://vger.kernel.org/majordomo-info.html
    > Please read the FAQ at http://www.tux.org/lkml/


    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread