PCI device driver file operations question - Linux

This is a discussion on PCI device driver file operations question - Linux ; In a previous entry, I wrote about how I used a userspace C program to read and write 32-bit binary data from/to custom PCI device memory by opening /dev/mem and mmaping the memory range that I want. This program worked ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: PCI device driver file operations question

  1. PCI device driver file operations question

    In a previous entry, I wrote about how I used a userspace C program to
    read and write 32-bit binary data from/to custom PCI device memory by
    opening /dev/mem and mmaping the memory range that I want. This program
    worked well.

    To try something similar, I wrote a PCI device driver and a userspace
    program. The PCI device driver contains file operations calls. The PCI
    device driver contains the following file operation functions:

    int test_fopsopen(struct inode *inode, struct file *file){
    return 0;
    }

    int test_fopsrelease(struct inode *inode, struct file *file){
    return 0;
    }

    ssize_t test_fopsread(struct file *file, char *buf, size_t count,
    loff_t *ppos) {
    u32 readData;
    char* region0;
    int offset = 0;
    u32 bar0;
    unsigned int barLen;

    struct pci_dev *dev;
    dev = pci_get_device(TEST_VENDORID, TEST_DEVICEID, NULL);
    if (dev) {
    pci_request_region(dev, 0, "testdriver");
    bar0 = pci_resource_start(dev, 0);
    barLen = pci_resource_len(dev, 0);
    region0 = ioremap_nocache(bar0, barLen);

    readData = readl(region0+offset);
    iounmap(region0);
    pci_dev_put(dev);

    if (copy_to_user(buf, &readData, sizeof(u32))) return -EFAULT;

    return 0;
    }
    else {
    printk(KERN_ALERT "Error in reading data at BA0\n");
    return 1;
    }
    }

    ssize_t test_fopswrite(struct file *file, char *buf, size_t count,
    loff_t *ppos) {
    u32 writeData;
    char* region0;
    int offset = 0;
    u32 bar0;
    unsigned int barLen;

    struct pci_dev *dev;
    dev = pci_get_device(TEST_VENDORID, TEST_DEVICEID, NULL);
    if (dev) {
    pci_request_region(dev, 0, "testdriver");
    bar0 = pci_resource_start(dev, 0);
    barLen = pci_resource_len(dev, 0);
    region0 = ioremap_nocache(bar0, barLen);

    if (copy_from_user(&writeData, buf, sizeof(writeData))) return
    -EFAULT;
    writel(wroteData, region0+offset);
    iounmap(region0);
    pci_dev_put(dev);
    return 0;
    }
    else {
    printk(KERN_ALERT "ERROR in writing data at BA0\n");
    return 1;
    }
    }

    where TEST_VENDORID and TEST_DEVICEID are defined by the Vendor ID and
    Device ID of the PCI device. My user program is a simple read test:

    int main(){
    int fd, len;
    char c[32];

    fd = open("/dev/test", O_RDWR);
    if (fd == -1) {
    printf("Error in opening /dev/test\n");
    return 1;
    }

    len = read(fd, c, 32);
    printf("Result: %s", c);

    close(fd);
    }

    Both the user program and the device driver compiled with no errors.

    Some questions:

    1) I went ahead and "mknod /dev/test c XXX 0" where XXX is the PCI
    device major number. I also "chmod 666 /dev/test" so that anyone can
    access it. After I insmod the kernel module, I went ahead to run the
    user program. The user program gave me the following error: Error in
    opening /dev/test. What caused the error to occur?

    2) How does the functions in the device driver and the user program
    look? Are there any improvements that can be made?

    3) Eventually, I want to be able to read and write to different offsets
    from base address 0. How can I send the offset number that I want from
    userspace?

    4) If I want the device driver to load automatically from bootup, how
    do I do that? What script do I need to edit?


  2. Re: PCI device driver file operations question

    I realized what happened. My init function was not working. I read
    through LDD 3rd Ed to see how to make the init function for a char
    device. Now this is how my read function looks like:

    ssize_t test_fopsread(struct file *file, char *buf, size_t count,
    loff_t *f_pos) {
    u32 readData;
    char* region0;
    u32 bar0;
    unsigned int barLen;

    printk(KERN_ALERT "We are reading from fops\n");
    printk(KERN_ALERT "This is the offset: 0x%x\n", *f_pos);

    struct pci_dev *dev;
    dev = pci_get_device(RPM_VENDORID, RPM_DEVICEID, NULL);
    if (dev) {
    pci_request_region(dev, 0, "testdriver");
    bar0 = pci_resource_start(dev, 0);
    barLen = pci_resource_len(dev, 0);
    region0 = ioremap_nocache(bar0, barLen);

    readData = readl(region0 + *f_pos);
    printk(KERN_ALERT "We read %x\n", readData);

    iounmap(region0);
    pci_dev_put(dev);

    if (copy_to_user(buf, &readData, sizeof(u32))) return -EFAULT;

    return 0;
    }
    else {
    printk(KERN_ALERT "Error in reading data at BA0\n");
    return 1;
    }
    }

    This is the code in userspace:
    unsigned long offset;
    int fd = open("/dev/test", O_RDWR);
    if (fd == -1)
    printf("Error in opening /dev/test\n");
    return 1;
    }

    offset = lseek(fd, 0x0, SEEK_SET);
    pread(fd, &readData, sizeof(unsigned long), offset);
    printf("Result: %x\n", readData);

    offset = lseek(fd, 0x08, SEEK_SET);
    pread(fd, &readData, sizeof(unsigned long), offset);
    printf("Result: %x\n", readData);

    I have been successful in reading the data.

    Two questions:
    1) How does the read function look? Whenever I do multiple reads, it
    sometimes throw me the following error: PCI: Unable to reserve mem
    region ... Is there any other function that I am supposed to call in
    the read to release the memory region?

    2) Eventually, I want to be able to read/write to different base
    addresses. Right now, I can only read/write to base address zero.
    This is what I tried:
    In the device driver, I got rid of pci_request_region,
    pci_resouce_start, and pci_resource_len. I ioremap_nocache to (*fops,
    0x04).
    In userspace, instead of lseek to an offset, I lseek to the base
    address plus offset. I send that in to pread.
    This did not work. What is the correct method to read/write to
    different base addresses?


    > In a previous entry, I wrote about how I used a userspace C program to
    > read and write 32-bit binary data from/to custom PCI device memory by
    > opening /dev/mem and mmaping the memory range that I want. This program
    > worked well.
    >
    > To try something similar, I wrote a PCI device driver and a userspace
    > program. The PCI device driver contains file operations calls. The PCI
    > device driver contains the following file operation functions:
    >
    > int test_fopsopen(struct inode *inode, struct file *file){
    > return 0;
    > }
    >
    > int test_fopsrelease(struct inode *inode, struct file *file){
    > return 0;
    > }
    >
    > ssize_t test_fopsread(struct file *file, char *buf, size_t count,
    > loff_t *ppos) {
    > u32 readData;
    > char* region0;
    > int offset = 0;
    > u32 bar0;
    > unsigned int barLen;
    >
    > struct pci_dev *dev;
    > dev = pci_get_device(TEST_VENDORID, TEST_DEVICEID, NULL);
    > if (dev) {
    > pci_request_region(dev, 0, "testdriver");
    > bar0 = pci_resource_start(dev, 0);
    > barLen = pci_resource_len(dev, 0);
    > region0 = ioremap_nocache(bar0, barLen);
    >
    > readData = readl(region0+offset);
    > iounmap(region0);
    > pci_dev_put(dev);
    >
    > if (copy_to_user(buf, &readData, sizeof(u32))) return -EFAULT;
    >
    > return 0;
    > }
    > else {
    > printk(KERN_ALERT "Error in reading data at BA0\n");
    > return 1;
    > }
    > }
    >
    > ssize_t test_fopswrite(struct file *file, char *buf, size_t count,
    > loff_t *ppos) {
    > u32 writeData;
    > char* region0;
    > int offset = 0;
    > u32 bar0;
    > unsigned int barLen;
    >
    > struct pci_dev *dev;
    > dev = pci_get_device(TEST_VENDORID, TEST_DEVICEID, NULL);
    > if (dev) {
    > pci_request_region(dev, 0, "testdriver");
    > bar0 = pci_resource_start(dev, 0);
    > barLen = pci_resource_len(dev, 0);
    > region0 = ioremap_nocache(bar0, barLen);
    >
    > if (copy_from_user(&writeData, buf, sizeof(writeData))) return
    > -EFAULT;
    > writel(wroteData, region0+offset);
    > iounmap(region0);
    > pci_dev_put(dev);
    > return 0;
    > }
    > else {
    > printk(KERN_ALERT "ERROR in writing data at BA0\n");
    > return 1;
    > }
    > }
    >
    > where TEST_VENDORID and TEST_DEVICEID are defined by the Vendor ID and
    > Device ID of the PCI device. My user program is a simple read test:
    >
    > int main(){
    > int fd, len;
    > char c[32];
    >
    > fd = open("/dev/test", O_RDWR);
    > if (fd == -1) {
    > printf("Error in opening /dev/test\n");
    > return 1;
    > }
    >
    > len = read(fd, c, 32);
    > printf("Result: %s", c);
    >
    > close(fd);
    > }
    >
    > Both the user program and the device driver compiled with no errors.
    >
    > Some questions:
    >
    > 1) I went ahead and "mknod /dev/test c XXX 0" where XXX is the PCI
    > device major number. I also "chmod 666 /dev/test" so that anyone can
    > access it. After I insmod the kernel module, I went ahead to run the
    > user program. The user program gave me the following error: Error in
    > opening /dev/test. What caused the error to occur?
    >
    > 2) How does the functions in the device driver and the user program
    > look? Are there any improvements that can be made?
    >
    > 3) Eventually, I want to be able to read and write to different offsets
    > from base address 0. How can I send the offset number that I want from
    > userspace?
    >
    > 4) If I want the device driver to load automatically from bootup, how
    > do I do that? What script do I need to edit?



  3. Re: PCI device driver file operations question

    Elliot wrote:

    > ssize_t test_fopsread(struct file *file, char *buf, size_t count,
    > loff_t *f_pos) {


    > struct pci_dev *dev;
    > dev = pci_get_device(RPM_VENDORID, RPM_DEVICEID, NULL);
    > if (dev) {
    > pci_request_region(dev, 0, "testdriver");


    > 1) How does the read function look? Whenever I do multiple reads, it
    > sometimes throw me the following error: PCI: Unable to reserve mem
    > region ... Is there any other function that I am supposed to call in
    > the read to release the memory region?


    The companion to pci_request_region is pci_release_region.

    Generally, though, one doesn't do this stuff within the read function.
    You typically do this in your open function (and the release in the
    "close" [actually release] function). That is, when the user program
    opens your abstract device, you then claim the physical device with
    pci_request_region. When the user program closes the device, your
    release function gets called; your release in turn releases the
    physical device.

    > 2) Eventually, I want to be able to read/write to different base
    > addresses. Right now, I can only read/write to base address zero.
    > This is what I tried:
    > In the device driver, I got rid of pci_request_region,
    > pci_resouce_start, and pci_resource_len. I ioremap_nocache to (*fops,
    > 0x04).
    > In userspace, instead of lseek to an offset, I lseek to the base
    > address plus offset. I send that in to pread.
    > This did not work. What is the correct method to read/write to
    > different base addresses?


    (a) There is no completely standard way of handling multiple BARs --
    you need to create an appropriate abstraction yourself. One fairly
    clean option in this case would be for your driver to create multiple
    minor devices, one for each BAR in the device. Then, the user program
    could open "/dev/mydev_bar0", "/dev/mydev_bar1" and so forth.

    (b) If you're mapping the BAR within kernel virtual address space, the
    user program shouldn't need to know anything about the physical
    addresses in the device. And it doesn't really make sense for the user
    program to use those as arguments to lseek. The physical addresses are
    hidden behind the abstraction presented by your driver. As far as the
    user program is concerned, your device address range is from 0 to N - 1
    (where N is the BAR size).

    One other note: if your device has a large address space (I seem to
    recall seeing 512M in one of your posts), you don't need to and should
    not map the entire thing. Instead, calculate the amount needed to
    fulfill a particular request or even part of a request. Then call
    ioremap / iounmap for each chunk. For example, if the user program
    seeks to offset 0x5050 and writes 0x1880 bytes, then you only need to
    map from (BAR0 + 0x5000) through (BAR0 + 0x6BFF) inclusive.

    (It may sometimes be possible to map it all, but bear in mind, there is
    limited kernel virtual address space; on the typical x86 configuration,
    the total kernel virtual address space is limited to around 900M and
    your driver shares the available space with the main kernel and all the
    other drivers.)

    GH


+ Reply to Thread