-->

Data bus error when using ioread32 in PCIE driver

2019-08-26 11:25发布

问题:

I am developing a PCIE device driver for Openwrt, which is also a linux system. Here is a weird situation. After Initializing the driver in probe function, I can read(by ioread32) correct data (preset value:123456) from the buffer address obtained from ioremap_nocache. Then when I try to read it every 1 second in periodic timer interrupt handler, the function ioread32 will crash and the serial console presents a Data bus error. Below are code details.

// content of function my_driver_request_mem, this function is called in probe function
int my_driver_request_mem(struct gps_time *gt) {
    u32 start, len;
    int ret;
    int bar = 0;
    u32 flags;

    ret = pcim_iomap_regions(gt->pdev, BIT(0), "My Driver");
    if (ret) {
        gt_log("Fail to request IO mem: err: %d\n", ret);
        return ret;
    }

    // gt is a custom struct, and gt->pdev is the pci_dev struct
    // obtained from probe function
    start = pci_resource_start(gt->pdev, bar);
    len = pci_resource_len(gt->pdev, bar);
    flags = pci_resource_flags(gt->pdev, bar);
    printk(KERN_ALERT "region start: 0x%x, len: %u\n", start, len);
    printk(KERN_ALERT "region flags: 0x%x\n", flags);
    gt->buffer = ioremap_nocache(start, len);
    gt->buffer_len = len;
    gt->buffer_start = start;
    return 0;
}

Afte the function above is invoked, I read data through gt->buffer:

u32 d = 0;
d = ioread32(gt->buffer);    // this operation does not cause fatal error
printk(KERN_ALERT "initial value is: %u", d);

By reading the console output, the ioread32 here is successful, and the right value 123456 is printed. Then I start a timer to read data multiple times

setup_timer(&gt->g_timer, _gps_timer_handler, gt);
mod_timer(&gt->g_timer, jiffies + msecs_to_jiffies(20000));
printk(KERN_ALERT "GPS_TIME: timer created.\n");

The handler function is quit simple:

void _gps_timer_handler(unsigned long data) {
    struct gps_time *gt = (struct gps_time*)data;
    u32 d;
    d = ioread32(gt->buffer);   // fatal error in this line
    printk(KERN_ALERT "Value: %u\n", d);
    mod_timer(&gt->g_timer, jiffies + msecs_to_jiffies(1000));
}

The ioread32 here will cause a fatal error here, and the error info is:

Data bus error, epc == 82db8030, ra == 8009a000
Oops[#1]
CPU: 0 PID: 853 Comm: dropbearkey Tainted: G        W       4.4.14 #2
task: 82dd1900 ti:8209a000 task.ti: 8209a000
...
(bunch of numbers)
...
Status: 1100d403 KERNEL EXL IE
Cause : 8080001c (ExcCode 07)
PrId:   0001974c (MIPS 74Kc)
...

First I though this is because IO process should not be done in interrupt context, so I put ioread32 in a tasklet, and invoke that tasklet by tasklet_schedule, but it still fails.

======== Update A. Below are my ->probe function:

static int gps_time_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
    int ret = 0;
    struct gps_time *gt;
    u8 tmp = 0;
    u32 bar_val = 0;
    u8 csz = 0;
    unsigned long start, end, len, flag;
    int bar = 0;

    if (gps_time_global_time) {
        printk(KERN_ALERT "GPS_TIME: more than one device detected\n");
        return -1;
    }

    gt = gps_time_alloc();
    if (gt == NULL) {
        printk(KERN_WARNING "GPS_TIME: out of memory\n");
        return -ENOMEM;
    }

    gt->pdev = pdev;
    gt->irq = pdev->irq;

    ret = pcim_enable_device(pdev);
    if (ret) {
        printk(KERN_ALERT "GPS_TIME: Fail to enable device %d\n", ret);
        goto err;
    }

    ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
    if (ret) {
        printk(KERN_WARNING "GPS_TIME: 32-bit DMA not available\n");
        return ret;
    }

    ret = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
    if (ret) {
        printk(KERN_WARNING "GPS_TIME: 32-bit DMA consistent DMA enable failed\n");
        return ret;
    }

    my_driver_request_mem(gt);

    ret = pci_read_config_byte(pdev, PCI_CACHE_LINE_SIZE, &tmp);
    if (ret) {
        printk(KERN_ALERT "GPS_TIME: Fail to read cache line size\n");
        goto err;
    }

    if (tmp == 0) {
        printk(KERN_ALERT "GPS_TIME: Write pci cache line size\n");
        pci_write_config_byte(
            pdev, PCI_CACHE_LINE_SIZE, L1_CACHE_BYTES / sizeof(u32));
    }
    pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0xa8);

    pci_set_master(pdev);
    pci_set_drvdata(pdev, gt);

    // This function is very simple. I just create timer here. The first ioread32 is also included in this function.
    gps_time_init_device(gt);

    ret = request_irq(pdev->irq, gps_time_isq, IRQF_SHARED, "gps_time", gt);
    if (ret) {
        printk(KERN_ALERT "GPS_TIME: Fail to request IRQ: %d", ret);
        goto err_irq;
    }
    return 0;
err_region:
    pci_release_regions(pdev);
err_irq:
err:
    gps_time_free(gt);
    return ret;
}

B. More info about the device:

This device is a self-designed chip with PCIE interface. It is built around a Altera Cyclone IV FPGA. The firmware in the chip does nothing except writing constant 123456 into its memory.