How to access physical addresses from user space i

Question

On a ARM based system running Linux, I have a device that's memory mapped to a physical address. From a user space program where all addresses are virtual, how can I read content from this address?

Answer 1

busybox devmem

busybox devmem is a tiny CLI utility that mmaps /dev/mem.

You can get it in Ubuntu with: sudo apt-get install busybox

Usage: read 4 bytes from the physical address 0x12345678:

sudo busybox devmem 0x12345678

Write 0x9abcdef0 to that address:

sudo busybox devmem 0x12345678 w 0x9abcdef0

Source: https://github.com/mirror/busybox/blob/1_27_2/miscutils/devmem.c#L85

mmap MAP_SHARED

When mmapping /dev/mem, you likely want to use:

open("/dev/mem", O_RDWR | O_SYNC);
mmap(..., PROT_READ | PROT_WRITE, MAP_SHARED, ...)

MAP_SHARED makes writes go to physical memory immediately, which makes it easier to observe, and makes more sense for hardware register writes.

CONFIG_STRICT_DEVMEM and nopat

To use /dev/mem to view and modify regular RAM on kernel v4.9, you must fist:

• disable CONFIG_STRICT_DEVMEM (set by default on Ubuntu 17.04)
• pass the nopat kernel command line option for x86

IO ports still work without those.

See also: mmap of /dev/mem fails with invalid argument for virt_to_phys address, but address is page aligned

Cache flushing

If you try to write to RAM instead of a register, the memory may be cached by the CPU: How to flush the CPU cache for a region of address space in Linux? and I don't see a very portable / easy way to flush it or mark the region as uncacheable:

• How to write kernel space memory (physical address) to a file using O_DIRECT?
• How to flush the CPU cache for a region of address space in Linux?
• Is it possible to allocate, in user space, a non cacheable block of memory on Linux?

So maybe /dev/mem can't be used reliably to pass memory buffers to devices?

This can't be observed in QEMU unfortunately, since QEMU does not simulate caches.

How to test it out

Now for the fun part. Here are a few cool setups:

• Userland memory
  • allocate volatile variable on an userland process
  • get the physical address with /proc/<pid>/maps + /proc/<pid>/pagemap
  • modify the value at the physical address with devmem, and watch the userland process react
• Kernelland memory
  • allocate kernel memory with kmalloc
  • get the physical address with virt_to_phys and pass it back to userland
  • modify the physical address with devmem
  • query the value from the kernel module
• IO mem and QEMU virtual platform device
  • create a platform device with known physical register addresses
  • use devmem to write to the register
  • watch printfs come out of the virtual device in response

Bonus: determine the physical address for a virtual address

Is there any API for determining the physical address from virtual address in Linux?

Answer 2

You can map a device file to a user process memory using mmap(2) system call. Usually, device files are mappings of physical memory to the file system. Otherwise, you have to write a kernel module which creates such a file or provides a way to map the needed memory to a user process.

Another way is remapping parts of /dev/mem to a user memory.

Edit: Example of mmaping /dev/mem (this program must have access to /dev/mem, e.g. have root rights):

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    if (argc < 3) {
        printf("Usage: %s <phys_addr> <offset>\n", argv[0]);
        return 0;
    }

    off_t offset = strtoul(argv[1], NULL, 0);
    size_t len = strtoul(argv[2], NULL, 0);

    // Truncate offset to a multiple of the page size, or mmap will fail.
    size_t pagesize = sysconf(_SC_PAGE_SIZE);
    off_t page_base = (offset / pagesize) * pagesize;
    off_t page_offset = offset - page_base;

    int fd = open("/dev/mem", O_SYNC);
    unsigned char *mem = mmap(NULL, page_offset + len, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, page_base);
    if (mem == MAP_FAILED) {
        perror("Can't map memory");
        return -1;
    }

    size_t i;
    for (i = 0; i < len; ++i)
        printf("%02x ", (int)mem[page_offset + i]);

    return 0;
}