How to create a multi partition SD disk image with

Question

Is it possible to create a complete SD image in linux without having root privileges (that is, no loopback mount)? I'm looking for a way to automate embedded system image creation. The image should include a specific partition structure and partitions formatted to FAT and ext2 populated with files from the build system.

Answer 1

you might want to look at genextfs, that creates an ext2 filesystem in a regular file without any sort of mounting.

Answer 2

I had this problem and couldn't find a viable solution, so I wrote this utility that we've open-sourced here.

From the README:

$ dd if=/dev/zero of=disk.image bs=1M count=4
4+0 records in
4+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.00470867 s, 891 MB/s
$ parted --script disk.image \
    mktable msdos mkpart primary 2048s 100% set 1 boot on
$ mkdir mntdir
$ partfs -o dev=disk.image mntdir
$ mkfs.ext4 mntdir/p1
mke2fs 1.42.13 (17-May-2015)
Creating filesystem with 3072 1k blocks and 768 inodes

Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

$ fusermount -u mntdir

Answer 3

Minimal runnable sfdisk + mke2fs example without sudo

In this example, we will create, without sudo or setsuid, an image file that contains two ext2 partitions, each populated with files from a host directory.

We will then use sudo losetup just to mount the partitions to test that the Linux kernel can actually read them as explained at: How to mount one partition from an image file that contains multiple partitions on Linux?

For more details, see:

• sfdisk: deals with the partition table: https://superuser.com/questions/332252/how-to-create-and-format-a-partition-using-a-bash-script/1132834#1132834
• mke2fs: deals with EXT formatting of partitions: https://superuser.com/questions/605196/how-to-create-ext2-image-without-superuser-rights/1366762#1366762

The example:

#!/usr/bin/env bash

# Input params.
root_dir_1=root1
root_dir_2=root2
partition_file_1=part1.ext2
partition_file_2=part2.ext2
partition_size_1_megs=32
partition_size_2_megs=32
img_file=img.img
block_size=512

# Calculated params.
mega="$(echo '2^20' | bc)"
partition_size_1=$(($partition_size_1_megs * $mega))
partition_size_2=$(($partition_size_2_megs * $mega))

# Create a test directory to convert to ext2.
mkdir -p "$root_dir_1"
echo content-1 > "${root_dir_1}/file-1"
mkdir -p "$root_dir_2"
echo content-2 > "${root_dir_2}/file-2"

# Create the 2 raw ext2 images.
rm -f "$partition_file_1"
mke2fs \
  -d "$root_dir_1" \
  -r 1 \
  -N 0 \
  -m 5 \
  -L '' \
  -O ^64bit \
  "$partition_file_1" \
  "${partition_size_1_megs}M" \
;
rm -f "$partition_file_2"
mke2fs \
  -d "$root_dir_2" \
  -r 1 \
  -N 0 \
  -m 5 \
  -L '' \
  -O ^64bit \
  "$partition_file_2" \
  "${partition_size_2_megs}M" \
;

# Default offset according to
part_table_offset=$((2**20))
cur_offset=0
bs=1024
dd if=/dev/zero of="$img_file" bs="$bs" count=$((($part_table_offset + $partition_size_1 + $partition_size_2)/$bs)) skip="$(($cur_offset/$bs))"
printf "
type=83, size=$(($partition_size_1/$block_size))
type=83, size=$(($partition_size_2/$block_size))
" | sfdisk "$img_file"
cur_offset=$(($cur_offset + $part_table_offset))
# TODO: can we prevent this and use mke2fs directly on the image at an offset?
# Tried -E offset= but could not get it to work.
dd if="$partition_file_1" of="$img_file" bs="$bs" seek="$(($cur_offset/$bs))"
cur_offset=$(($cur_offset + $partition_size_1))
rm "$partition_file_1"
dd if="$partition_file_2" of="$img_file" bs="$bs" seek="$(($cur_offset/$bs))"
cur_offset=$(($cur_offset + $partition_size_2))
rm "$partition_file_2"

# Test the ext2 by mounting it with sudo.
# sudo is only used for testing, the image is completely ready at this point.

# losetup automation functions from:
# https://stackoverflow.com/questions/1419489/how-to-mount-one-partition-from-an-image-file-that-contains-multiple-partitions/39675265#39675265
loop-mount-partitions() (
  set -e
  img="$1"
  dev="$(sudo losetup --show -f -P "$img")"
  echo "$dev" | sed -E 's/.*[^[:digit:]]([[:digit:]]+$)/\1/g'
  for part in "${dev}p"*; do
    if [ "$part" = "${dev}p*" ]; then
      # Single partition image.
      part="${dev}"
    fi
    dst="/mnt/$(basename "$part")"
    echo "$dst" 1>&2
    sudo mkdir -p "$dst"
    sudo mount "$part" "$dst"
  done
)
loop-unmount-partitions() (
  set -e
  for loop_id in "$@"; do
    dev="/dev/loop${loop_id}"
    for part in "${dev}p"*; do
      if [ "$part" = "${dev}p*" ]; then
        part="${dev}"
      fi
      dst="/mnt/$(basename "$part")"
      sudo umount "$dst"
    done
    sudo losetup -d "$dev"
  done
)

loop_id="$(loop-mount-partitions "$img_file")"
sudo cmp /mnt/loop0p1/file-1 "${root_dir_1}/file-1"
sudo cmp /mnt/loop0p2/file-2 "${root_dir_2}/file-2"
loop-unmount-partitions "$loop_id"

Tested on Ubuntu 18.04. GitHub upstream.

Answer 4

Yes, this is possible with guestfish:

$ cat << END > extlinux.conf
> default linux
> timeout 0
>
> label linux
> kernel /vmlinuz
> append initrd=/initrd.img root=/dev/vda1 rw console=ttyS0
END
$ guestfish -N debian-unstable.img=disk:2G -- \
    part-disk /dev/sda mbr : \
    part-set-bootable /dev/sda 1 true : \
    mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
    tar-in debian-unstable.tar / : \
    extlinux / : \
    copy-in extlinux.conf /

The result will be debian-unstable.img with one ext2 partition on it, containing all files from the tarball debian-unstable.tar and the whole thing is made bootable with extlinux. You can verify the disk image using qemu.

Answer 5

I'm trying to do the same thing. My first attempt used the loopback block device, but I have found work-arounds to both steps that require loopback.

Steps with loopback

Here's what I'm doing ( $1 is image file name, $2 is file size):

1. create zeroed disk image file with dd if=/dev/zero of=$1 bs=512 count=$(($2/512))
2. create partition table with parted -s $1 mklabel msdos
3. create partition with parted -s $1 "mkpart primary 0% 100%"
4. attach partition to loop sudo losetup --find $1 --offset $OFFSET_TO_PARTITION_BYTES
5. make file system with mkfs.ext4 with mkfs.ext4 -I 128 -L BOOT -b 2048 -O ^has_journal /dev/loop0 $SIZE_IN_2048_BLOCKS
6. mount /dev/loop0

The loopback is used because

• in step 4 & 5, mkfs doesn't have an offset option so losetup is used to solve that problem
• in step 6, mount allows the use of the operating systems ext4 driver

Looback workarounds

Shitty work-around for step 4 & 5:

• xmount --in dd --out vdi disk.img mnt/
• vdfuse -f mnt/disk.vdi -r ./mnt2
• ./mnt2 will now have two files: EntireDisk, and Partition1
• point mkfs.ext4 at ./mnt2/Partition1

Work-around solution to step 6:

• follow all steps for step 5 work around
• use fuseext2 to mount ./mnt2/Partition1

Caveat

Caveat: ext4 support is not advertised in their documentation, and attempts to mount come with a warning:

This is experimental code, opening rw a real file system could be
dangerous for your data. Please add "-o ro" if you want to open the file
system image in read-only mode, or "-o rw+" if you accept the risk to test
this module

Update

vdfuse should be able to mount a raw image without the help of xmount, but there is a bug which ignores the RAW option.

I tracked down and fixed the bug with a patch here:

https://bugs.launchpad.net/ubuntu/+source/virtualbox-ose/+bug/1019075