Emulate a slow block device with dm-delay

When debugging a problem caused by high I/O latency on Linux, it may be interesting to emulate a slow or congested block device. The device mapper driver which manages logical volumes on Linux has a solution for that: the dm-delay target.

In this article, we will use the dm-delay target to delay reads and writes to a block device. We will first create a ramdisk which is a blazingly fast block device. Then we will stack a dm-delay target on top of it and measure the I/O latency it introduces.

Creating a ramdisk

A ramdisk is a RAM backed disk. Since data written to RAM do not persist without power, a ramdisk should never be used to store real data. Compared to a hard disk drive, a ramdisk is much smaller in size: its size is capped by the computer RAM size. But as we will see, a ramdisk is much faster than a hard disk drive.

On Linux, a set of ramdisks is created by loading the brd kernel module. The number of ramdisks and their size is configured by passing arguments to modprobe: rd_nr is the maximum number of ramdisks and rd_size is the size of each ramdisk in kibibytes.

$ sudo modprobe brd rd_nr=1 rd_size=1048576
$ ls -l /dev/ram0
brw-rw---- 1 root disk 1, 0 Aug 24 20:00 /dev/ram0
$ sudo blockdev --getsize /dev/ram0 # Display the size in 512-byte sectors
2097152

Creating a delayed target with dm-delay

The kernel documentation explains how to configure a delayed target with dmsetup. For instance, you can use a script like this one to stack a delayed block device on top of a given block device:

#!/bin/sh
# Create a block device that delays reads for 500 ms
size=$(blockdev --getsize $1) # Size in 512-bytes sectors
echo "0 $size delay $1 0 500" | dmsetup create delayed

Checking the latency of dm-delay

Let's check the latency introduced by dm-delay. We use fio to compare the latency of the ramdisk (/dev/ram0) with the latency of the delayed device (/dev/dm-0). The job file for fio that describes the I/O workload is as follows:

[random]
# Perform 4K random reads for 10 seconds using direct I/Os
filename=/dev/dm-0
readwrite=randread
blocksize=4k
ioengine=sync
direct=1
time_based=1
runtime=10

At the end of the run, fio displays a bunch of statistics. One of them is the completion latency (denoted as clat):

• ramdisk: 1.33 µs
• delayed block device: 499735.14 µs

The latency of the ramdisk is very low whereas the latency of the delayed block device stacked on top of the ramdisk is close to the 500 ms delay we configured.

A similar experiment for writes shows that dm-delay also delays writes to the device. As a sidenote, if you want to delay writes with dm-delay, you have to provide a second set of parameters to dmsetup:

#!/bin/sh
# Create a block device that delays reads for 500 ms and writes for 300 ms
size=$(blockdev --getsize $1) # Size in 512-bytes sectors
echo "0 $size delay $1 0 500 $1 0 300" | dmsetup create delayed

Suspending I/Os

The device mapper can also be requested to suspend and resume I/Os.

$ sudo dmsetup suspend /dev/dm-0
$ sudo dmsetup resume  /dev/dm-0