When debugging a problem caused by high I/O latency on Linux, it may be interesting to emulate a slow or congested block device. The device mapper driver which manages logical volumes on Linux has a solution for that: the dm-delay target.
In this article, we will use the dm-delay target to delay reads and writes to a block device. We will first create a ramdisk which is a blazingly fast block device. Then we will stack a dm-delay target on top of it and measure the I/O latency it introduces.
Creating a ramdisk
A ramdisk is a RAM backed disk. Since data written to RAM do not persist without power, a ramdisk should never be used to store real data. Compared to a hard disk drive, a ramdisk is much smaller in size: its size is capped by the computer RAM size. But as we will see, a ramdisk is much faster than a hard disk drive.
On Linux, a set of ramdisks is created by loading the brd kernel module. The number of ramdisks and their size is configured by passing arguments to modprobe: rd_nr is the maximum number of ramdisks and rd_size is the size of each ramdisk in kibibytes.
$ sudo modprobe brd rd_nr=1 rd_size=1048576 $ ls -l /dev/ram0 brw-rw---- 1 root disk 1, 0 Aug 24 20:00 /dev/ram0 $ sudo blockdev --getsize /dev/ram0 # Display the size in 512-byte sectors 2097152
Creating a delayed target with dm-delay
explains how to configure a delayed target with
instance, you can use a script like this one to stack a delayed block
device on top of a given block device:
#!/bin/sh # Create a block device that delays reads for 500 ms size=$(blockdev --getsize $1) # Size in 512-bytes sectors echo "0 $size delay $1 0 500" | dmsetup create delayed
Checking the latency of dm-delay
Let's check the latency introduced by dm-delay. We use
fio to compare the latency of the
/dev/ram0) with the latency of the delayed device
/dev/dm-0). The job file for fio that describes the I/O workload is
[random] # Perform 4K random reads for 10 seconds using direct I/Os filename=/dev/dm-0 readwrite=randread blocksize=4k ioengine=sync direct=1 time_based=1 runtime=10
At the end of the run, fio displays a bunch of statistics. One of them is the completion latency (denoted as clat):
- ramdisk: 1.33 µs
- delayed block device: 499735.14 µs
The latency of the ramdisk is very low whereas the latency of the delayed block device stacked on top of the ramdisk is close to the 500 ms delay we configured.
A similar experiment for writes shows that
dm-delay also delays
writes to the device. As a sidenote, if you want to delay writes with
dm-delay, you have to provide a second set of parameters to dmsetup:
#!/bin/sh # Create a block device that delays reads for 500 ms and writes for 300 ms size=$(blockdev --getsize $1) # Size in 512-bytes sectors echo "0 $size delay $1 0 500 $1 0 300" | dmsetup create delayed
The device mapper can also be requested to suspend and resume I/Os.
$ sudo dmsetup suspend /dev/dm-0 $ sudo dmsetup resume /dev/dm-0