Answer by mikeserv for Why does dd from /dev/random give different file sizes?

dd is designed for blocking - it is usually the best tool at your disposal for reading from variable sized inputs if you need it done immediately because dd will not buffer current reads into some future write()(unless you very explicitly configure it that way with larger obs than ibs), but will instead write() everything it reads as soon as it read()s it (and optionally processes it).

Here are some important definitions:

• ibs=expr
  • Specify the input block size, in bytes, by expr(default is 512).
• obs=expr
  • Specify the output block size, in bytes, by expr(default is 512).
• bs=expr
  • Set both input and output block sizes to expr bytes, superseding ibs= and obs=. If no conversion other than sync, noerror, and notrunc is specified, each input block shall be copied to the output as a single block without aggregating short blocks.

So you see, when ibs and obs are defined together as bs then ibs takes precedence - but otherwise, if you are specific, then either obs or cbs does.

Here is an example in which ibs is most important. You might do something like this if you wanted to track how soon the /dev/random pool filled...

dd "ibs=$size" conv=sync "count=$lmt" \     if=/dev/random of="$somefile"

As long as if='s target is readable at all, that will always result in the same sized output file, because dd will synchronize blocks read-in on nulls. In other words, if ddread()s for an input-block of $((size=10))$((count=5)) times and the read() file returns 2 bytes, then 8 bytes, then 12 bytes, then 2 bytes, then 4 bytes, dd will write to its outfile something like

 2 read bytes 8NULs \ 8 read bytes 2NULs \10 read bytes 0NULs \ 4 read bytes 6NULs \ 4 read bytes 6NULs

...because dd, by default, does not delay. So if you need to track in-stream and delimit the writes of some other process, dd is the tool for you.

If you're just writing some amount of data to a regular file then, contrary to other statements made here, you can also use dd for this - and fairly easily - but you'll need more than one and a reliable blocking factor.

For example, if you did:

{   dd ibs="$size" obs="${size}x$block_factor" |    dd bs="${size}x$blockfactor""count=$lmt"}  <infile >outfile

...the first dd would buffer as many ibs="$size" input blocks as were necessary to fill at least one obs="${size}x$block_factor" output block for every write() to the pipe between it and the second dd. This means that the second dd can limit the output reliably with count="$lmt" because all of the write()s the first makes will match its i/o blocksize - regardless of how many read()s the first dd must do to make it so.

And that's how you can use dd to reliably read pipes or other types of special files - with just a little bit of math.