Quickly create a large file on a Linux system

Question

How can I quickly create a large file on a Linux (Red Hat Linux) system?

dd will do the job, but reading from /dev/zero and writing to the drive can take a long time when you need a file several hundreds of GBs in size for testing... If you need to do that repeatedly, the time really adds up.

I don't care about the contents of the file, I just want it to be created quickly. How can this be done?

Using a sparse file won't work for this. I need the file to be allocated disk space.

Answer 1

truncate -s 10M output.file

will create a 10 M file instantaneously (M stands for 1024*1024 bytes, MB stands for 1000*1000 - same with K, KB, G, GB...)

EDIT: as many have pointed out, this will not physically allocate the file on your device. With this you could actually create an arbitrary large file, regardless of the available space on the device

So, when doing this, you will be deferring physical allocation until the file is accessed. If you're mapping this file to memory, you may not have the expected performance.

But this is still a useful command to know

Answer 2

Shameless plug: OTFFS provides a file system providing arbitrarily large (well, almost. Exabytes is the current limit) files of generated content. It is Linux-only, plain C, and in early alpha.

See https://github.com/s5k6/otffs.

Answer 3

One approach: if you can guarantee unrelated applications won't use the files in a conflicting manner, just create a pool of files of varying sizes in a specific directory, then create links to them when needed.

For example, have a pool of files called:

• /home/bigfiles/512M-A
• /home/bigfiles/512M-B
• /home/bigfiles/1024M-A
• /home/bigfiles/1024M-B

Then, if you have an application that needs a 1G file called /home/oracle/logfile, execute a "ln /home/bigfiles/1024M-A /home/oracle/logfile".

If it's on a separate filesystem, you will have to use a symbolic link.

The A/B/etc files can be used to ensure there's no conflicting use between unrelated applications.

The link operation is about as fast as you can get.

Answer 4

You can use "yes" command also. The syntax is fairly simple:

#yes >> myfile

Press "Ctrl + C" to stop this, else it will eat up all your space available.

To clean this file run:

#>myfile

will clean this file.

Answer 5

This is the fastest I could do (which is not fast) with the following constraints:

• The goal of the large file is to fill a disk, so can't be compressible.
• Using ext3 filesystem. (fallocate not available)

This is the gist of it... `

// include stdlib.h, stdio.h, and stdint.h
int32_t buf[256]; // Block size.
for (int i = 0; i < 256; ++i)
{
    buf[i] = rand(); // random to be non-compressible.
}
FILE* file = fopen("/file/on/your/system", "wb");
int blocksToWrite = 1024 * 1024; // 1 GB
for (int i = 0; i < blocksToWrite; ++i)
{
   fwrite(buf, sizeof(int32_t), 256, file);
}

`

In our case this is for an embedded linux system and this works well enough, but would prefer something faster.

FYI the command "dd if=/dev/urandom of=outputfile bs=1024 count = XX" was so slow as to be unusable.

Answer 6

Examples where seek is the size of the file you want in bytes

#kilobytes
dd if=/dev/zero of=filename bs=1 count=0 seek=200K

#megabytes
dd if=/dev/zero of=filename bs=1 count=0 seek=200M

#gigabytes
dd if=/dev/zero of=filename bs=1 count=0 seek=200G

#terabytes
dd if=/dev/zero of=filename bs=1 count=0 seek=200T

From the dd manpage:

BLOCKS and BYTES may be followed by the following multiplicative suffixes: c=1, w=2, b=512, kB=1000, K=1024, MB=1000*1000, M=1024*1024, GB =1000*1000*1000, G=1024*1024*1024, and so on for T, P, E, Z, Y.