dcp [OPTION] SRC DEST
Parallel MPI application to recursively copy files and directories.
dcp is a file copy tool in the spirit of cp(1) that evenly distributes the work of scanning the directory tree, and copying file data across a large cluster without any centralized state. It is designed for copying files that are located on a distributed parallel file system, and it splits large file copies across multiple processes.
-b
,
--blocksize
SIZE
¶Set the I/O buffer to be SIZE bytes. Units like "MB" and "GB" may immediately follow the number without spaces (eg. 8MB). The default blocksize is 1MB.
-i
,
--input
FILE
¶Read source list from FILE. FILE must be generated by another tool from the mpiFileUtils suite.
-k
,
--chunksize
SIZE
¶Split large files into chunks of SIZE bytes to be processed. Multiple process ranks may copy a large file in parallel. Units like "MB" and "GB" can immediately follow the number without spaces (eg. 64MB). The default chunksize is 1MB.
-p
,
--preserve
¶Preserve permissions, group, timestamps, and extended attributes.
-s
,
--synchronous
¶Use synchronous read/write calls (open files with O_DIRECT). This also avoids caching the file data on the client nodes.
-S
,
--sparse
¶Create sparse files when possible.
--progress
N
¶Print progress message to stdout approximately every N seconds. The number of seconds must be a non-negative integer. A value of 0 disables progress messages.
-v
,
--verbose
¶Run in verbose mode.
-q
,
--quiet
¶Run tool silently. No output is printed.
-h
,
--help
¶Print a brief message listing the dcp(1) options and usage.
If a long-running copy is interrupted, one should delete the partial copy and run dcp again from the beginning. One may use drm to quickly remove a partial copy of a large directory tree.
To ensure the copy is successful, one should run dcmp after dcp completes to verify the copy, especially if dcp was not run with the -s option.
mpirun -np 128 dcp /source/dir1 /dest/dir2
mkdir /dest/dir2 mpirun -np 128 dcp /source/dir1/\* /dest/dir2
mpirun -np 128 dcp -p /source/dir1/ /dest/dir2
Using the -S option for sparse files does not work yet at LLNL. If you try to use it then dcp will default to a normal copy.
The maximum supported file name length for any file transferred is approximately 4068 characters. This may be less than the number of characters that your operating system supports.
The mpiFileUtils source code and all documentation may be downloaded from <https://github.com/hpc/mpifileutils>