mirror of https://github.com/pkolano/shift.git
Initial commit
Paul Kolano
parent
f31fa15b8c
commit
a43319c7cc
|
|
@ -0,0 +1,150 @@
|
|||
CHANGES
|
||||
=======
|
||||
|
||||
* Shift 3.0 (01/09/14)
|
||||
- First public release
|
||||
- Note that --no-offline is currently enabled by default due to
|
||||
DMF corruption issues found when releasing files immediately
|
||||
after they are copied
|
||||
|
||||
* Shift 3.1 (01/13/14)
|
||||
- Fixed bad metadata counts during --sync preventing dirs from completing
|
||||
- Removed some site-specific config that was mistakenly included in shiftc
|
||||
|
||||
* Shift 3.2 (02/07/14)
|
||||
- Added note about mcp/msum >= 1.76.7 since earlier versions do not
|
||||
support all required functionality
|
||||
- Added warning message when --no-check option specified
|
||||
- Fixed leftover temp files after --status
|
||||
- Fixed hung transfers with one but not both of --no-check/--no-preserve
|
||||
- Fixed not an array reference exception during some rsync scenarios
|
||||
- Fixed use of undefined value exception during some tar scenarios
|
||||
|
||||
* Shift 3.3 (10/15/14)
|
||||
- Added file name to fish error messages
|
||||
- Changed final file size checks to reread src size when dst size differs
|
||||
- Changed status totals to use total+ instead of - during initialization
|
||||
- Changed mkdir and dir chattr ops to be limited by --files setting
|
||||
- Fixed tar file renaming when multiple unsplit tar files created via stdin
|
||||
- Fixed remote tar extraction due to missing fields in shift-aux tar parsing
|
||||
- Fixed parsing of @ in remote addresses (user name still dropped however)
|
||||
- Fixed local lustre striping when stripes > 160 for lustre < 2.4
|
||||
- Fixed unescape of src for tar link validation
|
||||
- Fixed display of ssize in --stats when --verify not used
|
||||
- Fixed addition to meta file if --meta=0 specified
|
||||
- Fixed reversion of metadata corruption after abrupt termination
|
||||
- Fixed truncation of existing file during file corruption recovery
|
||||
- Fixed possible truncation of directory prefix during tar creation
|
||||
- Fixed writability check of remote files
|
||||
- Fixed hang when getting striping of link to fifo (due to lfs bug)
|
||||
|
||||
* Shift 3.4 (01/07/15)
|
||||
- Added support for GNU long names/links during tar creation
|
||||
- Added inclusion/exclusion of file names based on regular expressions
|
||||
- Added --io[rw] and --net[rw] options to throttle by read/write only
|
||||
- Added global throttling to throttle read/writes/both by fs/host/user
|
||||
- Added --sync-fast to synchronize files without verifying integrity
|
||||
- Added built-in metadata sync mechanism for improved scalability
|
||||
- Added estimated time until completion in status
|
||||
- Added ability to specify some transport options in manager config
|
||||
- Added background spawn of dmget/dmput to avoid intermittent hangs
|
||||
- Changed disk throttling based on high/low threshold for suspend/resume
|
||||
- Fixed inadvertent tar file renaming when transfer completes with errors
|
||||
- Fixed error parsing of mcp/msum based on coreutils 8.x
|
||||
- Fixed table of contents truncation during remote tar creation
|
||||
|
||||
* Shift 4.0 (07/23/15)
|
||||
- Note that this version is not backward compatible with previous versions
|
||||
- Added backgroundable, parallelizable, and restartable initialization
|
||||
- Added --newer and --older options for incremental backups
|
||||
- Added --preallocate option to preallocate files below given sparsity
|
||||
- Added --no-verify to disable verification, which is now enabled by default
|
||||
- Added preservation of extended attributes during non-tar transfers
|
||||
- Added backoff to alternate transports/checksums during retries
|
||||
- Added error handling when transfer metadata dir can't be created
|
||||
- Added ability to use --state w/o --id for brief status of xfers in state
|
||||
- Added situational attribute handling for optimized transfer initialization
|
||||
- Added ability to specify default tar split size in manager config
|
||||
- Changed handling of embedded modules to use App::FatPacker
|
||||
- Changed detailed status to show source file during tar creation
|
||||
- Changed intra-source symlink dereferencing due to initialization changes
|
||||
(location of dereferenced links may differ depending on traversal order)
|
||||
- Changed status output to omit some completed transfers beyond threshold
|
||||
- Changed --meta manager option to require --id
|
||||
- Fixed verification of files ending in whitespace
|
||||
- Fixed time estimate to reflect mgr overhead, parallelism, and chattr speed
|
||||
- Fixed sum handling of names containing trailing carriage returns
|
||||
- Fixed handling of pathological umasks
|
||||
- Fixed overrun of metadata directories when linux subdir limit reached
|
||||
- Fixed automatic striping of tar files on lustre
|
||||
- Fixed tar extraction of files with trailing spaces
|
||||
- Fixed preservation of symlink times
|
||||
- Fixed --no-cron propagation across parallel clients
|
||||
- Fixed tar header validation of symlinks containing escaped characters
|
||||
- Fixed bbftp inability to handle vt character
|
||||
- Fixed rsync inability to handle cr/lf characters (due to --files-from)
|
||||
- Fixed rsync unexpected rename behavior when one file in --files-from list
|
||||
- Fixed fish handling of errors with linefeeds
|
||||
- Removed --sync-fast since it is now equivalent to --sync --no-verify
|
||||
- Removed file/dir meter during initialization since init now backgrounded
|
||||
|
||||
* Shift 5.0 (07/12/16)
|
||||
- Added better detection of lan transfers in stats
|
||||
- Added error handling for malformed tar headers
|
||||
- Added --buffer option to adjust buffer size used by underlying transports
|
||||
- Added --streams option to adjust tcp streams used by tcp-based transports
|
||||
- Added --window option to set tcp window size used by tcp-based transports
|
||||
- Added --ports option to set the remote ports used by tcp-based transports
|
||||
- Added --threads option to set number of threads used by local transports
|
||||
- Added --bandwidth option to set bandwidth for stream/window calculation
|
||||
- Added adjustment of tcp window/streams based on b/w heuristics and latency
|
||||
- Added ability to set small file breakeven points for local/lan/wan cases
|
||||
- Added minimum split setting to prevent file system overload with metadata
|
||||
- Added rescan of mesh keys between batches to pick up newly generated keys
|
||||
- Added support for bbcp as underlying transport
|
||||
- Added support for all remote transports to be used as local transports
|
||||
- Added manager setting for lustre default stripe count
|
||||
- Added multi-threading of single/multi-file batches to built-in transports
|
||||
- Added multi-threading of single/multi-file batches to built-in hashing
|
||||
- Added multi-threaded tcp-based remote transport based on fish protocol
|
||||
- Added support for --verify-fast during fish gets
|
||||
- Added output across transfers of all users when --status invoked as root
|
||||
- Added zero-padding to duration and estimated time when --status=pad used
|
||||
- Changed status emails to limit length of original command sent
|
||||
- Changed brief status so at least one completed transfer is always shown
|
||||
- Changed --encrypt to --secure, which also changes ssh cipher/mac selection
|
||||
- Changed dmput handling so -r is no longer used in automatic offlining
|
||||
- Changed help output into functional units
|
||||
- Changed external invocations to eliminate all extra shell processes
|
||||
- Changed client selection to use selection hook instead of random policy
|
||||
- Changed shift-aux sums so file issues are errors and not bad checksums
|
||||
- Changed handling of estimated completion to reflect actual operation rates
|
||||
- Changed extraction of tar files to remove relative path components
|
||||
- Changed --stats output to omit rows without non-empty values
|
||||
- Fixed gridftp support using unbuffer utility to interlace stderr/stdout
|
||||
- Fixed umask for root transfers so won't inadvertently expose files
|
||||
- Fixed status after --wait, which sometimes did not appear when redirected
|
||||
- Fixed getting/setting of acls and xattrs on symlinks
|
||||
- Fixed existence check of target path when using openssh 7.x
|
||||
- Fixed infinite loop in built-in hashing when source file shrinks
|
||||
- Fixed reported rate when operations report in after transfer stopped
|
||||
- Fixed improper dst truncation in some non-tar corruption recovery cases
|
||||
- Fixed infinite loop when extracting tar files less than 512 bytes
|
||||
- Fixed host/process failures due to dmgets on every command line file
|
||||
- Fixed --include/--exclude options to handle malformed regular expressions
|
||||
- Fixed truncation of built-in local copies when dst larger than src
|
||||
- Fixed toc file empty blocks of increasing size during split tar creation
|
||||
- Fixed built-in hashing chopping off range when file has backslash/newline
|
||||
- Fixed correction of corruption in multiple byte ranges during tar creation
|
||||
- Fixed distribution of clients to remote hosts when more clients than hosts
|
||||
- Fixed built-in transport detection when user $PATH is empty
|
||||
- Fixed sum file rename when no regular files during tar creation
|
||||
- Fixed -d with unwritable or trailing slash dst (bug report by J. Otey)
|
||||
- Fixed exception in fish protocol when input stream is invalid
|
||||
- Fixed transport selection order when first transport not suitable
|
||||
- Fixed crontab handling with csh variants
|
||||
- Fixed exception when fish input stream is unreadable
|
||||
- Fixed mkdir errors during parallelization in some scenarios
|
||||
- Fixed sum file rename when transfer w/o regular files grouped with regular
|
||||
- Fixed abort due to embedded use of Time::HiRes in some perl versions
|
||||
- Removed use of File::Copy for built-in local copies
|
||||
|
|
@ -0,0 +1,256 @@
|
|||
NASA OPEN SOURCE AGREEMENT VERSION 1.3
|
||||
|
||||
THIS OPEN SOURCE AGREEMENT ("AGREEMENT") DEFINES THE RIGHTS OF USE,
|
||||
REPRODUCTION, DISTRIBUTION, MODIFICATION AND REDISTRIBUTION OF CERTAIN
|
||||
COMPUTER SOFTWARE ORIGINALLY RELEASED BY THE UNITED STATES GOVERNMENT
|
||||
AS REPRESENTED BY THE GOVERNMENT AGENCY LISTED BELOW ("GOVERNMENT
|
||||
AGENCY"). THE UNITED STATES GOVERNMENT, AS REPRESENTED BY GOVERNMENT
|
||||
AGENCY, IS AN INTENDED THIRD-PARTY BENEFICIARY OF ALL SUBSEQUENT
|
||||
DISTRIBUTIONS OR REDISTRIBUTIONS OF THE SUBJECT SOFTWARE. ANYONE WHO
|
||||
USES, REPRODUCES, DISTRIBUTES, MODIFIES OR REDISTRIBUTES THE SUBJECT
|
||||
SOFTWARE, AS DEFINED HEREIN, OR ANY PART THEREOF, IS, BY THAT ACTION,
|
||||
ACCEPTING IN FULL THE RESPONSIBILITIES AND OBLIGATIONS CONTAINED IN
|
||||
THIS AGREEMENT.
|
||||
|
||||
Government Agency:
|
||||
National Aeronautics and Space_Administration (NASA)
|
||||
Government Agency Original Software Designation:
|
||||
ARC-16940-1
|
||||
Government Agency Original Software Title:
|
||||
Shift: Self-Healing Independent File Transfer
|
||||
Government Agency Point of Contact for Original Software:
|
||||
Paul Kolano (paul.kolano@nasa.gov).
|
||||
|
||||
User Registration Requested. Please Visit http://opensource.arc.nasa.gov
|
||||
|
||||
|
||||
1. DEFINITIONS
|
||||
|
||||
A. "Contributor" means Government Agency, as the developer of the
|
||||
Original Software, and any entity that makes a Modification.
|
||||
B. "Covered Patents" mean patent claims licensable by a Contributor
|
||||
that are necessarily infringed by the use or sale of its Modification
|
||||
alone or when combined with the Subject Software.
|
||||
C. "Display" means the showing of a copy of the Subject Software,
|
||||
either directly or by means of an image, or any other device.
|
||||
D. "Distribution" means conveyance or transfer of the Subject
|
||||
Software, regardless of means, to another.
|
||||
E. "Larger Work" means computer software that combines Subject
|
||||
Software, or portions thereof, with software separate from the Subject
|
||||
Software that is not governed by the terms of this Agreement.
|
||||
F. "Modification" means any alteration of, including addition to or
|
||||
deletion from, the substance or structure of either the Original
|
||||
Software or Subject Software, and includes derivative works, as that
|
||||
term is defined in the Copyright Statute, 17 USC 101. However, the
|
||||
act of including Subject Software as part of a Larger Work does not in
|
||||
and of itself constitute a Modification.
|
||||
G. "Original Software" means the computer software first released
|
||||
under this Agreement by Government Agency with Government Agency
|
||||
designation ARC-16940-1 and entitled Shift: Self-Healing Independent
|
||||
File Transfer, including source code, object code and accompanying
|
||||
documentation, if any.
|
||||
H. "Recipient" means anyone who acquires the Subject Software under
|
||||
this Agreement, including all Contributors.
|
||||
I. "Redistribution" means Distribution of the Subject Software after a
|
||||
Modification has been made.
|
||||
J. "Reproduction" means the making of a counterpart, image or copy of
|
||||
the Subject Software.
|
||||
K. "Sale" means the exchange of the Subject Software for money or
|
||||
equivalent value.
|
||||
L. "Subject Software" means the Original Software, Modifications, or
|
||||
any respective parts thereof.
|
||||
M. "Use" means the application or employment of the Subject Software
|
||||
for any purpose.
|
||||
|
||||
|
||||
2. GRANT OF RIGHTS
|
||||
|
||||
A. Under Non-Patent Rights: Subject to the terms and conditions of
|
||||
this Agreement, each Contributor, with respect to its own contribution
|
||||
to the Subject Software, hereby grants to each Recipient a
|
||||
non-exclusive, world-wide, royalty-free license to engage in the
|
||||
following activities pertaining to the Subject Software:
|
||||
|
||||
1. Use
|
||||
2. Distribution
|
||||
3. Reproduction
|
||||
4. Modification
|
||||
5. Redistribution
|
||||
6. Display
|
||||
|
||||
B. Under Patent Rights: Subject to the terms and conditions of this
|
||||
Agreement, each Contributor, with respect to its own contribution to
|
||||
the Subject Software, hereby grants to each Recipient under Covered
|
||||
Patents a non-exclusive, world-wide, royalty-free license to engage in
|
||||
the following activities pertaining to the Subject Software:
|
||||
|
||||
1. Use
|
||||
2. Distribution
|
||||
3. Reproduction
|
||||
4. Sale
|
||||
5. Offer for Sale
|
||||
|
||||
C. The rights granted under Paragraph B. also apply to the combination
|
||||
of a Contributor's Modification and the Subject Software if, at the
|
||||
time the Modification is added by the Contributor, the addition of
|
||||
such Modification causes the combination to be covered by the Covered
|
||||
Patents. It does not apply to any other combinations that include a
|
||||
Modification.
|
||||
|
||||
D. The rights granted in Paragraphs A. and B. allow the Recipient to
|
||||
sublicense those same rights. Such sublicense must be under the same
|
||||
terms and conditions of this Agreement.
|
||||
|
||||
|
||||
3. OBLIGATIONS OF RECIPIENT
|
||||
|
||||
A. Distribution or Redistribution of the Subject Software must be made
|
||||
under this Agreement except for additions covered under paragraph 3H.
|
||||
|
||||
1. Whenever a Recipient distributes or redistributes the Subject
|
||||
Software, a copy of this Agreement must be included with each copy
|
||||
of the Subject Software; and
|
||||
2. If Recipient distributes or redistributes the Subject Software in
|
||||
any form other than source code, Recipient must also make the
|
||||
source code freely available, and must provide with each copy of
|
||||
the Subject Software information on how to obtain the source code
|
||||
in a reasonable manner on or through a medium customarily used for
|
||||
software exchange.
|
||||
|
||||
B. Each Recipient must ensure that the following copyright notice
|
||||
appears prominently in the Subject Software:
|
||||
|
||||
Copyright (C) 2012 United States Government as represented by the
|
||||
Administrator of the National Aeronautics and Space Administration
|
||||
(NASA). All Rights Reserved.
|
||||
|
||||
C. Each Contributor must characterize its alteration of the Subject
|
||||
Software as a Modification and must identify itself as the originator
|
||||
of its Modification in a manner that reasonably allows subsequent
|
||||
Recipients to identify the originator of the Modification. In
|
||||
fulfillment of these requirements, Contributor must include a file
|
||||
(e.g., a change log file) that describes the alterations made and the
|
||||
date of the alterations, identifies Contributor as originator of the
|
||||
alterations, and consents to characterization of the alterations as a
|
||||
Modification, for example, by including a statement that the
|
||||
Modification is derived, directly or indirectly, from Original
|
||||
Software provided by Government Agency. Once consent is granted, it
|
||||
may not thereafter be revoked.
|
||||
|
||||
D. A Contributor may add its own copyright notice to the Subject
|
||||
Software. Once a copyright notice has been added to the Subject
|
||||
Software, a Recipient may not remove it without the express permission
|
||||
of the Contributor who added the notice.
|
||||
|
||||
E. A Recipient may not make any representation in the Subject Software
|
||||
or in any promotional, advertising or other material that may be
|
||||
construed as an endorsement by Government Agency or by any prior
|
||||
Recipient of any product or service provided by Recipient, or that may
|
||||
seek to obtain commercial advantage by the fact of Government Agency's
|
||||
or a prior Recipient's participation in this Agreement.
|
||||
|
||||
F. In an effort to track usage and maintain accurate records of the
|
||||
Subject Software, each Recipient, upon receipt of the Subject
|
||||
Software, is requested to register with Government Agency by visiting
|
||||
the following website: http://opensource.arc.nasa.gov. Recipient's
|
||||
name and personal information shall be used for statistical purposes
|
||||
only. Once a Recipient makes a Modification available, it is requested
|
||||
that the Recipient inform Government Agency at the web site provided
|
||||
above how to access the Modification.
|
||||
|
||||
G. Each Contributor represents that that its Modification is believed
|
||||
to be Contributor's original creation and does not violate any
|
||||
existing agreements, regulations, statutes or rules, and further that
|
||||
Contributor has sufficient rights to grant the rights conveyed by this
|
||||
Agreement.
|
||||
|
||||
H. A Recipient may choose to offer, and to charge a fee for, warranty,
|
||||
support, indemnity and/or liability obligations to one or more other
|
||||
Recipients of the Subject Software. A Recipient may do so, however,
|
||||
only on its own behalf and not on behalf of Government Agency or any
|
||||
other Recipient. Such a Recipient must make it absolutely clear that
|
||||
any such warranty, support, indemnity and/or liability obligation is
|
||||
offered by that Recipient alone. Further, such Recipient agrees to
|
||||
indemnify Government Agency and every other Recipient for any
|
||||
liability incurred by them as a result of warranty, support, indemnity
|
||||
and/or liability offered by such Recipient.
|
||||
|
||||
I. A Recipient may create a Larger Work by combining Subject Software
|
||||
with separate software not governed by the terms of this agreement and
|
||||
distribute the Larger Work as a single product. In such case, the
|
||||
Recipient must make sure Subject Software, or portions thereof,
|
||||
included in the Larger Work is subject to this Agreement.
|
||||
|
||||
J. Notwithstanding any provisions contained herein, Recipient is
|
||||
hereby put on notice that export of any goods or technical data from
|
||||
the United States may require some form of export license from the
|
||||
U.S. Government. Failure to obtain necessary export licenses may
|
||||
result in criminal liability under U.S. laws. Government Agency
|
||||
neither represents that a license shall not be required nor that, if
|
||||
required, it shall be issued. Nothing granted herein provides any
|
||||
such export license.
|
||||
|
||||
|
||||
4. DISCLAIMER OF WARRANTIES AND LIABILITIES; WAIVER AND INDEMNIFICATION
|
||||
|
||||
A. No Warranty: THE SUBJECT SOFTWARE IS PROVIDED "AS IS" WITHOUT ANY
|
||||
WARRANTY OF ANY KIND, EITHER EXPRESSED, IMPLIED, OR STATUTORY,
|
||||
INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY THAT THE SUBJECT SOFTWARE
|
||||
WILL CONFORM TO SPECIFICATIONS, ANY IMPLIED WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR FREEDOM FROM
|
||||
INFRINGEMENT, ANY WARRANTY THAT THE SUBJECT SOFTWARE WILL BE ERROR
|
||||
FREE, OR ANY WARRANTY THAT DOCUMENTATION, IF PROVIDED, WILL CONFORM TO
|
||||
THE SUBJECT SOFTWARE. THIS AGREEMENT DOES NOT, IN ANY MANNER,
|
||||
CONSTITUTE AN ENDORSEMENT BY GOVERNMENT AGENCY OR ANY PRIOR RECIPIENT
|
||||
OF ANY RESULTS, RESULTING DESIGNS, HARDWARE, SOFTWARE PRODUCTS OR ANY
|
||||
OTHER APPLICATIONS RESULTING FROM USE OF THE SUBJECT SOFTWARE.
|
||||
FURTHER, GOVERNMENT AGENCY DISCLAIMS ALL WARRANTIES AND LIABILITIES
|
||||
REGARDING THIRD-PARTY SOFTWARE, IF PRESENT IN THE ORIGINAL SOFTWARE,
|
||||
AND DISTRIBUTES IT "AS IS."
|
||||
|
||||
B. Waiver and Indemnity: RECIPIENT AGREES TO WAIVE ANY AND ALL CLAIMS
|
||||
AGAINST THE UNITED STATES GOVERNMENT, ITS CONTRACTORS AND
|
||||
SUBCONTRACTORS, AS WELL AS ANY PRIOR RECIPIENT. IF RECIPIENT'S USE OF
|
||||
THE SUBJECT SOFTWARE RESULTS IN ANY LIABILITIES, DEMANDS, DAMAGES,
|
||||
EXPENSES OR LOSSES ARISING FROM SUCH USE, INCLUDING ANY DAMAGES FROM
|
||||
PRODUCTS BASED ON, OR RESULTING FROM, RECIPIENT'S USE OF THE SUBJECT
|
||||
SOFTWARE, RECIPIENT SHALL INDEMNIFY AND HOLD HARMLESS THE UNITED
|
||||
STATES GOVERNMENT, ITS CONTRACTORS AND SUBCONTRACTORS, AS WELL AS ANY
|
||||
PRIOR RECIPIENT, TO THE EXTENT PERMITTED BY LAW. RECIPIENT'S SOLE
|
||||
REMEDY FOR ANY SUCH MATTER SHALL BE THE IMMEDIATE, UNILATERAL
|
||||
TERMINATION OF THIS AGREEMENT.
|
||||
|
||||
|
||||
5. GENERAL TERMS
|
||||
|
||||
A. Termination: This Agreement and the rights granted hereunder will
|
||||
terminate automatically if a Recipient fails to comply with these
|
||||
terms and conditions, and fails to cure such noncompliance within
|
||||
thirty (30) days of becoming aware of such noncompliance. Upon
|
||||
termination, a Recipient agrees to immediately cease use and
|
||||
distribution of the Subject Software. All sublicenses to the Subject
|
||||
Software properly granted by the breaching Recipient shall survive any
|
||||
such termination of this Agreement.
|
||||
|
||||
B. Severability: If any provision of this Agreement is invalid or
|
||||
unenforceable under applicable law, it shall not affect the validity
|
||||
or enforceability of the remainder of the terms of this Agreement.
|
||||
|
||||
C. Applicable Law: This Agreement shall be subject to United States
|
||||
federal law only for all purposes, including, but not limited to,
|
||||
determining the validity of this Agreement, the meaning of its
|
||||
provisions and the rights, obligations and remedies of the parties.
|
||||
|
||||
D. Entire Understanding: This Agreement constitutes the entire
|
||||
understanding and agreement of the parties relating to release of the
|
||||
Subject Software and may not be superseded, modified or amended except
|
||||
by further written agreement duly executed by the parties.
|
||||
|
||||
E. Binding Authority: By accepting and using the Subject Software
|
||||
under this Agreement, a Recipient affirms its authority to bind the
|
||||
Recipient to all terms and conditions of this Agreement and that that
|
||||
Recipient hereby agrees to all terms and conditions herein.
|
||||
|
||||
F. Point of Contact: Any Recipient contact with Government Agency is
|
||||
to be directed to the designated representative as follows:
|
||||
Paul Kolano (paul.kolano@nasa.gov).
|
||||
|
|
@ -0,0 +1,314 @@
|
|||
Shift Installation and Configuration
|
||||
====================================
|
||||
|
||||
1. Deployments
|
||||
|
||||
Shift consists of three executable components:
|
||||
|
||||
o shiftc - the Shift client, which is invoked by users and other
|
||||
client instances and must exist on hosts that initiate
|
||||
transfers and hosts on which a transfer is to be
|
||||
parallelized (henceforth called "client hosts")
|
||||
|
||||
o shift-mgr - the Shift manager, which is invoked by client instances
|
||||
and must exist either on all client hosts or on a host
|
||||
accessible via ssh hostbased or pubkey authentication
|
||||
(henceforth called "manager hosts")
|
||||
|
||||
o shift-aux - the Shift auxiliary utility, which is invoked by client
|
||||
instances and is optional for basic functionality, but
|
||||
must be installed on any hosts accessed as the source or
|
||||
destination of a transfer initiated from a different host
|
||||
(henceforth called "remote hosts") for efficient remote
|
||||
file initialization, higher speed transfers via the
|
||||
built-in fish/fish-tcp protocols, remote acl/xattr
|
||||
preservation, remote lustre striping, and remote
|
||||
verification (if msum from mutil is not installed)
|
||||
|
||||
The main consideration for a Shift deployment is deciding which hosts will
|
||||
be the manager hosts. In the simplest case, all client hosts are manager
|
||||
hosts. If there is more than one client host, then the manager must be
|
||||
configured to store its metadata on a shared file system that supports
|
||||
locking.
|
||||
|
||||
In a more complex multi-user, multi-host environment without a shared file
|
||||
system across all potential client hosts, one or more hosts must be
|
||||
designated as manager hosts. Since the manager is not an active server
|
||||
and is simply a passive executable invoked by clients, the manager can be
|
||||
located on any host with ssh access. These hosts must support either
|
||||
hostbased or pubkey authentication, however, since Shift is an automated
|
||||
framework that performs actions without the user present. The manager has
|
||||
a synchronization mechanism allowing multiple hosts without a shared file
|
||||
system to be used for redundancy.
|
||||
|
||||
|
||||
2. Prerequisites
|
||||
|
||||
2.1. Required
|
||||
|
||||
o perl >= 5.8.5 (>= 5.10.1 is required for multi-threading support)
|
||||
o ssh (and sftp) access to manager/remote hosts via hostbased or
|
||||
pubkey authentication
|
||||
|
||||
2.2. Optional
|
||||
|
||||
o mcp/msum >= 1.76.7 - high speed local copy/sum
|
||||
(http://mutil.sf.net)
|
||||
o bbcp - high speed remote copy
|
||||
(http://www.slac.stanford.edu/~abh/bbcp)
|
||||
o bbftp - high speed remote copy
|
||||
(http://doc.in2p3.fr/bbftp)
|
||||
o gridftp - high speed remote copy
|
||||
(http://toolkit.globus.org/toolkit/data/gridftp)
|
||||
o rsync - bandwidth-efficient local/remote copy
|
||||
(http://rsync.samba.org)
|
||||
o mesh - lightweight single sign-on via ssh pubkeys
|
||||
(http://mesh.sf.net)
|
||||
|
||||
2.3. Invoked (standard on most systems or when specific file systems in use)
|
||||
|
||||
o chown - change symlink ownership
|
||||
o crontab - install cron jobs to recover from host/process failures
|
||||
o df - get file system utilization
|
||||
o dm{get,put} - recall and migrate DMF-managed files
|
||||
o fallocate - preallocate files
|
||||
o {get,set}facl - get and set file ACLs
|
||||
o {get,set}fattr - get and set file extended attributes
|
||||
o lfs - get and set Lustre striping
|
||||
o lspci - find 10GE adapters
|
||||
o mmlsmgr - get GPFS server information
|
||||
o mount - get file system information
|
||||
o ping - determine network latency
|
||||
o ps - find clients and PBS processes, and determine client CPU load
|
||||
o su - become non-root process to access manager during root transfers
|
||||
o sysctl - determine number of cpus on BSD
|
||||
o touch - change symlink modification time
|
||||
o unbuffer - interleave stdout/stderr when using gridftp
|
||||
|
||||
|
||||
3. Installation
|
||||
|
||||
Note that the following shows the exact files needed on each type of host.
|
||||
Since the number of files is small, however, there is minimal penalty to
|
||||
simply installing them all on every host.
|
||||
|
||||
3.1. Single-user installation
|
||||
|
||||
Note that the user's home directory is used as the default install
|
||||
prefix in all examples, but can be changed to any other desired location
|
||||
as long as the corresponding bin directory is in the user's path.
|
||||
|
||||
3.1.1. Client hosts
|
||||
|
||||
install -m 700 perl/shiftc ~/bin/shiftc
|
||||
install -m 600 doc/shiftc.1 ~/man/man1/shiftc.1
|
||||
|
||||
3.1.2. Manager hosts
|
||||
|
||||
install -m 700 perl/shift-mgr ~/bin/shift-mgr
|
||||
install -m 600 etc/shiftrc ~/.shiftrc
|
||||
|
||||
3.1.3. Remote hosts (optional but recommended when possible)
|
||||
|
||||
install -m 700 perl/shift-aux ~/bin/shift-aux
|
||||
|
||||
3.2. Multi-user installation
|
||||
|
||||
Note that /usr/local is used as the default install prefix in all
|
||||
examples, but can be changed to any other desired location as long
|
||||
as the corresponding bin directory is in the default system path.
|
||||
|
||||
3.2.1. Client hosts
|
||||
|
||||
install -m 755 perl/shiftc /usr/local/bin/shiftc
|
||||
install -m 644 doc/shiftc.1 /usr/local/man/man1/shiftc.1
|
||||
|
||||
3.2.2. Manager hosts
|
||||
|
||||
install -m 755 perl/shift-mgr /usr/local/bin/shift-mgr
|
||||
install -m 644 etc/shiftrc /etc/shiftrc
|
||||
|
||||
3.2.3. Remote hosts (optional but recommended when possible)
|
||||
|
||||
install -m 755 perl/shift-aux /usr/local/bin/shift-aux
|
||||
|
||||
|
||||
4. Configuration
|
||||
|
||||
4.1. Client hosts
|
||||
|
||||
4.1.1. ~/.ssh/id_rsa (or similar)
|
||||
|
||||
If hostbased authentication is not supported by client hosts,
|
||||
manager hosts, and/or remote hosts, pubkey authentication must be
|
||||
used. Clients must have access to private keys to access these
|
||||
systems, which may either be referenced explicitly via the
|
||||
--mgr-identity and --identity options for manager and remote
|
||||
hosts, respectively, or be added to an ssh agent on the client(s).
|
||||
Note that the private keys used for these options must not be
|
||||
protected by a passphrase. The use of an ssh agent to access
|
||||
manager or remote hosts may be preferable from a security standpoint
|
||||
but comes with a drop in reliability as any failure of the agent or
|
||||
agent host leaves any associated transfers with no way to recover.
|
||||
|
||||
4.1.2. ~/.ssh/authorized_keys
|
||||
|
||||
If hostbased authentication is not supported to other client hosts
|
||||
for parallelization, pubkey authentication must be used. In this
|
||||
case, the public key(s) corresponding to the private key(s) used by
|
||||
clients (those named ~/.ssh/id* loaded into an ssh agent) must be
|
||||
added to the invoking user's authorized_keys file on other client
|
||||
hosts.
|
||||
|
||||
4.1.3. ~/bin/shiftc (single-user) or /usr/local/bin/shiftc (multi-user)
|
||||
|
||||
If the manager hosts differ from the client hosts, the manager
|
||||
host(s) can be hardcoded within the shiftc program in the
|
||||
"site-specific options" section. This allows the client to be used
|
||||
without specifying the --mgr option every time. For example, the
|
||||
line:
|
||||
|
||||
$opts{"mgr"} = "mgr.example.com";
|
||||
|
||||
would be equivalent to specifying the option "--mgr=mgr.example.com".
|
||||
In general, any shiftc command-line option --X can be hardcoded using:
|
||||
|
||||
$opts{"X"} = "hardcoded_value";
|
||||
|
||||
Those familiar with perl can add more complex logic (e.g. choosing
|
||||
which manager host out of a set of hosts will be used for each
|
||||
invocation).
|
||||
|
||||
4.2. Manager hosts
|
||||
|
||||
4.2.1. ~/.shiftrc (single-user) or /etc/shiftrc (multi-user)
|
||||
|
||||
All items in the default config file should be reviewed.
|
||||
The only required setting is:
|
||||
|
||||
user_dir
|
||||
|
||||
If there are multiple manager hosts, the configured directory must
|
||||
either be a file system shared across all manager hosts that
|
||||
supports file locking or the setting:
|
||||
|
||||
sync_host
|
||||
|
||||
must be configured to sync the transfer metadata across two
|
||||
manager hosts. Note that the existing synchronization
|
||||
mechanism has been found to be a bottleneck when there are
|
||||
many clients in a single transfer or many simultaneous
|
||||
transfers by a single user. This will be fixed in a future
|
||||
version.
|
||||
|
||||
The transport options should definitely be reviewed to
|
||||
enable any higher performance transports that may be
|
||||
available. Higher performance remote transports based on
|
||||
TCP connections (fish-tcp) and SSH connections (fish) are
|
||||
built-in, but require that the shift-aux executable exist
|
||||
on remote hosts. Note that transports can be enabled on a
|
||||
case-by-case basic using the client --local and --remote
|
||||
options.
|
||||
|
||||
To allow Shift to make parallelization decisions and enable
|
||||
functionality specific to certain remote file systems (e.g. Lustre
|
||||
striping), it is desirable to configure:
|
||||
|
||||
db_file
|
||||
|
||||
with a database of host and file system information from within
|
||||
the local environment. A template for producing this database is
|
||||
provided in the file "etc/shift-mounts.pl". After the items
|
||||
indicated in the script are configured, it can be run
|
||||
once/periodically to initialize/update the file system database.
|
||||
|
||||
To optimize access to remote hosts on the LAN, the setting:
|
||||
|
||||
select_hook
|
||||
|
||||
may be configured to specify how remote hosts should be selected
|
||||
when more than one is available. This may include deciding which
|
||||
hosts are up, which are least loaded, which have the best
|
||||
connectivity to the given client host, etc. A skeleton for a
|
||||
suitable selection hook is provided in the file
|
||||
"etc/shift-select.hook", but must be fleshed out based on
|
||||
site-specific knowledge and/or calls to the local load balancing
|
||||
infrastructure. If not configured, hosts will be chosen randomly
|
||||
after a successful sshd ping test.
|
||||
|
||||
4.2.2. ~/.ssh/authorized_keys
|
||||
|
||||
If hostbased authentication is not supported to manager hosts,
|
||||
pubkey authentication must be used. In this case, the public key(s)
|
||||
corresponding to the private key(s) used by clients (either loaded
|
||||
into an ssh agent or specified via --mgr-identity) must be added to
|
||||
the appropriate user's authorized_keys file on manager hosts.
|
||||
This user will either be the invoking user or the one specified by
|
||||
--mgr-user.
|
||||
|
||||
|
||||
4.3. Remote hosts
|
||||
|
||||
4.3.1. ~/.ssh/authorized_keys
|
||||
|
||||
If hostbased authentication is not supported to remote hosts, pubkey
|
||||
authentication must be used. In this case, the public key(s)
|
||||
corresponding to the private key(s) used by clients (either loaded
|
||||
into an ssh agent or specified via --identity) must be added to the
|
||||
appropriate user's authorized_keys file on remote hosts. This user
|
||||
will either be the invoking user or the one specified by --user.
|
||||
|
||||
|
||||
5. Usage
|
||||
|
||||
5.1. shiftc
|
||||
|
||||
Client usage is detailed in the man page "doc/shiftc.1", which can be
|
||||
viewed with:
|
||||
|
||||
nroff -man doc/shiftc.1 |less -r
|
||||
|
||||
if not already in a manpath directory. Basic usage is drop-in
|
||||
compatible with cp/scp with special consideration for the authentication
|
||||
options --mgr, --mgr-identity, --mgr-user, --identity, and --user.
|
||||
Note that the scp "user@host" syntax is not currently supported (the
|
||||
"user@" portion will be dropped) so --user must be specified instead
|
||||
when the remote user differs from the local user.
|
||||
|
||||
5.2. shift-mgr
|
||||
|
||||
Normal users need not invoke shift-mgr directly as all relevant manager
|
||||
functionality is accessed indirectly through the client. Additional
|
||||
capabilities are available for administrators, however, to aid in
|
||||
collecting usage stats and debugging information. Below is a sampling
|
||||
of some of the most useful administrative shift-mgr commands.
|
||||
|
||||
To see the status of a particular user X's transfers, run:
|
||||
|
||||
shift-mgr --user=X --status
|
||||
|
||||
To see the history of a particular user X's transfers, run:
|
||||
|
||||
shift-mgr --user=X --history
|
||||
|
||||
To see the status of running transfers across all users, run:
|
||||
|
||||
shift-mgr --status --state=run
|
||||
|
||||
To see detailed stats across the transfers of all users including a
|
||||
sampling of error messages, run:
|
||||
|
||||
shift-mgr --stats
|
||||
|
||||
This can be added to a cron job (normally at the interval of the
|
||||
"data_expire" setting in shiftrc) with a redirection to a file to
|
||||
periodically save usage stats.
|
||||
|
||||
To see the metadata associated with a given user X's transfer T, run:
|
||||
|
||||
shift-mgr --user=X --id=T --meta
|
||||
|
||||
The metadata is log structured, so it is also possible to view the
|
||||
metadata at any number of steps S back in time using:
|
||||
|
||||
shift-mgr --user=X --id=T --meta=S
|
||||
41
README.md
41
README.md
|
|
@ -1 +1,40 @@
|
|||
# shift
|
||||
Self-Healing Independent File Transfer (Shift)
|
||||
==============================================
|
||||
|
||||
In high-end computing environments, remote file transfers of very large data
|
||||
sets to and from computational resources are commonplace as users are typically
|
||||
widely distributed across different organizations and must transfer in data to
|
||||
be processed and transfer out results for further analysis. Local transfers
|
||||
of this same data across file systems are also frequently performed by
|
||||
administrators to optimize resource utilization when new file systems come
|
||||
on-line or storage becomes imbalanced between existing file systems. In both
|
||||
cases, files must traverse many components on their journey from source to
|
||||
destination where there are numerous opportunities for performance optimization
|
||||
as well as failure. A number of tools exist for providing reliable and/or high
|
||||
performance file transfer capabilities, but most either do not support local
|
||||
transfers, require specific security models and/or transport applications, are
|
||||
difficult for individual users to deploy, and/or are not fully optimized for
|
||||
highest performance.
|
||||
|
||||
Shift is a framework for Self-Healing Independent File Transfer that provides
|
||||
high performance and resilience for local and remote transfers through a variety
|
||||
of techniques. These include end-to-end integrity via cryptographic hashes,
|
||||
throttling of transfers to prevent resource exhaustion, balancing transfers
|
||||
across resources based on load and availability, and parallelization of
|
||||
transfers across multiple source and destination hosts for increased redundancy
|
||||
and performance. In addition, Shift was specifically designed to accommodate
|
||||
the diverse heterogeneous environments of a widespread user base with minimal
|
||||
assumptions about operating environments. In particular, Shift is unique in its
|
||||
ability to provide advanced reliability and automatic single and multi-file
|
||||
parallelization to any stock command-line transfer application while being
|
||||
easily deployed by both individual users as well as entire organizations.
|
||||
|
||||
For full details of the Shift architecture, see
|
||||
https://pkolano.github.io/papers/resilience12.pdf. For installation
|
||||
details, see "INSTALL". For usage details, see "doc/shiftc.1" (in man
|
||||
page format, viewable with "nroff -man").
|
||||
|
||||
Questions, comments, fixes, and/or enhancements welcome.
|
||||
|
||||
--Paul Kolano <paul.kolano@nasa.gov>
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,950 @@
|
|||
.TH "shiftc" "1" "02 Dec 2015" "" ""
|
||||
./"################################################################
|
||||
.SH "NAME"
|
||||
./"################################################################
|
||||
\fBshiftc\fP \(em a high performance reliable file transfer tool
|
||||
./"################################################################
|
||||
.SH "SYNOPSIS"
|
||||
./"################################################################
|
||||
.nf
|
||||
\fBshiftc\fP [OPTION]... SOURCE DEST
|
||||
\fBshiftc\fP [OPTION]... SOURCE... DIRECTORY
|
||||
\fBshiftc\fP [OPTION]...
|
||||
.fi
|
||||
.PP
|
||||
Reliably transfer SOURCE to DEST, multiple SOURCE(s) to DIRECTORY, or
|
||||
arbitrary SOURCE to DEST and/or SOURCE(s) to DIRECTORY combinations
|
||||
read from stdin. By default, symbolic links to files on the command
|
||||
line are followed, but symbolic links to directories are not (identical
|
||||
to the default behavior of cp).
|
||||
.PP
|
||||
Local paths are specified normally. A path PATH on a remote host HOST
|
||||
is specified using scp-style "HOST:PATH". Note that transfers between
|
||||
two remote hosts are not supported.
|
||||
./"################################################################
|
||||
.SH "DESCRIPTION"
|
||||
./"################################################################
|
||||
.PP
|
||||
shiftc is the client for Shift, which is a framework for
|
||||
\fBS\fPelf-\fBH\fPealing \fBI\fPndependent \fBF\fPile \fBT\fPransfers. Shift
|
||||
includes the following features, among others:
|
||||
.IP -
|
||||
support for local, LAN, and WAN transfers
|
||||
.IP -
|
||||
drop-in replacement for both cp and scp (basic options only)
|
||||
.IP -
|
||||
tracking of individual file operations with on-demand status
|
||||
.IP -
|
||||
transfer stop and restart
|
||||
.IP -
|
||||
email notification of completion, errors, and warnings
|
||||
.IP -
|
||||
local and remote tar creation/extraction
|
||||
.IP -
|
||||
rsync-like synchronization based on modification times and checksums
|
||||
.IP -
|
||||
integrity verification of transfers with partial retransfer/resum to
|
||||
rectify corruption
|
||||
.IP -
|
||||
throttling based on local and remote resource utilization
|
||||
.IP -
|
||||
automatic retrieval and release of files residing on DMF-managed file
|
||||
systems
|
||||
.IP -
|
||||
automatic striping of files transferred to Lustre file systems
|
||||
.IP -
|
||||
fully self-contained besides perl core and ssh
|
||||
.IP -
|
||||
automatic detection and selection of higher performance transports and
|
||||
hash utilities when available including bbcp, bbftp, gridftp, mcp,
|
||||
msum, and rsync
|
||||
.IP -
|
||||
automatic many-to-many parallelization of single and multi-file
|
||||
transfers with file system equivalence detection and rewriting
|
||||
.PP
|
||||
To support these features, shiftc communicates with a manager component
|
||||
that is responsible for tracking transfers and directing the client to
|
||||
process batches of file operations and return the results.
|
||||
./"################################################################
|
||||
.SH "OPTIONS SUMMARY"
|
||||
./"################################################################
|
||||
The following options are available in shiftc. Defaults are shown in
|
||||
brackets. Some options have short and long variants separated by
|
||||
commas. The \(cq\&=\(cq\& for options that take a parameter is
|
||||
optional; whitespace may be used instead. Detailed descriptions are
|
||||
given in following sections.
|
||||
.PP
|
||||
.nf
|
||||
\fBInitialization options (defaults in brackets):\fP
|
||||
\-\-clients=NUM use at most NUM clients per host [1]
|
||||
\-\-create\-tar create tar file of SOURCE(s) at DEST
|
||||
\-L, \-\-dereference always follow symbolic links
|
||||
\-d, \-\-directory create any missing parent directories
|
||||
\-\-exclude=REGEX exclude files matching REGEX
|
||||
\-\-extract\-tar extract tar file(s) at SOURCE to DEST
|
||||
\-h, \-\-help help
|
||||
\-\-host\-file=FILE parallelize transfer on hosts in FILE (one per line)
|
||||
\-\-host\-list=LIST parallelize transfer on hosts in LIST
|
||||
\-\-hosts=NUM parallelize transfer on at most NUM client hosts [1]
|
||||
\-\-identity=FILE access remote systems with ssh identity in FILE
|
||||
\-I, \-\-ignore\-times do not skip files that match size and time
|
||||
\-\-include=REGEX include only files matching REGEX
|
||||
\-\-index\-tar create table of contents during tar creation
|
||||
\-\-newer=DATE include only files with mtime newer than DATE
|
||||
\-P, \-\-no\-dereference never follow symbolic links
|
||||
\-T, \-\-no\-target\-directory treat target as a normal file
|
||||
\-\-older=DATE include only files with mtime older than DATE
|
||||
\-\-ports=NUM1:NUM2 use ports NUM1\-NUM2 for remote TCP\-based transports
|
||||
\-R, \-r, \-\-recursive copy directories recursively
|
||||
\-\-secure encrypt data stream(s) and use secure ciphers/macs
|
||||
\-\-sync synchronize files at destination
|
||||
\-\-user=USER access remote systems as USER
|
||||
\-\-wait block until transfer completes
|
||||
(exit 0 = success, 1 = failure)
|
||||
.PP
|
||||
\fBFeature\-disablement options:\fP
|
||||
\-\-no\-check do not check file existence/size (benchmarking only)
|
||||
\-\-no\-cron do not recover from host/process failures via cron
|
||||
\-\-no\-mail do not send status emails
|
||||
\-\-no\-offline do not migrate DMF\-managed files after transfer
|
||||
\-\-no\-preserve do not preserve times, mode, owner, acls, or xattrs
|
||||
\-\-no\-verify do not verify/rectify integrity of destination files
|
||||
.PP
|
||||
\fBMonitoring/management options:\fP
|
||||
\-\-history show list of transfer commands and origin host/dir
|
||||
\-\-id=NUM use transfer identifier NUM for other commands
|
||||
\-\-mgr=HOST set host of shift manager to HOST
|
||||
\-\-mgr\-identity=FILE access manager host with ssh identity in FILE
|
||||
\-\-mgr\-user=USER access manager host as USER
|
||||
\-\-restart restart transfer with given \-\-id
|
||||
\-\-search=REGEX show only status/history matching REGEX
|
||||
\-\-state=STATE show status of only those operations in STATE
|
||||
(STATE one of {done,error,none,queue,run,warn})
|
||||
\-\-stats show stats across all transfers
|
||||
\-\-status[={csv,pad}] show brief status of all transfers
|
||||
\-\-stop stop transfer with given \-\-id
|
||||
or detailed status of transfer with given \-\-id
|
||||
.PP
|
||||
\fBTuning options (defaults in brackets):\fP
|
||||
\-\-bandwidth=BITS tune TCP\-based transports based on BITS per second
|
||||
(use suffix {k,m,g,t} for {Kb,Mb,Gb,Tb})
|
||||
\-\-buffer=SIZE use SIZE bytes for buffer in transports
|
||||
(use suffix {k,m,g,t} for {KiB,MiB,GiB,TiB}) [4m]
|
||||
\-\-files=COUNT process transfer in batches of COUNT files
|
||||
(use suffix {k,m,b/g,t} for 1E{3,6,9,12}) [1k]
|
||||
\-\-local=LIST set local transport mechanism to one of LIST
|
||||
(LIST subset of {bbcp,bbftp,fish,fish-tcp,gridftp,
|
||||
mcp,rsync,shift})
|
||||
\-\-preallocate=NUM preallocate files when sparsity under NUM percent
|
||||
\-\-remote=LIST set remote transport mechanism to one of LIST
|
||||
(LIST subset of {bbcp,bbftp,fish,fish-tcp,gridftp,
|
||||
rsync,shift})
|
||||
\-\-retry=NUM retry failed operations up to NUM times [2]
|
||||
\-\-size=SIZE process transfer in batches of SIZE bytes
|
||||
(use suffix {k,m,g,t} for {KB,MB,GB,TB}) [4g]
|
||||
\-\-split=SIZE parallelize single files using chunks of SIZE bytes
|
||||
(use suffix {k,m,g,t} for {KiB,MiB,GiB,TiB}) [0]
|
||||
\-\-split\-tar=SIZE create tar files of around SIZE bytes
|
||||
(use suffix {k,m,g,t} for {KB,MB,GB,TB}) [500g]
|
||||
\-\-streams=NUM use NUM streams in remote transports [4]
|
||||
\-\-stripe=SIZE|NUM use 1 stripe per SIZE bytes or NUM stripes
|
||||
(use suffix {k,m,g,t} for {KB,MB,GB,TB}) [1g]
|
||||
\-\-threads=NUM use NUM threads in local transports [4]
|
||||
\-\-verify\-fast verify faster but less safely by reusing src buffer
|
||||
\-\-window=SIZE use SIZE bytes for window in TCP\-based transports
|
||||
(use suffix {k,m,g,t} for {KB,MB,GB,TB}) [4m]
|
||||
.PP
|
||||
\fBThrottling options:\fP
|
||||
\-\-cpu=NUM throttle local cpu usage at NUM %
|
||||
\-\-disk=NUM1:NUM2 suspend/resume transfer when target NUM1%/NUM2% full
|
||||
\-\-io=NUM throttle local i/o usage at NUM MB/s
|
||||
\-\-ior=NUM throttle local i/o reads at NUM MB/s
|
||||
\-\-iow=NUM throttle local i/o writes at NUM MB/s
|
||||
\-\-net=NUM throttle local network usage at NUM MB/s
|
||||
\-\-netr=NUM throttle local network reads at NUM MB/s
|
||||
\-\-netw=NUM throttle local network writes at NUM MB/s
|
||||
.fi
|
||||
./"################################################################
|
||||
.SH "TRANSFER INITIALIZATION"
|
||||
./"################################################################
|
||||
Transfers are initialized using syntax identical to cp/scp for
|
||||
local/remote transfers, respectively. The most commonly used options
|
||||
during initialization are listed below.
|
||||
.IP "\fB\-\-clients=NUM\fP"
|
||||
Parallelize the transfer by using additional clients on each host. If
|
||||
the number given is one, no additional clients will be used. A number
|
||||
greater than one will fork additional processes on each host to more
|
||||
fully utilize system resources and increase transfer performance.
|
||||
.IP "\fB\-\-create\-tar\fP"
|
||||
Create a tar file of all sources at the destination, which must be a
|
||||
non-existing file name. This option implies \fB\-\-recursive\fP and
|
||||
\fB\-\-no\-offline\fP. By default, multiple tar files are created at
|
||||
500 GB boundaries. The split size may be changed or splitting disabled
|
||||
using the \fB\-\-split\-tar\fP option. The \fB\-\-index\-tar\fP option
|
||||
may be used to produce a table of contents file for each tar file
|
||||
created. Note that this option cannot be used with \fB\-\-sync\fP.
|
||||
.IP "\fB\-L, \-\-dereference\fP"
|
||||
Always follow symbolic links to both files and directories. Note that
|
||||
this can result in file and directory duplication at the destination as
|
||||
all symbolic links will become real files and directories.
|
||||
.IP "\fB\-d, \-\-directory\fP"
|
||||
Create any missing parent directories. This option allows files to be
|
||||
transferred to a directory hierarchy that may not already exist, similar
|
||||
to the \fB\-d\fP option of the "install" command.
|
||||
.IP "\fB\-\-exclude=REGEX\fP"
|
||||
Do not transfer source files matching the given regular expression.
|
||||
Note that regular expressions must be given in Perl syntax (see
|
||||
perlre(1) for details) and should be quoted on the command line when
|
||||
including characters normally expanded by the shell (e.g. "*"). Shell
|
||||
wildcard behavior can be approximated by using ".*" in place of "*".
|
||||
.IP "\fB\-\-extract\-tar\fP"
|
||||
Extract all source tar files to the destination, which must be an
|
||||
existing directory or non-existing directory name. This option implies
|
||||
\fB\-\-no\-offline\fP. Note that only tar archives in the POSIX ustar
|
||||
format are supported, but GNU extensions for large uids, gids, file
|
||||
sizes, and file names are handled appropriately. Also note that this
|
||||
option cannot be used with \fB\-\-sync\fP.
|
||||
.IP "\fB\-\-host\-file=FILE\fP"
|
||||
Parallelize the transfer by using additional clients on the hosts
|
||||
specified in the given file (one host name per line). This option
|
||||
implies a \fB\-\-hosts\fP value equal to the number of hosts in the file
|
||||
plus any additional hosts from the \fB\-\-host\-list\fP option. Less
|
||||
hosts may be used by explicitly specifying a \fB\-\-hosts\fP value.
|
||||
Note that the actual number of client hosts used will depend upon number
|
||||
of hosts that have equivalent access to the source and/or destination
|
||||
file systems. Within PBS job scripts, this option can be set to the
|
||||
$PBS_NODEFILE variable to use all nodes of the job.
|
||||
.IP "\fB\-\-host\-list=LIST\fP"
|
||||
Parallelize the transfer by using additional clients on the hosts
|
||||
specified in the given comma-separated list. This option implies a
|
||||
\fB\-\-hosts\fP value equal to the number of hosts on the list plus any
|
||||
additional hosts from the \fB\-\-host\-file\fP option. Less hosts may
|
||||
be used by explicitly specifying a \fB\-\-hosts\fP value. Note that the
|
||||
actual number of client hosts used will depend upon number of hosts that
|
||||
have equivalent access to the source and/or destination file systems.
|
||||
.IP "\fB\-\-hosts=NUM\fP"
|
||||
Parallelize the transfer by using additional clients on at most the
|
||||
given number of hosts. If the number given is one, no additional
|
||||
client hosts will be used. A number greater than one enables automatic
|
||||
transfer parallelization where additional clients may be invoked on
|
||||
additional hosts to increase transfer performance. Note that the actual
|
||||
number of client hosts used will depend upon the number of hosts for
|
||||
which Shift has file system information and the number of hosts that
|
||||
have equivalent access to the source and/or destination file systems.
|
||||
Client hosts will be accessed as the current user with hostbased
|
||||
authentication or an existing ssh agent that contains an ssh identity
|
||||
from a file matching ~/.ssh/id*.
|
||||
.IP "\fB\-\-identity=FILE\fP"
|
||||
Authenticate to remote systems using the given ssh identity file.
|
||||
The corresponding public key must reside in the appropriate user's
|
||||
~/.ssh/authorized_keys file on the remote host. Note that only
|
||||
identity files without passphrases are supported. If a passphrase is
|
||||
required, an ssh agent may be used instead, but with a loss of
|
||||
reliability. This option is not needed if the remote host accepts
|
||||
hostbased authentication from client hosts.
|
||||
.IP "\fB\-I, \-\-ignore\-times\fP"
|
||||
By default, the \fB\-\-sync\fP option skips the processing of files
|
||||
that have the same size and modification time at the source and
|
||||
destination. This option specifies that files should always be
|
||||
processed by checksum regardless of size and modification time.
|
||||
.IP "\fB\-\-include=REGEX\fP"
|
||||
Only transfer source files matching the given regular expression.
|
||||
Note that regular expressions must be given in Perl syntax (see
|
||||
perlre(1) for details) and should be quoted on the command line when
|
||||
including characters normally expanded by the shell (e.g. "*"). Shell
|
||||
wildcard behavior can be approximated by using ".*" in place of "*".
|
||||
.IP "\fB\-\-index\-tar\fP"
|
||||
Create a table of contents file for each tar file created with
|
||||
\fB\-\-create\-tar\fP. The table of contents will show each file in the
|
||||
tar file along with permissions, user/group ownership, and size. For a
|
||||
tar file "file.tar", the table of contents will be named "file.tar.toc".
|
||||
Unless the \fB\-\-no\-verify\fP option is used, a checksum file will
|
||||
also be created named "file.tar.sum", which is suitable as input for
|
||||
"msum --check-tree -c". Note that when \fB\-\-split\-tar\fP is used,
|
||||
multiple table of contents and checksum files may be created. For each
|
||||
split tar file "file.tar-i.tar", the table of contents will be named
|
||||
"file.tar-i.tar.toc" and the checksum file will be named
|
||||
"file.tar-i.tar.sum".
|
||||
.IP "\fB\-\-newer=DATE\fP"
|
||||
Only transfer source files whose modification time is newer (inclusive)
|
||||
than the given date. Any date string supported by the Perl
|
||||
Date::Parse module can be specified. Note that this option can be
|
||||
combined with \fB\-\-older\fP to specify exact date ranges.
|
||||
.IP "\fB\-P, \-\-no\-dereference\fP"
|
||||
Never follow symbolic links to file or directories. Note that this
|
||||
can result in broken links at the destination as files and directories
|
||||
referenced by symbolic links that were not explicitly transferred or
|
||||
implicitly transferred using \fB\-\-recursive\fP may not exist on the
|
||||
target.
|
||||
.IP "\fB\-T, \-\-no\-target\-directory\fP"
|
||||
Do not treat the destination specially when it is a directory or a
|
||||
symbolic link to a directory. This option can be used with recursive
|
||||
transfers to copy a directory's contents into an existing directory
|
||||
instead of into a new subdirectory beneath it as is done by default.
|
||||
.IP "\fB\-\-older=DATE\fP"
|
||||
Only transfer source files whose modification time is older than the
|
||||
given date. Any date string supported by the Perl Date::Parse module
|
||||
can be specified. Note that this option can be combined with
|
||||
\fB\-\-newer\fP to specify exact date ranges.
|
||||
.IP "\fB\-\-ports=NUM1:NUM2\fP"
|
||||
Use ports from the range NUM1-NUM2 for the data streams of TCP-based
|
||||
transports (currently, bbcp, bbftp, fish-tcp, and gridftp). All
|
||||
connections originate from the client host so the given port range must
|
||||
be allowed on the network path to the remote host and by the remote host
|
||||
itself.
|
||||
.IP "\fB\-R, \-r, \-\-recursive\fP"
|
||||
Transfer directories recursively. This option implies
|
||||
\fB\-\-no\-dereference\fP.Note that any symbolic links pointing
|
||||
to directories given on the command line will be followed during
|
||||
recursive transfers (identical to the default behavior of cp).
|
||||
.IP "\fB\-\-secure\fP"
|
||||
Encrypt data during remote transfers and use secure ciphers and MACs
|
||||
with SSH-based tranports. Note that this option will, in most cases,
|
||||
decrease performance as it eliminates some higher performance transports
|
||||
and increases CPU utilization during SSH connections.
|
||||
.IP "\fB\-\-sync\fP"
|
||||
Synchronize files between the source and destination, similar to the
|
||||
rsync command. By default, files that have the same size and
|
||||
modification time at the source and destination will not be transferred.
|
||||
If the size or modification time of a file differs between the two, the
|
||||
contents of the file will be compared via checksum and any portions that
|
||||
differ will be transferred to the destination. To skip the size and
|
||||
modification time checks and always begin with the checksum stage, use
|
||||
\fB\-I\fP or \fB\-\-ignore\-times\fP. If \fB\-\-no\-verify\fP is
|
||||
specified, integrity verification is not performed, which will increase
|
||||
performance when there are many files at the source that are not at
|
||||
the destination but will decrease performance when there are large files
|
||||
that have only small changes between the source and destination.
|
||||
Setting \fB\-\-retry\fP to zero with this option can be used to show
|
||||
which files differ without making any changes. Note that when syncing
|
||||
directories, the destination should be specified as the parent of the
|
||||
location where the source directory should be transferred to. Also note
|
||||
that this option cannot be used with \fB\-\-create\-tar\fP or
|
||||
\fB\-\-extract\-tar\fB.
|
||||
.IP "\fB\-\-user=USER\fP"
|
||||
Set the user that will be used to access remote systems.
|
||||
.IP "\fB\-\-wait\fP"
|
||||
Block until the transfer completes and print a summary of the transfer.
|
||||
This option implies \fB\-\-no\-mail\fP. An exit value of 0 indicates
|
||||
that the transfer has successfully completed while an exit value of 1
|
||||
indicates that the transfer has failed or that the waiting process was
|
||||
terminated prematurely.
|
||||
./"################################################################
|
||||
.SH "FEATURE DISABLEMENT
|
||||
./"################################################################
|
||||
.IP "\fB\-\-no\-check\fP"
|
||||
Disable file existence and size checks at the end of the transfer.
|
||||
This option was included for benchmarking and completeness purposes
|
||||
and is not recommended for general use.
|
||||
.IP "\fB\-\-no\-cron\fP"
|
||||
Do not attempt to recover from host/process failures via cron. Note
|
||||
that when such a failure occurs, the transfer will become stuck in the
|
||||
"run" state until stopped.
|
||||
.IP "\fB\-\-no\-mail\fP"
|
||||
Prevents sending of emails due to errors, warnings, or completion.
|
||||
This option may be desirable when performing a large number of scripted
|
||||
transfers. Note that equivalent transfer status and history information
|
||||
can always be manually retrieved using \fB\-\-status\fP and
|
||||
\fB\-\-history\fP, respectively.
|
||||
.IP "\fB\-\-no\-offline\fP"
|
||||
By default, files transferred to/from DMF-managed file systems will be
|
||||
migrated to offline media as soon as the transfer completes. This
|
||||
option specifies that files should not be migrated. Note that DMF may
|
||||
still choose to migrate (and possibly release) files even when this
|
||||
option is enabled.
|
||||
.IP "\fB\-\-no\-preserve\fP"
|
||||
By default, times, permissions, ownership, ACLs, and extended attributes
|
||||
of transferred files and directories are preserved when possible.
|
||||
This option specifies that these items should not be preserved. Note
|
||||
that permissions may be left in various states depending on the invoking
|
||||
user's umask and the transport utilized. In particular, read access at
|
||||
the destination may be more permissive than read access at the source.
|
||||
.IP "\fB\-\-no\-verify\fP"
|
||||
By default, files are checksummed at the source and destination to
|
||||
verify that they have not been corrupted and if corruption is detected,
|
||||
the corrupted portion of the destination file is automatically corrected
|
||||
using a partial transfer from the original source. This functionality
|
||||
decreases the performance of transfers in proportion to the file size.
|
||||
If assurance of integrity is not required, the \fB\-\-no\-verify\fP
|
||||
option may be used to disable verification.
|
||||
./"################################################################
|
||||
.SH "TRANSFER MONITORING AND MANAGEMENT
|
||||
./"################################################################
|
||||
Once one or more transfers have been initialized, the user may view
|
||||
transfer history, stop/restart transfers, and/or check transfer status
|
||||
with the following options.
|
||||
.IP "\fB\-\-history\fP"
|
||||
Show a brief history of all transfers including the transfer identifier,
|
||||
the origin host/directory and the original command.
|
||||
.IP "\fB\-\-id=NUM\fP"
|
||||
Specify the transfer identifier to be used with management and status
|
||||
commands.
|
||||
.IP "\fB\-\-mgr=HOST\fP"
|
||||
Set the host that will be used to manage transfers. By default, this
|
||||
host will be accessed as the current user with hostbased authentication
|
||||
or an existing ssh agent. The user and/or identity used to access the
|
||||
manager host may be changed with the \fB\-\-mgr\-user\fP and
|
||||
\fB\-\-mgr\-identity\fP options, respectively.
|
||||
.IP "\fB\-\-mgr\-identity=FILE\fP"
|
||||
Authenticate to the manager host using the given ssh identity file.
|
||||
The corresponding public key must reside in the appropriate user's
|
||||
~/.ssh/authorized_keys file on the manager host. Note that only
|
||||
identity files without passphrases are supported. If a passphrase is
|
||||
required, an ssh agent may be used instead, but with a loss of
|
||||
reliability. This option is not needed if the manager host accepts
|
||||
hostbased authentication from client hosts.
|
||||
.IP "\fB\-\-mgr\-user=USER\fP"
|
||||
Set the user that will be used to access the manager host. Note that if
|
||||
the transfer is initiated by root and \fB\-\-mgr\-identity\fP is not
|
||||
specified, manager communication will be performed as the given user
|
||||
so that user must be authorized to run processes locally. In
|
||||
particular, care should be taken on PBS-controlled nodes, where the
|
||||
given user should either own the node or be on the user exception list.
|
||||
.IP "\fB\-\-restart\fP"
|
||||
Restart the transfer associated with the given \fB\-\-id\fP that was
|
||||
stopped due to unrecoverable errors or stopped explicitly via
|
||||
\fB\-\-stop\fP. Note that transfers must be restarted on the original
|
||||
client host or one that has equivalent file system access. A subset of
|
||||
the available command-line options may be respecified during a restart
|
||||
including \fB\-\-bandwidth\fP, \fB\-\-buffer\fP, \fB\-\-clients\fP,
|
||||
\fB\-\-cpu\fP, \fB\-\-disk\fP, \fB\-\-files\fP, \fB\-\-host-file\fP,
|
||||
\fB\-\-host-list\fP, \fB\-\-hosts\fP, \fB\-\-io\fP, \fB\-\-ior\fP,
|
||||
\fB\-\-iow\fP, \fB\-\-local\fP, \fB\-\-net\fP, \fB\-\-netr\fP,
|
||||
\fB\-\-netw\fP, \fB\-\-no\-cron\fP, \fB\-\-no\-mail\fP,
|
||||
\fB\-\-no\-offline\fP, \fB\-\-ports\fP, \fB\-\-preallocate\fP,
|
||||
\fB\-\-remote\fP, \fB\-\-retry\fP, \fB\-\-secure\fP, \fB\-\-size\fP,
|
||||
\fB\-\-streams\fP, \fB\-\-stripe\fP, \fB\-\-threads\fP, and \fB\-\-window\fP.
|
||||
.IP "\fB\-\-search=REGEX\fP"
|
||||
When \fB\-\-status\fP and \fB\-\-id\fP are specified, this option will
|
||||
show the full status of file operations in the associated transfer whose
|
||||
source or destination file name match the given regular expression.
|
||||
.IP
|
||||
When \fB\-\-history\fP is specified, this option will show a brief
|
||||
history of the transfers whose origin host or original command match the
|
||||
given regular expression.
|
||||
.IP
|
||||
Note that regular expressions must be given in Perl syntax (see
|
||||
perlre(1) for details).
|
||||
.IP "\fB\-\-state=STATE\fP"
|
||||
When \fB\-\-status\fP and \fB\-\-id\fP are specified, this option will
|
||||
show the full status of file operations in the associated transfer that
|
||||
have the given state. When \fB\-\-id\fP is not specified, this option
|
||||
will show the brief status of transfers in the given state. Valid
|
||||
states are done, error, none, queue, run, and warn. A state of "none"
|
||||
will show a summary of the given transfer.
|
||||
.IP "\fB\-\-stats\fP"
|
||||
Show stats across all transfers including transfer counts, rates, tool
|
||||
usage, initialization options, error counts, and error messages.
|
||||
.IP "\fB\-\-status[={csv,pad}]\fP"
|
||||
Show a brief status of all transfers including the transfer identifier,
|
||||
the current state, the number of directories completed, the number of
|
||||
files transferred, the number of files checksummed, the number of
|
||||
attributes preserved, the amount of data transferred, the amount of data
|
||||
checksummed, the time the transfer started, the duration of the
|
||||
transfer, the estimated time remaining in the transfer, and the rate of
|
||||
the transfer. When the number of transfers exceeds a set threshold (20
|
||||
by default), older successfully completed transfers beyond that limit
|
||||
will be omitted for readability. These omitted transfers can be shown
|
||||
using \fB\-\-status\fP with \fB\-\-state=done\fP. Status will be
|
||||
returned in CSV format when \fB\-\-status=csv\fP is specified. Duration
|
||||
and estimated time will be zero-padded when \fB\-\-status=pad\fP is
|
||||
specified.
|
||||
.IP
|
||||
When \fB\-\-id\fP is specified, this option will show the full status of
|
||||
every file operation in the associated transfer. For each operation,
|
||||
this includes the state, the type, the tool used for processing, the
|
||||
target path, associated information (error messages, checksums, byte
|
||||
ranges, and/or running host) when applicable, the size of the file,
|
||||
the time processing started, and the rate of the operation. Note that
|
||||
not all of these items will be applicable at all times (e.g. rate will
|
||||
be empty if the state is error). Also note that operations are
|
||||
processed in batches so the rate shown for a single operation will
|
||||
depend on the other operations processed in the same batch.
|
||||
.IP "\fB\-\-stop\fP"
|
||||
Stop the transfer associated with the given \fB\-\-id\fP. Note that
|
||||
transfer operations currently in progress will run to completion but new
|
||||
operations will not be processed. Stopped transfers may be restarted
|
||||
with \fB\-\-restart\fP.
|
||||
./"################################################################
|
||||
.SH "TRANSFER TUNING"
|
||||
./"################################################################
|
||||
Some advanced options are available to tune various aspects of shiftc
|
||||
behavior. These options are not needed by most users.
|
||||
.IP "\fB\-\-bandwidth=BITS\fP"
|
||||
Choose the TCP window size and number of TCP streams of TCP-based
|
||||
transports (currently, bbcp, bbftp, fish-tcp, and gridftp) based on
|
||||
the given bits per second. The suffixes k, m, g, and t may be used for
|
||||
Kb, Mb, Gb, and Tb, respectively. The default bandwidth is estimated to
|
||||
be 10 Gb/s if a 10 GE adapter is found on the client host, 1 Gb/s if the
|
||||
client host can be resolved to an organization domain (by default, one
|
||||
of the six original generic top-level domains), and 100 Mb/s otherwise.
|
||||
.IP "\fB\-\-buffer=SIZE\fP"
|
||||
Use memory buffer(s) of the given size when configurable in the
|
||||
underlying tranport being utilized (currently, all but rsync). The
|
||||
suffixes k, m, g, and t may be used for KiB, MiB, GiB, and TiB,
|
||||
respectively. The default buffer size is 4 MiB. Increasing the
|
||||
buffer size trades higher memory utilization for more efficient I/O.
|
||||
.IP "\fB\-\-files=COUNT\fP"
|
||||
Process transfers in batches of the given number of files. The
|
||||
suffixes k, m, b or g, and t may be used for 1E3, 1E6, 1E9, and 1E12,
|
||||
respectively. The default batch count is 1000 files. Lowering the
|
||||
batch count will increase the number of checkpoints and the overhead of
|
||||
transfer management. Raising the batch count will have the opposite
|
||||
effect. A batch will be sent for processing when the number of files in
|
||||
the batch reaches the given value. Note that batches of less than the
|
||||
given count can occur if the batch size specified by \fB\-\-size\fP is
|
||||
reached first.
|
||||
.IP "\fB\-\-local=LIST\fP"
|
||||
Specify one or more local transports to be used for the transfer in
|
||||
order of preference, separated by commas. Valid transports for this
|
||||
option currently include bbcp, bbftp, cp, fish, fish-tcp, gridftp,
|
||||
mcp, and rsync. Note that the given transport(s) will be given
|
||||
priority, but may not be used in some cases (e.g. rsync is not capable
|
||||
of transferring a specific portion of a file as needed by verification
|
||||
mode). In such cases, the default transport based on File::Copy will be
|
||||
used. The tool actually used for each file operation can be shown using
|
||||
\fB\-\-status\fP with \fB\-\-id\fP set to the given transfer identifier.
|
||||
.IP "\fB\-\-preallocate=NUM\fP"
|
||||
Preallocate files when their sparsity is under the given percent, where
|
||||
sparsity is defined as the number of bytes a file takes up on disk
|
||||
divided by its size. Note that this option will only have an effect
|
||||
when the fallocate command is available, the destination file does not
|
||||
already exist, and the target file system properly supports fallocate's
|
||||
-n option. Also note that this option will not function properly when
|
||||
either bbftp or rsync (to a DMF file system) is utilized as the
|
||||
transport due to their use of temporary files.
|
||||
.IP "\fB\-\-remote=LIST\fP"
|
||||
Specify one or more remote transports to be used for the transfer in
|
||||
order of preference, separated by commas. Valid transports for this
|
||||
option currently include bbcp, bbftp, fish, fish-tcp, gridftp, rsync,
|
||||
and sftp. Note that the given transport(s) will be given priority, but
|
||||
may not be used in some cases (e.g. bbftp is not capable of transferring
|
||||
files with spaces in their names and is also incompatible with
|
||||
\fB\-\-secure\fP). In such cases, the default transport based on sftp
|
||||
will be used. The tool actually used for each file operation can be
|
||||
shown using \fB\-\-status\fP with \fB\-\-id\fP set to the given transfer
|
||||
identifier.
|
||||
.IP "\fB\-\-retry=NUM\fP"
|
||||
Retry operations deemed recoverable up to the given number of attempts
|
||||
per file. The default number of retries is 2. A value of zero disables
|
||||
retries. Note that disabling retries also disables the ability of
|
||||
\fB\-\-sync\fP to change file contents. Also note that the given
|
||||
value is cumulative across all stages of a file's processing so
|
||||
different stages may not be retried the same number of times.
|
||||
.IP "\fB\-\-size=SIZE\fP"
|
||||
Process transfers in batches of approximately the given total file size.
|
||||
The suffixes k, m, g, and t may be used for KB, MB, GB, and TB,
|
||||
respectively. The default batch size is 4 GB. Lowering the batch size
|
||||
will increase the number of checkpoints and the overhead of transfer
|
||||
management. Raising the batch size will have the opposite effect. A
|
||||
batch will be sent for processing when the total size of all files in
|
||||
the batch reaches the given value. Note that batches of less than the
|
||||
given size can occur if the batch count specified by \fB\-\-files\fP
|
||||
is reached first.
|
||||
.IP "\fB\-\-split=SIZE\fP"
|
||||
Parallelize the processing of single files using chunks of the given
|
||||
size. The suffixes k, m, g, and t may be used for KB, MB, GB, and TB,
|
||||
respectively. The default split size is zero, which disables single
|
||||
file parallelization. A split size of less than 1 GB is not
|
||||
recommended. Lowering the split size will increase parallelism but
|
||||
decrease the performance of each file chunk and increase the overhead of
|
||||
transfer management. Raising the split size will have the opposite
|
||||
effect. The ideal split size for a given file is the size of the file
|
||||
divided by the number of concurrent clients available. Note that this
|
||||
option does not have an effect unless \fB\-\-hosts\fP is greater than
|
||||
one. Also note that this option can, in some cases, decrease remote
|
||||
transfer performance as it eliminates some higher performance
|
||||
transports.
|
||||
.IP "\fB\-\-split\-tar=SIZE\fP"
|
||||
Create tar files of around the given size when used with
|
||||
\fB\-\-create\-tar\fP. When multiple tar files are created for a
|
||||
destination tar file "file.tar", the resulting split tar files will be
|
||||
named "file.tar-i.tar" starting from "file.tar-1.tar". The suffixes k,
|
||||
m, g, and t may be used for KB, MB, GB, and TB, respectively. The
|
||||
default split tar size is 500 GB. A value of zero disables splitting.
|
||||
A split tar size of greater than 2 TB is not recommended. Note that
|
||||
resulting tar files may still be larger than specified when source files
|
||||
exist that are larger than the given size.
|
||||
.IP "\fB\-\-streams=NUM\fP"
|
||||
Use the given number of TCP streams in TCP-based transports (currently,
|
||||
bbcp, bbftp, fish-tcp, and gridftp). The default is the number of
|
||||
streams necessary to fully utilize the specified/estimated bandwidth
|
||||
using the maximum TCP window size. Note that it is usually preferable
|
||||
to specify \fB\-\-bandwidth\fP, which allows an appropriate number of
|
||||
streams to be set automatically. Increasing the number of streams can
|
||||
increase performance when the maximum window size is set too low or
|
||||
there is cross-traffic on the network, but too high a value can decrease
|
||||
performance due to increased congestion and packet loss.
|
||||
.IP "\fB\-\-stripe=SIZE|NUM\fP"
|
||||
By default, a file transferred to a Lustre file system will be striped
|
||||
according to size (one stripe per GB) unless the source resides on
|
||||
Lustre and has non-default striping, in which case striping will be
|
||||
preserved. Directory striping is preserved when applicable. If a
|
||||
positive number is specified, the stripe count of all destination files
|
||||
and directories will be set to the given value. If the given value is a
|
||||
size (specified with the suffixes k, m, g, and t for KB, MB, GB, and TB,
|
||||
respectively), files will be allocated one stripe per given size while
|
||||
directories will be striped according to the default policy. A value of
|
||||
zero disables automatic striping and uses the default policy for all
|
||||
files and directories.
|
||||
.IP "\fB\-\-threads=NUM\fP"
|
||||
Use the given number of threads in multi-threaded transports and
|
||||
checksum utilities (currently, mcp and msum). The default number of
|
||||
threads is 4. Increasing the number of threads can increase
|
||||
transfer/checksum performance when a host has excess resource capacity,
|
||||
but can reduce performance when any associated resource has reached
|
||||
its maximum.
|
||||
.IP "\fB\-\-verify\-fast\fP"
|
||||
Checksum files at the source and destination to verify that they have
|
||||
not been corrupted. If corruption is detected in a file, the corrupted
|
||||
portion will be automatically corrected using a partial transfer. This
|
||||
option differs from the default verification in that the source buffer
|
||||
will be reused when possible for the source checksum calculations. This
|
||||
potentially increases performance up to 33%, but is more subject to
|
||||
corruption as the source is only read once.
|
||||
.IP "\fB\-\-window=SIZE\fP"
|
||||
Use a TCP send/receive window of the given size in TCP-based transports
|
||||
(currently, bbcp, bbftp, fish-tcp, and gridftp). The suffixes k, m,
|
||||
g, and t may be used for KB, MB, GB, and TB, respectively. The default
|
||||
is the product of the specified/estimated bandwidth and the round-trip
|
||||
time between source and destination. Note that it is usually preferable
|
||||
to specify \fB\-\-bandwidth\fP, which allows an appropriate window size
|
||||
to be set automatically. Increasing the window size allows TCP to
|
||||
operate more efficiently over high bandwidth and/or high latency
|
||||
networks, but too high a value can overrun the receiver and cause packet
|
||||
loss.
|
||||
./"################################################################
|
||||
.SH "TRANSFER THROTTLING"
|
||||
./"################################################################
|
||||
.IP "\fB\-\-cpu=NUM\fP"
|
||||
Throttle the transfer when the local CPU usage reaches the specified
|
||||
percent of the total available. This option is disabled by default but
|
||||
may be desirable to prevent transfers from consuming too much of the
|
||||
local CPU. Once the given threshold is reached, a sleep period will
|
||||
be induced between each batch of files to achieve an average CPU
|
||||
utilization equal to the value specified. Note that this functionality
|
||||
is currently only supported on Unix-like systems.
|
||||
.IP "\fB\-\-disk=NUM1:NUM2\fP"
|
||||
Suspend/resume the transfer when the target file system disk usage
|
||||
reaches the specified percent of the total available. This option is
|
||||
disabled by default but may be desirable to prevent transfers from
|
||||
consuming too much local or remote disk space. Once the first
|
||||
threshold is reached, the transfer will suspend until enough disk
|
||||
resources have been freed on the target to bring the disk utilization
|
||||
under the second threshold. Note that this functionality is currently
|
||||
only supported on Unix-like systems.
|
||||
.IP "\fB\-\-io=NUM\fP"
|
||||
Throttle the transfer when the local I/O usage reaches the specified
|
||||
rate in MB/s. This option is disabled by default but may be desirable
|
||||
to prevent transfers from consuming too much of the local I/O bandwidth.
|
||||
Once the given threshold is reached, a sleep period will be induced
|
||||
between each batch of files to achieve an average I/O rate equal to
|
||||
the value specified.
|
||||
.IP "\fB\-\-ior=NUM\fP"
|
||||
Throttle the transfer when the local I/O reads reach the specified
|
||||
rate in MB/s. This option is similar to \fB\-\-io\fP but only applies
|
||||
to reads.
|
||||
.IP "\fB\-\-iow=NUM\fP"
|
||||
Throttle the transfer when the local I/O writes reach the specified
|
||||
rate in MB/s. This option is similar to \fB\-\-io\fP but only applies
|
||||
to writes.
|
||||
.IP "\fB\-\-net=NUM\fP"
|
||||
Throttle the transfer when the local network usage reaches the specified
|
||||
rate in MB/s. This option is disabled by default but may be desirable
|
||||
to prevent transfers from consuming too much of the local network
|
||||
bandwidth. Once the given threshold is reached, a sleep period will be
|
||||
induced between each batch of files to achieve an average network rate
|
||||
equal to the value specified.
|
||||
.IP "\fB\-\-netr=NUM\fP"
|
||||
Throttle the transfer when the local network reads reach the specified
|
||||
rate in MB/s. This option is similar to \fB\-\-net\fP but only applies
|
||||
to reads.
|
||||
.IP "\fB\-\-netw=NUM\fP"
|
||||
Throttle the transfer when the local network writes reach the specified
|
||||
rate in MB/s. This option is similar to \fB\-\-net\fP but only applies
|
||||
to writes.
|
||||
./"################################################################
|
||||
.SH "EXAMPLES"
|
||||
./"################################################################
|
||||
Copy local file "file1" in the current directory to existing local
|
||||
directory "/dir1":
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc file1 /dir1\fP
|
||||
|
||||
Shift id is 1
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Copy local file "file1" in the current directory to the user's home
|
||||
directory on host "host2" while preserving file attributes:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc -p file1 host2:\fP
|
||||
|
||||
Shift id is 2
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Recursively copy local directory "/dir1" to local directory "/dir2"
|
||||
and skip verifying that the contents have not been corrupted during the
|
||||
transfer:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc -r --no-verify /dir1 /dir2\fP
|
||||
|
||||
Shift id is 3
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Recursively copy remote directory "/dir2" on host "host2" to the current
|
||||
directory using a secure transport:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc -r --secure host2:/dir2 .\fP
|
||||
|
||||
Shift id is 4
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Recursively copy local directory "/bigdir1" to local directory
|
||||
"/bigdir2" using 4 client hosts to perform the transfer.
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc -r --hosts=4 /bigdir1 /bigdir2\fP
|
||||
|
||||
Shift id is 5
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Show the status of all transfers:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --status\fP
|
||||
|
||||
id | state | dirs | files | file size | date | length | rate
|
||||
| | sums | attrs | sum size | time | |
|
||||
---+-------+------+-------+---------------+-------+--------+---------
|
||||
1 | done | 0/0 | 1/1 | 92KB/92KB | 10/03 | 2s | 46KB/s
|
||||
| | 0/0 | 0/0 | 0.0B/0.0B | 17:06 | |
|
||||
2 | done | 0/0 | 1/1 | 92KB/92KB | 10/03 | 8s | 11.5KB/s
|
||||
| | 0/0 | 1/1 | 0.0B/0.0B | 17:06 | |
|
||||
3 | done | 1/1 | 2/2 | 99KB/99KB | 10/03 | 1s | 99KB/s
|
||||
| | 4/4 | 0/0 | 198KB/198KB | 17:07 | |
|
||||
4 | error | 1/1 | 1/2 | 92KB/99KB | 10/03 | 3s | 30.7KB/s
|
||||
| | 0/0 | 0/0 | 0.0B/0.0B | 17:08 | |
|
||||
5 | done | 1/1 | 64/64 | 65.5GB/65.5GB | 10/03 | 29s | 2.26GB/s
|
||||
| | 0/0 | 0/0 | 0.0B/0.0B | 17:09 | |
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Show the detailed status of all operations in transfer #2:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --status --id=2\fP
|
||||
|
||||
state | op | target | size | date | length | rate
|
||||
| tool | info | | time | |
|
||||
------+--------+-------------------------+------+-------+--------+-------
|
||||
done | cp | host2:/home/user1/file1 | 92KB | 10/03 | 5s | 18KB/s
|
||||
| bbftp | - | | 17:06 | |
|
||||
done | chattr | host2:/home/user1/file1 | - | 10/03 | 1s | -
|
||||
| sftp | - | | 17:06 | |
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Show the detailed status of all operations in transfer #4 that have an
|
||||
error state:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --status --id=4 --state=error\fP
|
||||
|
||||
state | op | target | size | date | length | rate
|
||||
| tool | info | | time | |
|
||||
------+-------+-------------------+------+------+--------+-----
|
||||
error | cp | /tmp/dir2/file2 | 7KB | - | - | -
|
||||
| rsync | rsync: send_files | | | |
|
||||
| | failed to open | | | |
|
||||
| | "/dir2/file2": | | | |
|
||||
| | Permission denied | | | |
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Show the detailed status of all operations in transfer #3 that involve a
|
||||
file name containing "file2":
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --status --id=3 --search=file2\fP
|
||||
|
||||
state | op | target | size | date | length | rate
|
||||
| tool | info | | time | |
|
||||
------+-------+-------------+------+-------+--------+------
|
||||
done | cp | /dir2/file2 | 7KB | 10/03 | 1s | 7KB/s
|
||||
| mcp | - | | 17:07 | |
|
||||
done | cksum | /dir2/file2 | 7KB | 10/03 | 1s | 7KB/s
|
||||
| msum | - | | 17:07 | |
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Show the history of all transfers:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --history\fP
|
||||
|
||||
id | origin | command
|
||||
---+---------------+--------------------------------------
|
||||
1 | host1.domain | shiftc file1 /dir1
|
||||
| [/home/user1] |
|
||||
2 | host1.domain | shiftc -p file1 host2:
|
||||
| [/home/user1] |
|
||||
3 | host1.domain | shiftc -r --no-verify /dir1 /dir2
|
||||
| [/home/user1] |
|
||||
4 | host1.domain | shiftc -r --secure host2:/dir2 .
|
||||
| [/tmp] |
|
||||
5 | host1.domain | shiftc -r --hosts=4 /bigdir1 /bigdir2
|
||||
| [/home/user1] |
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Show the history of all transfers that involve a host or a command
|
||||
containing "host2":
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --history --search=host2\fP
|
||||
|
||||
id | origin | command
|
||||
---+---------------+----------------------------------
|
||||
2 | host1.domain | shiftc -p file1 host2:
|
||||
| [/home/user1] |
|
||||
4 | host1.domain | shiftc -r --secure host2:/dir2 .
|
||||
| [/tmp] |
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Create a tar file "bigdir1.tar" in the current directory that consists
|
||||
of the contents of "/bigdir1" with a corresponding table of contents
|
||||
stored in "bigdir1.tar.toc" in the current directory:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --create-tar --index-tar /bigdir1 bigdir1.tar\fP
|
||||
|
||||
Shift id is 6
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Create tar files prefixed with "bd1.tar" in the remote directory
|
||||
"/dir2" on host "host2" that consist of the contents of "/bigdir1",
|
||||
split at 16 GB boundaries:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --create-tar --split-tar=16g /bigdir1 host2:/dir2/bd1.tar\fP
|
||||
|
||||
Shift id is 7
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Extract the split tar files prefixed with "bd1.tar" in the remote
|
||||
directory "/dir2" on host "host2" to the current directory:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --extract-tar host2:'/dir2/bd1.*tar' .\fP
|
||||
|
||||
Shift id is 8
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Sychronize the local directory "/dir1" with the remote directory
|
||||
"/dir2/dir1" on host "host2" while waiting for completion:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc -r --sync --wait /dir1 host2:/dir2\fP
|
||||
|
||||
Shift id is 9
|
||||
Detaching process (use --status option to monitor progress)
|
||||
Waiting for transfer to complete...
|
||||
|
||||
id | state | dirs | files | file size | date | length | rate
|
||||
| | sums | attrs | sum size | time | |
|
||||
---+-------+------+-------+---------------+-------+--------+-------
|
||||
9 | done | 1/1 | 2/2 | 99KB/99KB | 10/03 | 5s | 18KB/s
|
||||
| | 4/4 | 3/3 | 198KB/198KB | 17:14 | |
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Recursively copy local directory "/bigdir1" to local directory
|
||||
"/bigdir2" but exclude files ending in ".log".
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc -r --exclude='\\.log$' /bigdir1 /bigdir2\fP
|
||||
|
||||
Shift id is 10
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Extract the files "1g.20" through "1g.29" from "bigdir.tar" to the
|
||||
current directory:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
\fBshiftc --extract-tar --include='1g\\.2[0-9]' bigdir1.tar .\fP
|
||||
|
||||
Shift id is 11
|
||||
Detaching process (use --status option to monitor progress)
|
||||
.fi
|
||||
.RE
|
||||
./"################################################################
|
||||
.SH "NOTES"
|
||||
Transfers of files from DMF-managed file systems can take significantly
|
||||
longer than other transfers as files may need to be retrieved from
|
||||
tertiary storage before they can be copied.
|
||||
./"################################################################
|
||||
./"################################################################
|
||||
.SH "EXIT STATUS"
|
||||
./"################################################################
|
||||
shiftc exits with 0 on success or >0 if an error occurs.
|
||||
./"################################################################
|
||||
.SH "FILES"
|
||||
./"################################################################
|
||||
/var/spool/cron/tabs/$USER
|
||||
.RS
|
||||
An entry is added into the user's crontab on each client host on which
|
||||
a given transfer is being processed unless \fB\-\-no\-cron\fP is
|
||||
specified. This entry periodically invokes the client with specific
|
||||
arguments to check if the original client is still running. If so, the
|
||||
manager is notified that the transfer is still in progress. If not, the
|
||||
cron-invoked client will take over transfer processing.
|
||||
.RE
|
||||
./"################################################################
|
||||
.SH "AUTHOR"
|
||||
./"################################################################
|
||||
shiftc was written by Paul Kolano.
|
||||
./"################################################################
|
||||
.SH "SEE ALSO"
|
||||
./"################################################################
|
||||
bbcp(1), bbftp(1), cp(1), Date::Parse(3), globus-url-copy(1), mcp(1),
|
||||
msum(1), perlre(1), rsync(1), scp(1), sftp(1)
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
#!/usr/bin/perl -T
|
||||
|
||||
# This program is a template that can be used to periodically collect file
|
||||
# system information for Shift, which is used to determine file system
|
||||
# equivalence for client spawns and transfer load balancing.
|
||||
|
||||
use strict;
|
||||
use File::Temp;
|
||||
use POSIX qw(setuid);
|
||||
|
||||
our $VERSION = 0.06;
|
||||
|
||||
# untaint PATH
|
||||
$ENV{PATH} = "/bin:/usr/bin:/usr/local/bin";
|
||||
|
||||
############################
|
||||
#### begin config items ####
|
||||
############################
|
||||
|
||||
# user to use for ssh
|
||||
# (it is assumed this script will be invoked from root's crontab)
|
||||
my $user = "someuser";
|
||||
|
||||
# set of hosts to collect mount information from
|
||||
# (it is assumed hostbased authentication can be used to reach all hosts)
|
||||
# (it is assumed shift-aux is in the default path on all hosts)
|
||||
my @hosts = qw(
|
||||
host1 host2 ... hostN
|
||||
);
|
||||
|
||||
# host where manager invoked
|
||||
# (it is assumed hostbased authentication can be used to reach this host)
|
||||
# (it is assumed shift-mgr is in the default path on this host)
|
||||
my $mgr = "somehost";
|
||||
|
||||
##########################
|
||||
#### end config items ####
|
||||
##########################
|
||||
|
||||
# drop privileges and become defined user
|
||||
my $uid = getpwnam($user);
|
||||
setuid($uid) if (defined $uid);
|
||||
die "Unable to setuid to $user\n"
|
||||
if (!defined $uid || $< != $uid || $> != $uid);
|
||||
|
||||
# create temporary file (automatically unlinked on exit)
|
||||
my $tmp = File::Temp->new;
|
||||
my $file = $tmp->filename;
|
||||
close $tmp;
|
||||
|
||||
# gather info from all hosts
|
||||
foreach my $host (@hosts) {
|
||||
open(TMP, ">>$file");
|
||||
# use shift-aux to collect mount information and append to file
|
||||
open(FILE, "ssh -aqx -oHostBasedAuthentication=yes -oBatchMode=yes -l $user $host shift-aux mount |");
|
||||
while (<FILE>) {
|
||||
# print once for fully qualified host
|
||||
print TMP $_;
|
||||
# ignore shell line for plain host
|
||||
next if (!/^args=/ || /^args=shell/);
|
||||
# replace fully qualified host with plain host
|
||||
s/(host=$host)\.\S+/$1/;
|
||||
# duplicate line for plain host
|
||||
print TMP $_;
|
||||
}
|
||||
close FILE;
|
||||
close TMP;
|
||||
}
|
||||
|
||||
# call shift-mgr to add collected info to global database
|
||||
system("ssh -aqx -oHostBasedAuthentication=yes -oBatchMode=yes -l $user $mgr shift-mgr --mounts < $file");
|
||||
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
#!/usr/bin/perl -T
|
||||
|
||||
# This is a skeleton for a script that can be used in the shiftrc
|
||||
# select_hook setting, which chooses a remote host given the local
|
||||
# client host, the original remote host, and a Storable file mapping
|
||||
# candidate host names to the relevant file system information for
|
||||
# each. The file system information is in hash format, with contents
|
||||
# similar to the following:
|
||||
#
|
||||
# host => host23.example.com
|
||||
# local => /home1
|
||||
# opts => rw
|
||||
# remote => /mnt/home1
|
||||
# servers => nfsserver1.example.com
|
||||
# type => nfs
|
||||
|
||||
use strict;
|
||||
use Storable qw(retrieve);
|
||||
|
||||
# untaint path
|
||||
$ENV{PATH} = "/bin:/usr/bin:/usr/local/bin";
|
||||
|
||||
exit if (scalar(@ARGV) != 3 || ! -f $ARGV[2]);
|
||||
my $lhost = $ARGV[0];
|
||||
my $rhost = $ARGV[1];
|
||||
my %hosts = %{retrieve($ARGV[2])};
|
||||
|
||||
# do something to decide which host(s) of keys(%hosts) is best/least loaded,
|
||||
# which might involve calls to the local load balancing infrastruture,
|
||||
# and/or site-specific knowledge of which hosts have best connectivity
|
||||
# (either in general or to the given local host)
|
||||
|
||||
# print a set of comma-separated host names of the best choices or
|
||||
# print nothing to revert to default policy
|
||||
|
||||
|
|
@ -0,0 +1,357 @@
|
|||
#
|
||||
# Skeleton configuration file for Shift manager.
|
||||
#
|
||||
# This file should be located in /etc/shiftrc for multi-user installs
|
||||
# and in ~/.shiftrc for single-user installs.
|
||||
#
|
||||
# Items that are commented out show the default value.
|
||||
#
|
||||
# A value of nodefault indicates no default value for that item.
|
||||
#
|
||||
# Items that are not commented out indicate values that must be
|
||||
# explicitly configured. The values given for these items are
|
||||
# examples only.
|
||||
#
|
||||
|
||||
#########################
|
||||
#### manager options ####
|
||||
#########################
|
||||
|
||||
# directory where transfer metadata will be stored
|
||||
# (use %u as substitution for user name)
|
||||
# (parent dir must be world writable with sticky bit for multi-user installs)
|
||||
# (multi-user example: user_dir /var/lib/shift/%u)
|
||||
user_dir /home/%u/.shift
|
||||
|
||||
# time (seconds) to store transfer metadata after last activity
|
||||
#data_expire 604800
|
||||
|
||||
# location of file system information database
|
||||
# (must be world readable for multi-user installs)
|
||||
# (example: db_file /var/lib/shift/db)
|
||||
#db_file nodefault
|
||||
|
||||
# log debugging information for user X in user_dir/X.debug
|
||||
# (may be specified multiple times for different users)
|
||||
# (example: debug_alice 1)
|
||||
#debug_X 1
|
||||
|
||||
# domain to which user accounts belong for email notifications
|
||||
# (assumes user X can receive email at X@email_domain)
|
||||
# (assumes localhost:25 SMTP server running on manager host)
|
||||
# (example: email_domain example.com)
|
||||
#email_domain nodefault
|
||||
|
||||
# command to invoke to make host selection decisions
|
||||
# (must be world readable/executable for multi-user installs)
|
||||
# (example: select_hook /usr/local/bin/shift-select.hook)
|
||||
#select_hook nodefault
|
||||
|
||||
# host to which transfer metadata should be synchronized
|
||||
# (assumes hostbased authentication available to host)
|
||||
# (example: sync_host mgr2.example.com)
|
||||
#sync_host nodefault
|
||||
|
||||
|
||||
###########################
|
||||
#### transport options ####
|
||||
###########################
|
||||
|
||||
# supported local transports by decreasing performance/preference
|
||||
# (example: default_local mcp,rsync,shift)
|
||||
#default_local fish,shift
|
||||
|
||||
# supported remote transports by decreasing performance/preference
|
||||
# (example: default_remote fish-tcp,bbftp,rsync,fish,shift)
|
||||
#default_remote shift
|
||||
|
||||
# supported local transports by decreasing small file performance/preference
|
||||
# (example: local_small mcp,rsync,shift)
|
||||
#local_small fish,shift
|
||||
|
||||
# minimum size allowed for --split and --split-tar options
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
# (it is not recommended to set this below 1GB)
|
||||
#min_split 1g
|
||||
|
||||
# command-line options that will be used by bbcp on client hosts
|
||||
# (example: opts_bbcp -s 4 -w 4194304)
|
||||
#opts_bbcp nodefault
|
||||
|
||||
# behavior commands that will be used by bbftp on client hosts
|
||||
# (see the "behavior commands" section of bbftp man page for details)
|
||||
# (options must be separated by "\n" as shown in example below)
|
||||
# (example: opts_bbftp setnbstream 4\nsetrecvwinsize 4096\nsetsendwinsize 4096)
|
||||
#opts_bbftp nodefault
|
||||
|
||||
# command-line options that will be used by globus-url-copy on client hosts
|
||||
# (example: opts_gridftp -p 4 -tcp-bs 4194304)
|
||||
#opts_gridftp nodefault
|
||||
|
||||
# command-line options that will be used by mcp on client hosts
|
||||
# (if mcp >= 1.822.1, a --preallocate setting is recommended for DMF on CXFS)
|
||||
#opts_mcp --double-buffer
|
||||
|
||||
# command-line options that will be used by msum on client hosts
|
||||
#opts_msum --double-buffer
|
||||
|
||||
# command-line options that will be used by ssh on client hosts
|
||||
# (example: opts_ssh -c arcfour256 -m umac-64@openssh.com)
|
||||
#opts_ssh nodefault
|
||||
|
||||
# command-line options that will be used by ssh on client hosts with --secure
|
||||
# (example: opts_ssh_secure -c aes256-ctr -m hmac-sha2-512)
|
||||
#opts_ssh_secure nodefault
|
||||
|
||||
# supported remote transports by decreasing small file performance/preference
|
||||
# (example: remote_small fish-tcp,rsync,fish,shift,bbftp)
|
||||
#remote_small shift
|
||||
|
||||
# average size under which a lan batch will be optimized for small files
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#small_size_lan 256m
|
||||
|
||||
# average size under which a local batch will be optimized for small files
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#small_size_local 1g
|
||||
|
||||
# average size under which a wan batch will be optimized for small files
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#small_size_wan 64m
|
||||
|
||||
|
||||
################################
|
||||
#### general tuning options ####
|
||||
################################
|
||||
|
||||
# size of buffer to use in transports
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#default_buffer 4m
|
||||
|
||||
# maximum number of files to transfer in each batch
|
||||
# (use suffix {k,m,b/g,t} for 10E{3,6,9,12})
|
||||
#default_files 1k
|
||||
|
||||
# minimum number of files to process in each initialization batch
|
||||
# (note use of dash instead of underscore)
|
||||
# (use suffix {k,m,b/g,t} for 10E{3,6,9,12})
|
||||
#default_find-files 2k
|
||||
|
||||
# source sparsity percentage at which to preallocate destination file
|
||||
# (sparsity defined as 1 - (512 * allocated_blocks / size))
|
||||
# (note that transports using temporary files are not supported)
|
||||
# (example: default_preallocate 10)
|
||||
#default_preallocate 0
|
||||
|
||||
# number of times to retry failed operations
|
||||
# (must be at least 1 for --sync to function)
|
||||
#default_retry 2
|
||||
|
||||
# approximate maximum amount of data to transfer in each batch
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#default_size 4g
|
||||
|
||||
# approximate maximum size of created tar files
|
||||
# (note use of dash instead of underscore)
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#default_split-tar 500g
|
||||
|
||||
# amount of data per lustre stripe
|
||||
# (set to 0 to use default striping)
|
||||
# (note that transports using temporary files are not supported)
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#default_stripe 1g
|
||||
|
||||
# number of stripes that local lustre file systems use by default
|
||||
# (if file systems use different values, use max default across them)
|
||||
#lustre_default_stripe 1
|
||||
|
||||
# number of status entries at which some older done transfers may be omitted
|
||||
#status_lines 20
|
||||
|
||||
|
||||
################################
|
||||
#### network tuning options ####
|
||||
################################
|
||||
|
||||
# WAN bandwidth assumed for non-organization/non-xge hosts
|
||||
# (use suffix {k,m,g,t} for {Kb,Mb,Gb,Tb})
|
||||
#bandwidth_ind 100m
|
||||
|
||||
# WAN bandwidth assumed when end of host FQDN matches org_domains setting
|
||||
# (use suffix {k,m,g,t} for {Kb,Mb,Gb,Tb})
|
||||
#bandwidth_org 1g
|
||||
|
||||
# WAN bandwidth assumed when 10GE adapter found on host via lspci
|
||||
# (use suffix {k,m,g,t} for {Kb,Mb,Gb,Tb})
|
||||
#bandwidth_xge 10g
|
||||
|
||||
# minimum number of streams to use in remote LAN transfers
|
||||
#min_streams_lan 1
|
||||
|
||||
# minimum number of streams to use in remote WAN transfers
|
||||
#min_streams_wan 4
|
||||
|
||||
# minimum window size to use over LAN with TCP-based transports
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#min_window_lan 1m
|
||||
|
||||
# minimum window size to use over WAN with TCP-based transports
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#min_window_wan 4m
|
||||
|
||||
# maximum number of streams to use in remote LAN transfers
|
||||
#max_streams_lan 8
|
||||
|
||||
# maximum number of streams to use in remote WAN transfers
|
||||
#max_streams_wan 16
|
||||
|
||||
# latency (seconds) assumed to remote LAN hosts when measurement fails
|
||||
#latency_lan 0.001
|
||||
|
||||
# latency (seconds) assumed to remote WAN hosts when measurement fails
|
||||
#latency_wan 0.05
|
||||
|
||||
# regular expression for determining if host may have higher bandwidth
|
||||
#org_domains com|edu|gov|mil|net|org
|
||||
|
||||
|
||||
#################################
|
||||
#### parallelization options ####
|
||||
#################################
|
||||
|
||||
# number of clients to spawn on each host
|
||||
#default_clients 1
|
||||
|
||||
# maximum number of source hosts to involve in a transfer
|
||||
#default_hosts 1
|
||||
|
||||
# amount of data at which single files will be parallelized
|
||||
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
|
||||
#default_split 0
|
||||
|
||||
# number of threads to use in multi-threaded transports
|
||||
#default_threads 4
|
||||
|
||||
|
||||
###################################
|
||||
#### client throttling options ####
|
||||
###################################
|
||||
|
||||
# local avg. cpu usage (%) at which to throttle transfer
|
||||
# (example: default_cpu 75)
|
||||
#default_cpu nodefault
|
||||
|
||||
# target disk usage (%) at which to suspend/resume transfer
|
||||
# (example: default_disk 95:85)
|
||||
#default_disk nodefault:nodefault
|
||||
|
||||
# local avg. i/o usage (MB/s) at which to throttle transfer
|
||||
# (example: default_io 1000)
|
||||
#default_io nodefault
|
||||
|
||||
# local avg. i/o reads (MB/s) at which to throttle transfer
|
||||
# (example: default_ior 1000)
|
||||
#default_ior nodefault
|
||||
|
||||
# local avg. i/o writes (MB/s) at which to throttle transfer
|
||||
# (example: default_iow 1000)
|
||||
#default_iow nodefault
|
||||
|
||||
# local avg. network usage (MB/s) at which to throttle transfer
|
||||
# (example: default_net 80)
|
||||
#default_net nodefault
|
||||
|
||||
# local avg. network reads (MB/s) at which to throttle transfer
|
||||
# (example: default_netr 80)
|
||||
#default_netr nodefault
|
||||
|
||||
# local avg. network writes (MB/s) at which to throttle transfer
|
||||
# (example: default_netw 80)
|
||||
#default_netw nodefault
|
||||
|
||||
|
||||
#################################
|
||||
#### user throttling options ####
|
||||
#################################
|
||||
|
||||
# global avg. i/o usage (MB/s) at which to throttle user X's transfers
|
||||
# (example: throttle_io_user_alice 1000)
|
||||
#throttle_io_user_X nodefault
|
||||
|
||||
# global avg. i/o reads (MB/s) at which to throttle user X's transfers
|
||||
# (example: throttle_ior_user_alice 1000)
|
||||
#throttle_ior_user_X nodefault
|
||||
|
||||
# global avg. i/o writes (MB/s) at which to throttle user X's transfers
|
||||
# (example: throttle_iow_user_alice 1000)
|
||||
#throttle_iow_user_X nodefault
|
||||
|
||||
# global avg. network usage (MB/s) at which to throttle user X's transfers
|
||||
# (example: throttle_net_user_alice 80)
|
||||
#throttle_net_user_X nodefault
|
||||
|
||||
# global avg. network reads (MB/s) at which to throttle user X's transfers
|
||||
# (example: throttle_netr_user_alice 80)
|
||||
#throttle_netr_user_X nodefault
|
||||
|
||||
# global avg. network writes (MB/s) at which to throttle user X's transfers
|
||||
# (example: throttle_netw_user_alice 80)
|
||||
#throttle_netw_user_X nodefault
|
||||
|
||||
|
||||
########################################
|
||||
#### file system throttling options ####
|
||||
########################################
|
||||
|
||||
# disk usage (%) at which to suspend/resume transfers to file system X
|
||||
# (the format for each file system X is SERVERS:REMOTE, where SERVERS
|
||||
# and REMOTE are the values of "servers" and "remote", respectively,
|
||||
# listed for X when running "shift-aux mount" on a host that mounts X)
|
||||
# (example: throttle_disk_homenfs1.example.com:/home 95:85)
|
||||
#throttle_disk_X nodefault:nodefault
|
||||
|
||||
# global avg. i/o usage (MB/s) at which to throttle transfers to file system X
|
||||
# (format of X is identical to disk usage above)
|
||||
# (example: throttle_io_fs_homenfs1.example.com:/home 1000)
|
||||
#throttle_io_fs_X nodefault
|
||||
|
||||
# global avg. i/o reads (MB/s) at which to throttle transfers to file system X
|
||||
# (format of X is identical to disk usage above)
|
||||
# (example: throttle_ior_fs_homenfs1.example.com:/home 1000)
|
||||
#throttle_ior_fs_X nodefault
|
||||
|
||||
# global avg. i/o writes (MB/s) at which to throttle transfers to file system X
|
||||
# (format of X is identical to disk usage above)
|
||||
# (example: throttle_iow_fs_homenfs1.example.com:/home 1000)
|
||||
#throttle_iow_fs_X nodefault
|
||||
|
||||
|
||||
#################################
|
||||
#### host throttling options ####
|
||||
#################################
|
||||
|
||||
# global avg. i/o usage (MB/s) at which to throttle transfers on host X
|
||||
# (example: throttle_io_host_host1.example.com 10000)
|
||||
#throttle_io_host_X nodefault
|
||||
|
||||
# global avg. i/o reads (MB/s) at which to throttle transfers on host X
|
||||
# (example: throttle_ior_host_host1.example.com 10000)
|
||||
#throttle_ior_host_X nodefault
|
||||
|
||||
# global avg. i/o writes (MB/s) at which to throttle transfers on host X
|
||||
# (example: throttle_iow_host_host1.example.com 10000)
|
||||
#throttle_iow_host_X nodefault
|
||||
|
||||
# global avg. network usage (MB/s) at which to throttle transfers on host X
|
||||
# (example: throttle_net_host_host1.example.com 10000)
|
||||
#throttle_net_host_X nodefault
|
||||
|
||||
# global avg. network reads (MB/s) at which to throttle transfers on host X
|
||||
# (example: throttle_netr_host_host1.example.com 10000)
|
||||
#throttle_netr_host_X nodefault
|
||||
|
||||
# global avg. network writes (MB/s) at which to throttle transfers on host X
|
||||
# (example: throttle_netw_host_host1.example.com 10000)
|
||||
#throttle_netw_host_X nodefault
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Loading…
Reference in New Issue