NASA
/
shift
mirror of https://github.com/pkolano/shift.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Shift 6.0

master
Paul Kolano 2019-01-16 17:21:12 -08:00
parent 5a43b0af40
commit 21924e068b
12 changed files with 3411 additions and 1809 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
CHANGES
View File

@ -194,3 +194,58 @@ CHANGES
- Fixed lfs setstripe calls to use -S instead of -s (lustre >= 2.3.0)
- Fixed descent into directories matching --exclude during initialization
- Fixed hash errors when remote source mapped to local file system
* Shift 6.0 (01/16/19)
- Note that this version is not backward compatible with previous versions
- Added depth-first file stage processing using --pipeline
- Added CSV history output using --history=csv
- Added disablement of email status for states given in --no-mail
- Added disablement of preservation for attributes given in --no-preserve
- Added continuous real-time display of status using --monitor
- Added error handling for manager disk exhaustion
- Added overwrite of read-only files using --force
- Added ability to ignore unrecoverable errors using --restart=ignore
- Added faster get/set of external file attributes using shift-bin
- Added plotting by client using --plot=client
- Added alert state for items that must be externally checked/fixed
- Added encrypted data streams to fish-tcp transport using --secure
- Added better support for scp-style "user@" syntax
- Added silent corruption detection for previously completed transfers
- Added --last-sum option to query silent corruption database
- Added fadvise of source after transfer when shift-bin available
- Added per file dynamic striping via perl-based expressions
- Added manager config for hash algorithm and hash leaf size
- Added advanced --older/--newer expressions using atime/ctime/mtime
- Added --doing manager option to see past/present doing logs
- Added abort during tar creation if first split file already exists
- Added same file detection across remote hosts
- Changed --no-check option to --no-sanity
- Changed manager --meta output to handle deeper data structures
- Changed manager db_file config setting to mount_db
- Changed dmget handling to use -a during tar creation to keep files online
- Changed manager meta file format from base64 to yEnc
- Changed manager log structured files to seekable compressed format
- Changed manager doing logs to log structured format
- Changed HMAC authentication from SHA1 to SHA512 in fish-tcp
- Changed manager sync mechanism to better support multiple failovers
- Changed estimated completion to use running avgs instead of elapsed time
- Fixed mount detection for DMF file systems reported as dmapi
- Fixed mount detection for lustre file systems with multiple MDS's
- Fixed overwrite of symlinks when exist at destination
- Fixed deprecated File::Glob usage on newer perls
- Fixed Net::Ping invocations during latency measurements on redhat/centos
- Fixed detection of automounted file systems
- Fixed hang on PBS-controlled hosts during file system detection
- Fixed --sync on DMF to copy file instead of summing when time/size match
- Fixed DMF recalls during file overwrite when using built-in transports
- Fixed incorrect striping when --stripe=0 given and transport is mcp
- Fixed warning output in some --status=color scenarios
- Fixed exception in detailed status when very large file with small files
- Fixed race condition between check and creation of user metadata dir
- Fixed tracking of corruption stats during --sync
- Fixed stalls after host/process failures due to bad metadata recovery
- Fixed throttling deadlock due to file not counting own preallocation
- Fixed error detection in some gridftp scenarios
- Removed manager lustre_default_stripe config setting
- Removed manager min_split config setting
- Removed previously allowed inverse options --preserve and --verify

113
INSTALL
View File

@ -3,7 +3,7 @@ Shift Installation and Configuration
1. Deployments
Shift consists of three executable components:
Shift consists of four executable components:
o shiftc - the Shift client, which is invoked by users and other
client instances and must exist on hosts that initiate
@ -25,6 +25,12 @@ Shift Installation and Configuration
preservation, remote lustre striping, and remote
verification (if msum from mutil is not installed)
o shift-bin - the Shift binary helper, which is an optional component
invoked by shiftc and shift-aux to perform file
operations that cannot be implemented directly in pure
perl and/or cannot be executed efficiently in
batches by standard command line utilities.
The main consideration for a Shift deployment is deciding which hosts will
be the manager hosts. In the simplest case, all client hosts are manager
hosts. If there is more than one client host, then the manager must be
@ -52,6 +58,8 @@ Shift Installation and Configuration
2.2. Optional
o IO::Socket::SSL - allows fish-tcp encryption with --secure
(https://metacpan.org/pod/IO::Socket::SSL)
o mcp/msum >= 1.76.7 - high speed local copy/sum
(http://mutil.sf.net)
o bbcp - high speed remote copy
@ -86,58 +94,85 @@ Shift Installation and Configuration
o unbuffer - interleave stdout/stderr when using gridftp
3. Installation
3. Build (optional - linux only)
Note that the following shows the exact files needed on each type of host.
Since the number of files is small, however, there is minimal penalty to
simply installing them all on every host.
An optional binary component called "shift-bin" can be built on
linux systems to significantly improve the efficiency of some file
operations. These operations include equivalents to those provided
by posix_fadvise, fallocate, {get,set}facl, {get,set}fattr, and lfs.
This utility must be built differently depending on whether the
system mount lustre (and has access to the corresponding liblustreapi
library) or not.
3.1. Single-user installation
3.1. Systems mounting lustre file systems
cd c
make lustre
3.2. Systems not mounting lustre file systems
cd c
make nolustre
4. Installation
Note that the following shows the exact files needed on each type
of host. Since the number of files is small, however, there is
minimal penalty to simply installing them all on every host. The
installation commands assume that the optional shift-bin component
has been built as needed for client and remote hosts.
4.1. Single-user installation
Note that the user's home directory is used as the default install
prefix in all examples, but can be changed to any other desired location
as long as the corresponding bin directory is in the user's path.
3.1.1. Client hosts
4.1.1. Client hosts
install -m 700 perl/shiftc ~/bin/shiftc
install -m 600 doc/shiftc.1 ~/man/man1/shiftc.1
install -m 700 perl/shiftc ~/bin/
install -m 700 perl/shift-bin ~/bin/
install -m 600 doc/shiftc.1 ~/man/man1/
3.1.2. Manager hosts
4.1.2. Manager hosts
install -m 700 perl/shift-mgr ~/bin/shift-mgr
install -m 700 perl/shift-mgr ~/bin/
install -m 600 etc/shiftrc ~/.shiftrc
3.1.3. Remote hosts (optional but recommended when possible)
4.1.3. Remote hosts (optional but recommended when possible)
install -m 700 perl/shift-aux ~/bin/shift-aux
install -m 700 perl/shift-aux ~/bin/
install -m 700 perl/shift-bin ~/bin/
3.2. Multi-user installation
4.2. Multi-user installation
Note that /usr/local is used as the default install prefix in all
examples, but can be changed to any other desired location as long
as the corresponding bin directory is in the default system path.
3.2.1. Client hosts
4.2.1. Client hosts
install -m 755 perl/shiftc /usr/local/bin/shiftc
install -m 644 doc/shiftc.1 /usr/local/man/man1/shiftc.1
install -m 755 perl/shiftc /usr/local/bin/
install -m 755 perl/shift-bin /usr/local/bin/
install -m 644 doc/shiftc.1 /usr/local/man/man1/
3.2.2. Manager hosts
4.2.2. Manager hosts
install -m 755 perl/shift-mgr /usr/local/bin/shift-mgr
install -m 644 etc/shiftrc /etc/shiftrc
install -m 755 perl/shift-mgr /usr/local/bin/
install -m 644 etc/shiftrc /etc/
3.2.3. Remote hosts (optional but recommended when possible)
4.2.3. Remote hosts (optional but recommended when possible)
install -m 755 perl/shift-aux /usr/local/bin/shift-aux
install -m 755 perl/shift-aux /usr/local/bin/
install -m 755 perl/shift-bin /usr/local/bin/
4. Configuration
5. Configuration
4.1. Client hosts
5.1. Client hosts
4.1.1. ~/.ssh/id_rsa (or similar)
5.1.1. ~/.ssh/id_rsa (or similar)
If hostbased authentication is not supported by client hosts,
manager hosts, and/or remote hosts, pubkey authentication must be
@ -151,7 +186,7 @@ Shift Installation and Configuration
but comes with a drop in reliability as any failure of the agent or
agent host leaves any associated transfers with no way to recover.
4.1.2. ~/.ssh/authorized_keys
5.1.2. ~/.ssh/authorized_keys
If hostbased authentication is not supported to other client hosts
for parallelization, pubkey authentication must be used. In this
@ -160,7 +195,7 @@ Shift Installation and Configuration
added to the invoking user's authorized_keys file on other client
hosts.
4.1.3. ~/bin/shiftc (single-user) or /usr/local/bin/shiftc (multi-user)
5.1.3. ~/bin/shiftc (single-user) or /usr/local/bin/shiftc (multi-user)
If the manager hosts differ from the client hosts, the manager
host(s) can be hardcoded within the shiftc program in the
@ -179,9 +214,9 @@ Shift Installation and Configuration
which manager host out of a set of hosts will be used for each
invocation).
4.2. Manager hosts
5.2. Manager hosts
4.2.1. ~/.shiftrc (single-user) or /etc/shiftrc (multi-user)
5.2.1. ~/.shiftrc (single-user) or /etc/shiftrc (multi-user)
All items in the default config file should be reviewed.
The only required setting is:
@ -195,11 +230,7 @@ Shift Installation and Configuration
sync_host
must be configured to sync the transfer metadata across two
manager hosts. Note that the existing synchronization
mechanism has been found to be a bottleneck when there are
many clients in a single transfer or many simultaneous
transfers by a single user. This will be fixed in a future
version.
manager hosts.
The transport options should definitely be reviewed to
enable any higher performance transports that may be
@ -214,7 +245,7 @@ Shift Installation and Configuration
functionality specific to certain remote file systems (e.g. Lustre
striping), it is desirable to configure:
db_file
mount_db
with a database of host and file system information from within
the local environment. A template for producing this database is
@ -236,7 +267,7 @@ Shift Installation and Configuration
infrastructure. If not configured, hosts will be chosen randomly
after a successful sshd ping test.
4.2.2. ~/.ssh/authorized_keys
5.2.2. ~/.ssh/authorized_keys
If hostbased authentication is not supported to manager hosts,
pubkey authentication must be used. In this case, the public key(s)
@ -247,9 +278,9 @@ Shift Installation and Configuration
--mgr-user.
4.3. Remote hosts
5.3. Remote hosts
4.3.1. ~/.ssh/authorized_keys
5.3.1. ~/.ssh/authorized_keys
If hostbased authentication is not supported to remote hosts, pubkey
authentication must be used. In this case, the public key(s)
@ -259,9 +290,9 @@ Shift Installation and Configuration
will either be the invoking user or the one specified by --user.
5. Usage
6. Usage
5.1. shiftc
6.1. shiftc
Client usage is detailed in the man page "doc/shiftc.1", which can be
viewed with:
@ -275,7 +306,7 @@ Shift Installation and Configuration
"user@" portion will be dropped) so --user must be specified instead
when the remote user differs from the local user.
5.2. shift-mgr
6.2. shift-mgr
Normal users need not invoke shift-mgr directly as all relevant manager
functionality is accessed indirectly through the client. Additional

4
README.md
View File

@ -48,6 +48,8 @@ Shift includes the following features, among others:
- integrity verification of transfers with partial retransfer/resum to
rectify corruption
- detection of silent corruption between transfers of the same file
- throttling based on local and remote resource utilization
- automatic retrieval/release of files residing on DMF-managed file systems
@ -65,7 +67,7 @@ Shift includes the following features, among others:
Shift is in active production at the NASA Advanced Supercomputing Facility
(https://www.nas.nasa.gov/hecc/support/kb/entry/300) and has facilitated
approximately 600k transfers over 1.32B files totalling 88.5 PB (as of May
approximately 780k transfers over 1.7B files totalling 105 PB (as of Dec.
2018) since deployment in March 2012.
For full details of the Shift architecture, see

7
c/Makefile 100644
View File

@ -0,0 +1,7 @@
# use "make lustre" if you have lustre, "make nolustre" if not
lustre:
gcc -o shift-bin -g shift-bin.c -lattr -lacl -llustreapi
nolustre:
gcc -o shift-bin -g shift-bin.c -lattr -lacl -D_NO_LUSTRE

317
c/shift-bin.c 100644
View File

@ -0,0 +1,317 @@
//
// Copyright (C) 2012-2019 United States Government as represented by the
// Administrator of the National Aeronautics and Space Administration
// (NASA). All Rights Reserved.
//
// This software is distributed under the NASA Open Source Agreement
// (NOSA), version 1.3. The NOSA has been approved by the Open Source
// Initiative. See http://www.opensource.org/licenses/nasa1.3.php
// for the complete NOSA document.
//
// THE SUBJECT SOFTWARE IS PROVIDED "AS IS" WITHOUT ANY WARRANTY OF ANY
// KIND, EITHER EXPRESSED, IMPLIED, OR STATUTORY, INCLUDING, BUT NOT
// LIMITED TO, ANY WARRANTY THAT THE SUBJECT SOFTWARE WILL CONFORM TO
// SPECIFICATIONS, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR
// A PARTICULAR PURPOSE, OR FREEDOM FROM INFRINGEMENT, ANY WARRANTY THAT
// THE SUBJECT SOFTWARE WILL BE ERROR FREE, OR ANY WARRANTY THAT
// DOCUMENTATION, IF PROVIDED, WILL CONFORM TO THE SUBJECT SOFTWARE. THIS
// AGREEMENT DOES NOT, IN ANY MANNER, CONSTITUTE AN ENDORSEMENT BY
// GOVERNMENT AGENCY OR ANY PRIOR RECIPIENT OF ANY RESULTS, RESULTING
// DESIGNS, HARDWARE, SOFTWARE PRODUCTS OR ANY OTHER APPLICATIONS RESULTING
// FROM USE OF THE SUBJECT SOFTWARE. FURTHER, GOVERNMENT AGENCY DISCLAIMS
// ALL WARRANTIES AND LIABILITIES REGARDING THIRD-PARTY SOFTWARE, IF
// PRESENT IN THE ORIGINAL SOFTWARE, AND DISTRIBUTES IT "AS IS".
//
// RECIPIENT AGREES TO WAIVE ANY AND ALL CLAIMS AGAINST THE UNITED STATES
// GOVERNMENT, ITS CONTRACTORS AND SUBCONTRACTORS, AS WELL AS ANY PRIOR
// RECIPIENT. IF RECIPIENT'S USE OF THE SUBJECT SOFTWARE RESULTS IN ANY
// LIABILITIES, DEMANDS, DAMAGES, EXPENSES OR LOSSES ARISING FROM SUCH USE,
// INCLUDING ANY DAMAGES FROM PRODUCTS BASED ON, OR RESULTING FROM,
// RECIPIENT'S USE OF THE SUBJECT SOFTWARE, RECIPIENT SHALL INDEMNIFY AND
// HOLD HARMLESS THE UNITED STATES GOVERNMENT, ITS CONTRACTORS AND
// SUBCONTRACTORS, AS WELL AS ANY PRIOR RECIPIENT, TO THE EXTENT PERMITTED
// BY LAW. RECIPIENT'S SOLE REMEDY FOR ANY SUCH MATTER SHALL BE THE
// IMMEDIATE, UNILATERAL TERMINATION OF THIS AGREEMENT.
//
#define _GNU_SOURCE
#include <fcntl.h>
#include <grp.h>
#include <pwd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#ifndef _NO_LUSTRE
# include <lustre/lustreapi.h>
#endif
#include <sys/acl.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/xattr.h>
////////////////
//// escape ////
////////////////
// adapted from http://www.geekhideout.com/urlcode.shtml
char *escape(char *str) {
if (str == NULL) return str;
static char hex[] = "0123456789ABCDEF";
char *pstr = str;
char *buf = malloc(strlen(str) * 3 + 1);
char *pbuf = buf;
while (*pstr) {
if (isalnum(*pstr) || *pstr == '-' || *pstr == '_' ||
*pstr == '.' || *pstr == '~') {
*pbuf++ = *pstr;
} else {
*pbuf++ = '%';
*pbuf++ = hex[(*pstr >> 4) & 15];
*pbuf++ = hex[*pstr & 15];
}
pstr++;
}
*pbuf = '\0';
return buf;
}
//////////////////
//// unescape ////
//////////////////
// adapted from http://www.geekhideout.com/urlcode.shtml
char *unescape(char *str) {
if (str == NULL) return str;
char *pstr = str;
char *buf = malloc(strlen(str) + 1);
char *pbuf = buf;
while (*pstr) {
if (*pstr == '%') {
if (pstr[1] && pstr[2]) {
*pbuf = (isdigit(pstr[1]) ? pstr[1] - '0' :
tolower(pstr[1]) - 'a' + 10) << 4;
*pbuf++ |= isdigit(pstr[2]) ? pstr[2] - '0' :
tolower(pstr[2]) - 'a' + 10;
pstr += 2;
}
} else {
*pbuf++ = *pstr;
}
pstr++;
}
*pbuf = '\0';
return buf;
}
//////////////////
//// do_chown ////
//////////////////
int do_chown(char *file, char *user, char *group) {
struct passwd *pwd = getpwnam(user);
if (pwd == NULL) return -1;
uid_t uid = pwd->pw_uid;
struct group *grp = getgrnam(group);
if (grp == NULL) return -1;
gid_t gid = grp->gr_gid;
return lchown(file, uid, gid);
}
////////////////////
//// do_fadvise ////
////////////////////
int do_fadvise(char *file, off_t off, off_t len) {
int fd = open(file, O_RDONLY);
if (fd == -1) return fd;
return posix_fadvise(fd, off, len, POSIX_FADV_DONTNEED);
}
//////////////////////
//// do_fallocate ////
//////////////////////
int do_fallocate(char *file, off_t len) {
int fd = open(file, O_CREAT | O_WRONLY);
if (fd == -1) return fd;
int rc = fallocate(fd, FALLOC_FL_KEEP_SIZE, 0, len);
close(fd);
return rc;
}
////////////////////
//// do_getfacl ////
////////////////////
int do_getfacl(char *file) {
acl_t acl = acl_get_file(file, ACL_TYPE_ACCESS);
if (acl == NULL || acl_equiv_mode(acl, NULL) == 0) return -1;
char *s = acl_to_text(acl, NULL);
if (s == NULL) return -1;
char *es = escape(s);
printf(" %s", es);
free(es);
acl_free(s);
return 0;
}
////////////////////
//// do_setfacl ////
////////////////////
int do_setfacl(char *file, char *s_acl) {
acl_t acl = acl_from_text(s_acl);
if (acl == NULL) return -1;
int rc = acl_set_file(file, ACL_TYPE_ACCESS, acl);
acl_free(acl);
return rc;
}
/////////////////////
//// do_getfattr ////
/////////////////////
int do_getfattr(char *file) {
ssize_t buflen, keylen, vallen;
char *buf, *key, *val;
buflen = listxattr(file, NULL, 0);
if (buflen <= 0) return -1;
buf = malloc(buflen);
if (buf == NULL) return -1;
buflen = listxattr(file, buf, buflen);
if (buflen == -1) return -1;
int count = 0;
key = buf;
while (buflen > 0) {
if (!strncmp(key, "user.", 5)) {
vallen = getxattr(file, key, NULL, 0);
if (vallen > 0) {
val = malloc(vallen + 1);
if (val != NULL) {
vallen = getxattr(file, key, val, vallen);
if (vallen != -1) {
val[vallen] = 0;
char *eval = escape(val);
printf("%s", count == 0 ? " " : ",");
printf("%s\%3D%s", key, eval);
free(eval);
count++;
}
}
free(val);
}
}
keylen = strlen(key) + 1;
buflen -= keylen;
key += keylen;
}
free(buf);
return count ? 0 : -1;
}
/////////////////////
//// do_setfattr ////
/////////////////////
int do_setfattr(char *file, char *s_xattr) {
int rc = 0;
char *keyval = strtok(s_xattr, ",");
while (keyval != NULL) {
char *key = unescape(keyval);
char *val = index(key, '=');
if (val == NULL) continue;
*val++ = '\0';
rc |= lsetxattr(file, key, val, strlen(val), 0);
free(key);
keyval = strtok(NULL, ",");
}
return rc;
}
//////////////////////
//// do_getstripe ////
//////////////////////
int do_getstripe(char *file) {
#ifndef _NO_LUSTRE
int v1 = sizeof(struct lov_user_md_v1) +
LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data_v1);
int v3 = sizeof(struct lov_user_md_v3) +
LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data_v1);
struct lov_user_md *lum = malloc(v1 > v3 ? v1 : v3);
if (lum == NULL) return -1;
int rc = llapi_file_get_stripe(file, lum);
if (!rc) printf(" %d,%d", lum->lmm_stripe_count, lum->lmm_stripe_size);
free(lum);
return rc;
#else
return -1;
#endif
}
//////////////////////
//// do_setstripe ////
//////////////////////
int do_setstripe(char *file, int scount, unsigned long long ssize, char *pool) {
#ifndef _NO_LUSTRE
return llapi_file_create_pool(file, ssize, -1, scount, 0, pool);
#else
return -1;
#endif
}
//////////////////
//// do_utime ////
//////////////////
int do_utime(char *file, long atime, long mtime) {
struct timeval tv[2];
tv[0].tv_sec = atime;
tv[0].tv_usec = 0;
tv[1].tv_sec = mtime;
tv[1].tv_usec = 0;
return lutimes(file, tv);
}
//////////////
//// main ////
//////////////
int main(int argc, char *argv[]) {
char *buf = NULL;
size_t buf_max = 0;
while (getline(&buf, &buf_max, stdin) > 0) {
buf[strcspn(buf, "\n")] = '\0';
char *cmd = strtok(buf, " ");
if (cmd == NULL) continue;
char *efile = strtok(NULL, " ");
char *file = unescape(efile);
if (file == NULL) continue;
char *a1, *a2, *a3;
a1 = strtok(NULL, " ");
a2 = strtok(NULL, " ");
a3 = strtok(NULL, " ");
printf("%s %s", cmd, efile);
int rc = -1;
if (!strcmp(cmd, "chown") && a1 != NULL && a2 != NULL) {
rc = do_chown(file, a1, a2);
} else if (!strcmp(cmd, "fadvise") && a1 != NULL && a2 != NULL) {
rc = do_fadvise(file, atoll(a1), atoll(a2));
} else if (!strcmp(cmd, "fallocate") && a1 != NULL) {
rc = do_fallocate(file, atoll(a1));
} else if (!strcmp(cmd, "getfacl")) {
rc = do_getfacl(file);
} else if (!strcmp(cmd, "setfacl") && a1 != NULL) {
char *ua1 = unescape(a1);
rc = do_setfacl(file, ua1);
free(ua1);
} else if (!strcmp(cmd, "getstripe")) {
rc = do_getstripe(file);
} else if (!strcmp(cmd, "setstripe") && a1 != NULL && a2 != NULL) {
rc = do_setstripe(file, atol(a1), atoll(a2), a3);
} else if (!strcmp(cmd, "getfattr")) {
rc = do_getfattr(file);
} else if (!strcmp(cmd, "setfattr")) {
char *ua1 = unescape(a1);
rc = do_setfattr(file, ua1);
free(ua1);
} else if (!strcmp(cmd, "utime") && a1 != NULL && a2 != NULL) {
rc = do_utime(file, atol(a1), atol(a2));
}
printf(" %d\n", rc);
fflush(stdout);
free(file);
}
free(buf);
}

338
doc/shiftc.1
View File

@ -1,4 +1,4 @@
.TH "shiftc" "1" "21 Dec 2017" "" ""
.TH "shiftc" "1" "10 May 2018" "" ""
./"################################################################
.SH "NAME"
./"################################################################
@ -26,8 +26,8 @@ two remote hosts are not supported.
./"################################################################
.PP
shiftc is the client for Shift, which is a framework for
\fBS\fPelf-\fBH\fPealing \fBI\fPndependent \fBF\fPile \fBT\fPransfers. Shift
includes the following features, among others:
\fBS\fPelf-\fBH\fPealing \fBI\fPndependent \fBF\fPile \fBT\fPransfers.
Shift includes the following features, among others:
.IP -
support for local, LAN, and WAN transfers
.IP -
@ -46,6 +46,8 @@ rsync-like synchronization based on modification times and checksums
integrity verification of transfers with partial retransfer/resum to
rectify corruption
.IP -
detection of silent corruption between transfers of the same file
.IP -
throttling based on local and remote resource utilization
.IP -
automatic retrieval and release of files residing on DMF-managed file
@ -82,6 +84,7 @@ given in following sections.
\-d, \-\-directory create any missing parent directories
\-\-exclude=REGEX exclude files matching REGEX
\-\-extract\-tar extract tar file(s) at SOURCE to DEST
\-f, \-\-force overwrite existing read-only files at DEST
\-h, \-\-help help
\-\-host\-file=FILE parallelize transfer on hosts in FILE (one per line)
\-\-host\-list=LIST parallelize transfer on hosts in LIST
@ -90,10 +93,13 @@ given in following sections.
\-I, \-\-ignore\-times do not skip files that match size and time
\-\-include=REGEX include only files matching REGEX
\-\-index\-tar create table of contents during tar creation
\-\-newer=DATE include only files with mtime newer than DATE
\-\-newer=[TYPE:]DATE include only files with mtime [TYPE] newer than DATE
(TYPE in form [acmACM]+(|[acmACM]+)*)
\-P, \-\-no\-dereference never follow symbolic links
\-T, \-\-no\-target\-directory treat target as a normal file
\-\-older=DATE include only files with mtime older than DATE
\-\-older=[TYPE:]DATE include only files with mtime [TYPE] older than DATE
(TYPE in form [acmACM]+(|[acmACM]+)*)
\-\-pipeline emit verified files sooner for parallel processing
\-\-ports=NUM1:NUM2 use ports NUM1\-NUM2 for remote TCP\-based transports
\-R, \-r, \-\-recursive copy directories recursively
\-\-secure encrypt data stream(s) and use secure ciphers/macs
@ -103,34 +109,39 @@ given in following sections.
(exit 0 = success, 1 = failure)
.PP
\fBFeature\-disablement options:\fP
\-\-no\-check do not check file existence/size (benchmarking only)
\-\-no\-cron do not recover from host/process failures via cron
\-\-no\-mail do not send status emails
\-\-no\-mail[=LIST] do not send status emails [for LIST of states]
(LIST subset of {alert,done,error,run,stop,
throttle,warn})
\-\-no\-offline do not migrate DMF\-managed files after transfer
\-\-no\-preserve do not preserve times, mode, owner, acls, or xattrs
\-\-no\-preserve[=LIST] do not preserve attributes [on specific LIST]
(LIST subset of {acl,mode,owner,stripe,time,xattr})
\-\-no\-recall do not recall DMF\-managed files before transfer
\-\-no\-sanity do not check file existence/size (benchmarking only)
\-\-no\-verify do not verify/rectify integrity of destination files
.PP
\fBMonitoring/management options:\fP
\-\-history show list of transfer commands and origin host/dir
\fBMonitoring and management options:\fP
\-\-history[=csv] show command line/origin of transfers [in CSV form]
\-\-id=NUM use transfer identifier NUM for other commands
\-\-last-sum show last stored sum for SOURCE(s)
\-\-mgr=HOST set host of shift manager to HOST
\-\-mgr\-identity=FILE access manager host with ssh identity in FILE
\-\-mgr\-user=USER access manager host as USER
\-\-monitor[=FORMAT] monitor progress of running transfers
(FORMAT one of {color,csv,pad})
\-\-plot[=[BY:]LIST] plot detailed performance when piped to gnuplot
(BY one of {host,id,user})
(BY one of {client,host,id,user})
(LIST subset of {chattr,cksum,cp,find,io,ln,meta,
mkdir,sum})
\-\-restart restart transfer with given \-\-id
\-\-restart[=ignore] restart transfer with given \-\-id [ignoring errors]
\-\-search=REGEX show only status/history matching REGEX
\-\-state=STATE show status of only those operations in STATE
(STATE one of {done,error,none,queue,run,warn})
\-\-stats show stats across all transfers
\-\-stats[=csv] show stats across all transfers [in CSV form]
\-\-status[=FORMAT] show brief status of all transfers
or detailed status of transfer with given \-\-id
(FORMAT one of {color,csv,pad})
\-\-stop stop transfer with given \-\-id
or detailed status of transfer with given \-\-id
.PP
\fBTuning options (defaults in brackets):\fP
\-\-bandwidth=BITS tune TCP\-based transports based on BITS per second
@ -154,9 +165,10 @@ given in following sections.
\-\-split\-tar=SIZE create tar files of around SIZE bytes
(use suffix {k,m,g,t} for {KB,MB,GB,TB}) [500g]
\-\-streams=NUM use NUM streams in remote transports [4]
\-\-stripe=[SIZE|NUM] use 1 stripe per SIZE bytes or NUM stripes
[:[SIZE][:POOL]] optionally use stripe size SIZE and/or pool POOL
(use suffix {k,m,g,t} for {KB,MB,GB,TB}) [1g]
\-\-stripe=[CEXP] choose stripe {count,size,pool} via expr {C,S,P}EXP
[::[SEXP][::PEXP]] (EXP may be NUM, SIZE, or full perl expression w/
const {NM,SZ,SC,SS} for src {name,size,scnt,ssz})
(use suffix {k,m,g,t} for {KiB,MiB,GiB,TiB})
\-\-threads=NUM use NUM threads in local transports [4]
\-\-verify\-fast verify faster but less safely by reusing src buffer
\-\-window=SIZE use SIZE bytes for window in TCP\-based transports
@ -212,6 +224,13 @@ existing directory or non-existing directory name. This option implies
format are supported, but GNU extensions for large uids, gids, file
sizes, and file names are handled appropriately. Also note that this
option cannot be used with \fB\-\-sync\fP.
.IP "\fB\-f, \-\-force\fP"
Overwrite existing read-only files at the destination by temporarily
adding owner write permission. File permissions will be restored
later in the transfer. Note, however, that if the transfer does not
complete successfully, files may be left with the wrong permissions.
Also note that files marked as immutable using "chattr +i" cannot
be overwritten even when this option is in effect.
.IP "\fB\-\-host\-file=FILE\fP"
Parallelize the transfer by using additional clients on the hosts
specified in the given file (one host name per line). This option
@ -273,11 +292,20 @@ multiple table of contents and checksum files may be created. For each
split tar file "file.tar-i.tar", the table of contents will be named
"file.tar-i.tar.toc" and the checksum file will be named
"file.tar-i.tar.sum".
.IP "\fB\-\-newer=DATE\fP"
Only transfer source files whose modification time is newer (inclusive)
than the given date. Any date string supported by the Perl
Date::Parse module can be specified. Note that this option can be
combined with \fB\-\-older\fP to specify exact date ranges.
.IP "\fB\-\-newer=[TYPE:]DATE\fP"
Only transfer source files whose modification time (or combination of
modification, access, and/or creation times) is newer (inclusive) than
the given date. Any date string supported by the Perl Date::Parse
module (see Date::Parse(3) for details) can be specified. An optional
type expression of the form "[acmACM]+(|[acmACM]+)*)", where "a" is
access time, "c" is creation time, "m" is modification time, and "A",
"C", and "M", are their inverses, respectively, can be given to specify
conditions in which one or more conditions are or are not newer than the
date. For example, "aM|cm" would transfer source files whose access
time was newer than the date but whose modification time was not newer,
or files whose creation time and modification time were newer. Note
that this option can be combined with \fB\-\-older\fP to specify exact
date ranges.
.IP "\fB\-P, \-\-no\-dereference\fP"
Never follow symbolic links to file or directories. Note that this
can result in broken links at the destination as files and directories
@ -289,11 +317,41 @@ Do not treat the destination specially when it is a directory or a
symbolic link to a directory. This option can be used with recursive
transfers to copy a directory's contents into an existing directory
instead of into a new subdirectory beneath it as is done by default.
.IP "\fB\-\-older=DATE\fP"
Only transfer source files whose modification time is older than the
given date. Any date string supported by the Perl Date::Parse module
can be specified. Note that this option can be combined with
\fB\-\-newer\fP to specify exact date ranges.
.IP "\fB\-\-older=[TYPE:]DATE\fP"
Only transfer source files whose modification time (or combination of
modification, access, and/or creation times) is older than the given
date. Any date string supported by the Perl Date::Parse module (see
Date::Parse(3) for details) can be specified. An optional type
expression of the form "[acmACM]+(|[acmACM]+)*)", where "a" is access
time, "c" is creation time, "m" is modification time, and "A", "C", and
"M", are their inverses, respectively, can be given to specify
conditions in which one or more conditions are or are not older than the
date. For example, "aM|cm" would transfer source files whose access
time was older than the date but whose modification time was not older,
or files whose creation time and modification time were both newer.
Note that this option can be combined with \fB\-\-newer\fP to specify
exact date ranges.
.IP "\fB\-\-pipeline\fP"
Produce verified files earlier in the transfer by preferring to process
the normal sequence of operations (find, copy, checksum, verify
ckecksum, change attributes) in reverse order. In default non-pipeline
operation, these stages are performed in order where all files are found
before any are copied before any are checksummed, etc. When this option
is enabled, files that have reached the change attribute stage will be
processed before files that have reached the verify checksum stage,
which will be processed before files that have reached the checksum
stage, etc. This allows users to perform parallel processing on
verified files while the transfer is still ongoing. To determine the
list of files that have been successfully verified in a transfer with id
"N", use \fB\-\-status \-\-id=N \-\-state=done \-\-search=chattr\fP.
When multiple clients are participating in the transfer (i.e.
\fB\-\-clients\fP or \fB\-\-hosts\fP greater than one), different
clients will prefer different stages for more overlap of reads and
writes between the source and destination file systems. Note that while
several strategies are employed to ensure that checksums are computed
from disk and not from cache, it is safest to only use this option when
there is actually a need to process destination files during the
transfer.
.IP "\fB\-\-ports=NUM1:NUM2\fP"
Use ports from the range NUM1-NUM2 for the data streams of TCP-based
transports (currently, bbcp, bbftp, fish-tcp, and gridftp). All
@ -307,7 +365,7 @@ to directories given on the command line will be followed during
recursive transfers (identical to the default behavior of cp).
.IP "\fB\-\-secure\fP"
Encrypt data during remote transfers and use secure ciphers and MACs
with SSH-based tranports. Note that this option will, in most cases,
with SSH-based transports. Note that this option will, in most cases,
decrease performance as it eliminates some higher performance transports
and increases CPU utilization during SSH connections.
.IP "\fB\-\-sync\fP"
@ -336,34 +394,39 @@ Block until the transfer completes and print a summary of the transfer.
This option implies \fB\-\-no\-mail\fP. An exit value of 0 indicates
that the transfer has successfully completed while an exit value of 1
indicates that the transfer has failed or that the waiting process was
terminated prematurely.
terminated prematurely. This option may be used together with
\fB\-\-monitor\fP to show the real-time status of the transfer while
waiting.
./"################################################################
.SH "FEATURE DISABLEMENT
./"################################################################
.IP "\fB\-\-no\-check\fP"
Disable file existence and size checks at the end of the transfer.
This option was included for benchmarking and completeness purposes
and is not recommended for general use.
.IP "\fB\-\-no\-cron\fP"
Do not attempt to recover from host/process failures via cron. Note
that when such a failure occurs, the transfer will become stuck in the
"run" state until stopped.
.IP "\fB\-\-no\-mail\fP"
Prevents sending of emails due to errors, warnings, or completion.
This option may be desirable when performing a large number of scripted
transfers. Note that equivalent transfer status and history information
can always be manually retrieved using \fB\-\-status\fP and
\fB\-\-history\fP, respectively.
.IP "\fB\-\-no\-mail[=LIST]\fP"
By default, emails are sent when a transfer completes successfully,
aborts with errors, or is stopped, and for the first instances of
alerts, errors, throttling, and/or warnings while running. This option
prevents emails from being sent altogether or, optionally, for a specific
subset of states. The given list may be a comma-separated subset of
{alert, done, error, run, stop, throttle, warn}. This option may be
desirable when performing a large number of scripted transfers. Note
that equivalent transfer status and history information can always be
manually retrieved using \fB\-\-status\fP and \fB\-\-history\fP,
respectively.
.IP "\fB\-\-no\-offline\fP"
By default, files transferred to/from DMF-managed file systems will be
migrated to offline media as soon as the transfer completes. This
option specifies that files should not be migrated. Note that DMF may
still choose to migrate (and possibly release) files even when this
option is enabled.
.IP "\fB\-\-no\-preserve\fP"
By default, times, permissions, ownership, ACLs, and extended attributes
of transferred files and directories are preserved when possible.
This option specifies that these items should not be preserved. Note
.IP "\fB\-\-no\-preserve[=LIST]\fP"
By default, times, permissions, ownership, striping, ACLs, and extended
attributes of transferred files and directories are preserved when
possible. This option specifies that these items (or an optional
specified subset) should not be preserved. The given list may be a
comma-separated subset of {acl, mode, owner, stripe, time, xattr}. Note
that permissions may be left in various states depending on the invoking
user's umask and the transport utilized. In particular, read access at
the destination may be more permissive than read access at the source.
@ -373,6 +436,10 @@ recalled from offline media as soon as the transfer begins and again
before each batch of files is processed. This option specifies that
files should not be recalled. Note that DMF will still recall files
as needed even when this option is enabled.
.IP "\fB\-\-no\-sanity\fP"
Disable file existence and size checks at the end of the transfer.
This option was included for benchmarking and completeness purposes
and is not recommended for general use.
.IP "\fB\-\-no\-verify\fP"
By default, files are checksummed at the source and destination to
verify that they have not been corrupted and if corruption is detected,
@ -387,12 +454,26 @@ option may be used to disable verification.
Once one or more transfers have been initialized, the user may view
transfer history, stop/restart transfers, and/or check transfer status
with the following options.
.IP "\fB\-\-history\fP"
.IP "\fB\-\-history[=csv]\fP"
Show a brief history of all transfers including the transfer identifier,
the origin host/directory and the original command.
the origin host/directory and the original command. When
\fB\-\-history=csv\fP is specified, history is shown in CSV format.
.IP "\fB\-\-id=NUM\fP"
Specify the transfer identifier to be used with management and status
commands.
.IP "\fB\-\-last\-sum\fP"
The checksums of all files transferred with Shift are stored in a
per-user db. When a file with a known checksum is transferred and has
not been modified since the checksum was stored, the transfer will be
put into the "alert" state if the current checksum does not match the
stored checksum. This option queries the silent corruption database for
all files given on the command line and prints (one file per line) the
last known checksum, the file modification time associated with this
checksum, and the file name. When \fB\-\-index\-tar\fP is given, the
first file argument is assumed to be a tar file and the remaining
arguments names of files within the tar for which checksum information
will be printed. A checksum of "-" means that no information is stored
for the file.
.IP "\fB\-\-mgr=HOST\fP"
Set the host that will be used to manage transfers. By default, this
host will be accessed as the current user with hostbased authentication
@ -414,37 +495,60 @@ specified, manager communication will be performed as the given user
so that user must be authorized to run processes locally. In
particular, care should be taken on PBS-controlled nodes, where the
given user should either own the node or be on the user exception list.
.IP "\fB\-\-monitor[=FORMAT]\fP"
Show the real-time status of all running transfers including the
transfer identifier, the current state, the number of directories
completed, the number of files transferred, the number of files
checksummed, the number of attributes preserved, the amount of data
transferred, the amount of data checksummed, the time the transfer
started, the duration of the transfer, the estimated time remaining in
the transfer, and the rate of the transfer. Note that updates are
real-time with respect to the information available to the manager and
not with respect to the transports that may be carrying out the
transfer. Status will be returned in CSV format when
\fB\-\-monitor=csv\fP is specified. Duration and estimated time will be
zero-padded when \fB\-\-monitor=pad\fP is specified. When
\fB\-\-monitor=color\fP is specified, transfers in the {error, run,
throttle, warn} states will be shown with {red, green, magenta, yellow}
coloring, respectively. When \fB\-\-id\fP is specified, only the given
transfer will be shown. When all transfers (or the one specified)
have completed, the command will exit. This option may be used with
\fB\-\-wait\fP to monitor progress while waiting.
.IP "\fB\-\-plot=[=[BY:]LIST]\fP"
Produce output suitable for piping into gnuplot (version 5 or above) that shows
detailed performance over time across all transfers. The \fB\-\-id\fP and
\fB\-\-state\fP options may be used to plot only a single transfer or transfers
in a particular state, respectively. The default plot will show the aggregate
performance of each I/O operation (i.e. cp, sum, and cksum) and the aggregate
performance of each metadata operation (i.e. find, mkdir, ln, and chattr). I/O
operations are plotted against the left y-axis while metadata operations are
plotted against the right y-axis. The list of plotted items may be changed by
giving a comma-separated list consisting of one of more of {chattr, cksum, cp,
find, io, ln, meta, mkdir, sum}. Note that "io" is a shorthand for
"cp,sum,cksum" and "meta" is a shorthand for "find,mkdir,ln,chattr". The list
of items may be grouped by any of {host, id, user} by prefixing one of these
terms to the list. For example, "--plot=id:cp" would show a curve for the copy
performance of each tranfer id. When a grouping is given without a specific
list of metrics (e.g. "--plot=id"), "io" is assumed.
.IP "\fB\-\-restart\fP"
Produce output suitable for piping into gnuplot (version 5 or above)
that shows detailed performance over time across all transfers. The
\fB\-\-id\fP and \fB\-\-state\fP options may be used to plot only a
single transfer or transfers in a particular state, respectively. The
default plot will show the aggregate performance of each I/O operation
(i.e. cp, sum, and cksum) and the aggregate performance of each metadata
operation (i.e. find, mkdir, ln, and chattr). I/O operations are
plotted against the left y-axis while metadata operations are plotted
against the right y-axis. The list of plotted items may be changed by
giving a comma-separated list consisting of one of more of {chattr,
cksum, cp, find, io, ln, meta, mkdir, sum}. Note that "io" is a
shorthand for "cp,sum,cksum" and "meta" is a shorthand for
"find,mkdir,ln,chattr". The list of items may be grouped by any of
{host, id, user} by prefixing one of these terms to the list. For
example, \fB\-\-plot=id:cp\fP would show a curve for the copy
performance of each tranfer id. When a grouping is given without a
specific list of metrics (e.g. \fB\-\-plot=id\fP), "io" is assumed.
.IP "\fB\-\-restart[=ignore]\fP"
Restart the transfer associated with the given \fB\-\-id\fP that was
stopped due to unrecoverable errors or stopped explicitly via
\fB\-\-stop\fP. Note that transfers must be restarted on the original
client host or one that has equivalent file system access. A subset of
the available command-line options may be respecified during a restart
including \fB\-\-bandwidth\fP, \fB\-\-buffer\fP, \fB\-\-clients\fP,
\fB\-\-cpu\fP, \fB\-\-disk\fP, \fB\-\-files\fP, \fB\-\-host\-file\fP,
\fB\-\-host\-list\fP, \fB\-\-hosts\fP, \fB\-\-io\fP, \fB\-\-ior\fP,
\fB\-\-iow\fP, \fB\-\-local\fP, \fB\-\-net\fP, \fB\-\-netr\fP,
\fB\-\-netw\fP, \fB\-\-no\-cron\fP, \fB\-\-no\-mail\fP,
\fB\-\-no\-offline\fP, \fB\-\-no\-recall\fP, \fB\-\-ports\fP,
\fB\-\-preallocate\fP, \fB\-\-remote\fP, \fB\-\-retry\fP,
\fB\-\-secure\fP, \fB\-\-size\fP, \fB\-\-streams\fP, \fB\-\-stripe\fP,
\fB\-\-threads\fP, and \fB\-\-window\fP.
\fB\-\-stop\fP. If \fB\-\-restart=ignore\fP is specified, all existing
errors will be ignored and the transfer will progress as if the associated
files and directories were no longer part of the transfer. Note that
transfers must be restarted on the original client host or one that has
equivalent file system access. A subset of the available command-line
options may be respecified during a restart including \fB\-\-bandwidth\fP,
\fB\-\-buffer\fP, \fB\-\-clients\fP, \fB\-\-cpu\fP, \fB\-\-disk\fP,
\fB\-\-files\fP, \fB\-\-force\fP, \fB\-\-host\-file\fP, \fB\-\-host\-list\fP,
\fB\-\-hosts\fP, \fB\-\-io\fP, \fB\-\-ior\fP, \fB\-\-iow\fP, \fB\-\-local\fP,
\fB\-\-net\fP, \fB\-\-netr\fP, \fB\-\-netw\fP, \fB\-\-no\-cron\fP,
\fB\-\-no\-mail\fP, \fB\-\-no\-offline\fP, \fB\-\-no\-recall\fP,
\fB\-\-pipeline\fP, \fB\-\-ports\fP, \fB\-\-preallocate\fP, \fB\-\-remote\fP,
\fB\-\-retry\fP, \fB\-\-secure\fP, \fB\-\-size\fP, \fB\-\-streams\fP,
\fB\-\-stripe\fP, \fB\-\-threads\fP, and \fB\-\-window\fP.
.IP "\fB\-\-search=REGEX\fP"
When \fB\-\-status\fP and \fB\-\-id\fP are specified, this option will
show the full status of file operations in the associated transfer whose
@ -463,9 +567,11 @@ have the given state. When \fB\-\-id\fP is not specified, this option
will show the brief status of transfers in the given state. Valid
states are done, error, none, queue, run, and warn. A state of "none"
will show a summary of the given transfer.
.IP "\fB\-\-stats\fP"
.IP "\fB\-\-stats[=csv]\fP"
Show stats across all transfers including transfer counts, rates, tool
usage, initialization options, error counts, and error messages.
usage, initialization options, error counts, and error messages. When
\fB\-\-stats=csv\fP is specified, stats are shown in CSV format
without error messages.
.IP "\fB\-\-status[=FORMAT]\fP"
Show a brief status of all transfers including the transfer identifier,
the current state, the number of directories completed, the number of
@ -579,9 +685,9 @@ given size can occur if the batch count specified by \fB\-\-files\fP
is reached first.
.IP "\fB\-\-split=SIZE\fP"
Parallelize the processing of single files using chunks of the given
size. The suffixes k, m, g, and t may be used for KB, MB, GB, and TB,
respectively. The default split size is zero, which disables single
file parallelization. A split size of less than 1 GB is not
size. The suffixes k, m, g, and t may be used for KiB, MiB, GiB, and
TiB, respectively. The default split size is zero, which disables
single file parallelization. A split size of less than 1 GiB is not
recommended. Lowering the split size will increase parallelism but
decrease the performance of each file chunk and increase the overhead of
transfer management. Raising the split size will have the opposite
@ -611,25 +717,45 @@ streams to be set automatically. Increasing the number of streams can
increase performance when the maximum window size is set too low or
there is cross-traffic on the network, but too high a value can decrease
performance due to increased congestion and packet loss.
.IP "\fB\-\-stripe=[SIZE|NUM][:[SIZE][:POOL]]\fP"
.IP "\fB\-\-stripe=[CEXP][::[SEXP][::PEXP]]\fP"
By default, a file transferred to a Lustre file system will be striped
according to size (one stripe per GB) unless the source resides on
Lustre and has non-default striping, in which case striping will be
preserved. Directory striping is preserved when applicable. If a
positive number is specified, the stripe count of all destination files
and directories will be set to the given value. If the given value is a
size (specified with the suffixes k, m, g, and t for KB, MB, GB, and TB,
respectively), files will be allocated one stripe per given size while
directories will be striped according to the default policy. A value of
zero disables automatic striping and uses the default policy for all
files and directories. Striping behavior may be further refined by
specifying a stripe size (using the size suffixes above) and/or Lustre
pool name. The stripe count and/or stripe size can be left empty before
the colon when specifying the stripe size or pool, respectively. For
example, "--stripe=:4m" would specify the stripe size to be 4 MB while
using the default stripe count policy and, similarly, "--stripe=::pool1"
would use the pool "pool1" while using the default stripe count and
stripe size.
according to an administrator-defined policy (one stripe per GiB when
not configured). It is recommended, although not required, that this
policy preserve existing striping when the source resides on Lustre and
has non-default striping. To disregard existing striping, "stripe" may
be used with \fB\-\-no\-preserve\fP=stripe. To disable automatic
striping completely and use the default lustre behavior for all files
and directories, use \fB\-\-stripe=0\fP.
.IP
The user may override the default policy by specifying expressions for
one or more of the stripe count (CEXP), stripe size (SEXP), and stripe
pool (PEXP). For the stripe count, a positive number less than 65,536
indicates a fixed number of stripes to use for all destination files and
directories. A greater number or size defined with the suffixes k, m,
g, and t for KiB, MiB, GiB, and TiB, respectively, specifies that files
will be allocated one stripe per given size while directories will be
striped according to the default policy. Finally, an arbitrary Perl
expression (see perlsyn(1) for details) involving the constants NM,
SZ, SC, and SS for source name, size, stripe count, and stripe size,
respectively, may be specified to dynamically define the stripe count
differently for every file and directory in the transfer. For example,
the expression "NM =~ /foo/ ? 4 : (SZ < 10g ? 2g : 10g)" would set the
stripe count of files whose name contains "foo" to 4, and the stripe
count of files whose name does not contain "foo" to either one stripe
per 2 GiB when the file size is less than 10 GiB or one stripe per 10
GiB otherwise.
.IP
Striping behavior may be further refined by specifying a stripe size
expression and/or Lustre pool name expression with similar conventions.
The stripe count and/or stripe size can be left empty before the colons
when specifying the stripe size or pool, respectively. For example,
\fB\-\-stripe=::4m\fP would specify the stripe size to be 4 MiB while
using the default stripe count policy and, similarly,
\fB\-\-stripe=::::pool1\fP would use the pool "pool1" while using the
default stripe count and stripe size. Note that if the stripe pool is a
perl expression and not a simple alphanumeric pool name, pool names must
use perl conventions for indicating strings such as quotes and/or
quote-like operators (e.g. "NM =~ /foo/ ? q(poolfoo) : q(poolbar)").
.IP "\fB\-\-threads=NUM\fP"
Use the given number of threads in multi-threaded transports and
checksum utilities (currently, mcp and msum). The default number of
@ -638,13 +764,13 @@ transfer/checksum performance when a host has excess resource capacity,
but can reduce performance when any associated resource has reached
its maximum.
.IP "\fB\-\-verify\-fast\fP"
Checksum files at the source and destination to verify that they have
not been corrupted. If corruption is detected in a file, the corrupted
portion will be automatically corrected using a partial transfer. This
option differs from the default verification in that the source buffer
will be reused when possible for the source checksum calculations. This
potentially increases performance up to 33%, but is more subject to
corruption as the source is only read once.
By default, files are checksummed at the source and destination to
verify that they have not been corrupted with the source being read once
during the copy and again during the checksum. The options specifies
that the source copy buffer should be reused when possible for the
source checksum calculations. This potentially increases performance up
to 33%, but does not allow bits corrupted during the initial read to be
detected.
.IP "\fB\-\-window=SIZE\fP"
Use a TCP send/receive window of the given size in TCP-based transports
(currently, bbcp, bbftp, fish-tcp, and gridftp). The suffixes k, m,
@ -918,7 +1044,7 @@ Detaching process (use --status option to monitor progress)
.fi
.RE
.PP
Sychronize the local directory "/dir1" with the remote directory
Synchronize the local directory "/dir1" with the remote directory
"/dir2/dir1" on host "host2" while waiting for completion:
.PP
.RS
@ -990,4 +1116,4 @@ shiftc was written by Paul Kolano.
.SH "SEE ALSO"
./"################################################################
bbcp(1), bbftp(1), cp(1), Date::Parse(3), globus-url-copy(1), mcp(1),
msum(1), perlre(1), rsync(1), scp(1), sftp(1)
msum(1), perlre(1), perlsyn(1), rsync(1), scp(1), sftp(1)

10
etc/shift-mounts.pl
View File

@ -8,7 +8,7 @@ use strict;
use File::Temp;
use POSIX qw(setuid);
our $VERSION = 0.06;
our $VERSION = 0.09;
# untaint PATH
$ENV{PATH} = "/bin:/usr/bin:/usr/local/bin";
@ -46,26 +46,24 @@ die "Unable to setuid to $user\n"
# create temporary file (automatically unlinked on exit)
my $tmp = File::Temp->new;
my $file = $tmp->filename;
close $tmp;
# gather info from all hosts
foreach my $host (@hosts) {
open(TMP, ">>$file");
# use shift-aux to collect mount information and append to file
open(FILE, "ssh -aqx -oHostBasedAuthentication=yes -oBatchMode=yes -l $user $host shift-aux mount |");
while (<FILE>) {
# print once for fully qualified host
print TMP $_;
print $tmp $_;
# ignore shell line for plain host
next if (!/^args=/ || /^args=shell/);
# replace fully qualified host with plain host
s/(host=$host)\.\S+/$1/;
# duplicate line for plain host
print TMP $_;
print $tmp $_;
}
close FILE;
close TMP;
}
close $tmp;
# call shift-mgr to add collected info to global database
system("ssh -aqx -oHostBasedAuthentication=yes -oBatchMode=yes -l $user $mgr shift-mgr --mounts < $file");

21
etc/shift-select.hook
View File

@ -2,28 +2,20 @@
# This is a skeleton for a script that can be used in the shiftrc
# select_hook setting, which chooses a remote host given the local
# client host, the original remote host, and a Storable file mapping
# candidate host names to the relevant file system information for
# each. The file system information is in hash format, with contents
# similar to the following:
#
# host => host23.example.com
# local => /home1
# opts => rw
# remote => /mnt/home1
# servers => nfsserver1.example.com
# type => nfs
# client host, the original remote host, and a comma-separated list
# of candidate host names.
use strict;
use Storable qw(retrieve);
# untaint path
$ENV{PATH} = "/bin:/usr/bin:/usr/local/bin";
# untaint insecure environment variables
delete $ENV{$_} foreach (qw(BASH_ENV CDPATH ENV IFS));
exit if (scalar(@ARGV) != 3 || ! -f $ARGV[2]);
exit if (scalar(@ARGV) != 3);
my $lhost = $ARGV[0];
my $rhost = $ARGV[1];
my %hosts = %{retrieve($ARGV[2])};
my %hosts = map {$_ => 1} split(/,/, $ARGV[2]);
# do something to decide which host(s) of keys(%hosts) is best/least loaded,
# which might involve calls to the local load balancing infrastruture,
@ -32,4 +24,3 @@ my %hosts = %{retrieve($ARGV[2])};
# print a set of comma-separated host names of the best choices or
# print nothing to revert to default policy

78
etc/shiftrc
View File

@ -21,6 +21,8 @@
# (use %u as substitution for user name)
# (parent dir must be world writable with sticky bit for multi-user installs)
# (multi-user example: user_dir /var/lib/shift/%u)
# (single-user example: user_dir /home/%u/.shift)
#user_dir nodefault
user_dir /home/%u/.shift
# time (seconds) to store transfer metadata after last activity
@ -28,8 +30,8 @@ user_dir /home/%u/.shift
# location of file system information database
# (must be world readable for multi-user installs)
# (example: db_file /var/lib/shift/db)
#db_file nodefault
# (example: mount_db /var/lib/shift/mounts)
#mount_db nodefault
# log debugging information for user X in user_dir/X.debug
# (may be specified multiple times for different users)
@ -73,11 +75,6 @@ user_dir /home/%u/.shift
# (example: mcp,shift,fish,fish-tcp,rsync,bbftp)
#local_small shift,fish,fish-tcp
# minimum size allowed for --split and --split-tar options
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
# (it is not recommended to set this below 1GB)
#min_split 1g
# command-line options that will be used by bbcp on client hosts
# (example: opts_bbcp -s 4 -w 4194304)
#opts_bbcp nodefault
@ -123,6 +120,19 @@ user_dir /home/%u/.shift
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
#small_size_wan 64m
# size of chunks at which corruption is detected/rectified
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
# (it is not recommended to set this below 1GB)
# (changing this value voids all existing values in checksum dbs)
#sum_split 1g
# type of checksum to use for integrity verification
# (format is TYPE_BITS, BITS required when TYPE supports multiple lengths)
# (for msum, see "msum --help" --hash-type list, "_" will be ignored)
# (for built-in perl sums, will use Digest::TYPE->new(BITS))
# (example: sum_type sha3_256)
#sum_type md5
################################
#### general tuning options ####
@ -160,16 +170,15 @@ user_dir /home/%u/.shift
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
#default_split-tar 500g
# amount of data per lustre stripe
# (set to 0 to use default striping)
# expressions by which to select lustre stripe count/size/pool
# (format is same as --stripe: [CEXP][::[SEXP][::PEXP]])
# (EXP may be NUM, SIZE, or full perl expression)
# (EXP may use const {NM,SZ,SC,SS} for src {name,size,scnt,ssz})
# (set to 0 to use system default striping)
# (note that transports using temporary files are not supported)
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
#default_stripe 1g
# number of stripes that local lustre file systems use by default
# (if file systems use different values, use max default across them)
#lustre_default_stripe 1
# number of status entries at which some older done transfers may be omitted
#status_lines 20
@ -190,6 +199,18 @@ user_dir /home/%u/.shift
# (use suffix {k,m,g,t} for {Kb,Mb,Gb,Tb})
#bandwidth_xge 10g
# latency (seconds) assumed to remote LAN hosts when measurement fails
#latency_lan 0.001
# latency (seconds) assumed to remote WAN hosts when measurement fails
#latency_wan 0.05
# maximum number of streams to use in remote LAN transfers
#max_streams_lan 8
# maximum number of streams to use in remote WAN transfers
#max_streams_wan 16
# minimum number of streams to use in remote LAN transfers
#min_streams_lan 1
@ -204,22 +225,29 @@ user_dir /home/%u/.shift
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
#min_window_wan 4m
# maximum number of streams to use in remote LAN transfers
#max_streams_lan 8
# maximum number of streams to use in remote WAN transfers
#max_streams_wan 16
# latency (seconds) assumed to remote LAN hosts when measurement fails
#latency_lan 0.001
# latency (seconds) assumed to remote WAN hosts when measurement fails
#latency_wan 0.05
# regular expression for determining if host may have higher bandwidth
#org_domains com|edu|gov|mil|net|org
#######################
#### cache options ####
#######################
# estimated amount of i/o needed on clients to evict newly cached data
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
#cache_size_client nodefault
# estimated amount of i/o needed on file servers to evict newly cached data
# (use suffix {k,m,g,t} for {KB,MB,GB,TB})
#cache_size_server nodefault
# estimated number of seconds needed on clients to evict newly cached data
#cache_time_client nodefault
# estimated number of seconds needed on file servers to evict newly cached data
#cache_time_server nodefault
#################################
#### parallelization options ####
#################################

364
perl/shift-aux
View File

@ -1,6 +1,6 @@
#!/usr/bin/perl -T
#
# Copyright (C) 2012-2016 United States Government as represented by the
# Copyright (C) 2012-2019 United States Government as represented by the
# Administrator of the National Aeronautics and Space Administration
# (NASA). All Rights Reserved.
#
@ -47,7 +47,6 @@
require 5.007_003;
use strict;
use Cwd qw(abs_path);
use Digest::MD5 qw(md5);
use Fcntl qw(:DEFAULT :mode);
use File::Basename;
use File::Path;
@ -59,43 +58,49 @@ use IO::Handle;
use IO::Socket::INET;
# use embedded IPC::Open3 since versions prior to perl 5.14.0 are buggy
require IPC::Open3;
use List::Util qw(min);
use List::Util qw(first min);
use MIME::Base64;
use POSIX;
use Socket;
use Symbol qw(gensym);
use Sys::Hostname;
use Text::ParseWords;
our $VERSION = 0.95;
# need threads and version of Thread::Queue from perl >= 5.10.1
my $have_threads = eval 'require 5.010_001; use threads; use Thread::Queue; 1';
our $VERSION = 0.96;
# do not die when receiving sigpipe
$SIG{PIPE} = 'IGNORE';
# untaint path
$ENV{PATH} = "/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin";
# untaint env
delete $ENV{ENV};
my %opts;
my $cmd = shift @ARGV;
# untaint insecure environment variables
delete $ENV{$_} foreach (qw(BASH_ENV CDPATH ENV IFS));
# make sure user can read, write, execute/traverse files/dirs
# make sure root transfers do not inadvertently expose files
umask ($< == 0 ? 077 : 077 & umask);
my %perl;
$perl{ssl} = eval 'use IO::Socket::SSL; 1';
# need threads and version of Thread::Queue from perl >= 5.10.1
$perl{threads} = eval 'require 5.010_001; use threads; use Thread::Queue; 1';
my %opts;
my $cmd = shift @ARGV;
# parse options and perform corresponding command
if (!$cmd) {
die "Invalid command\n";
} elsif ($cmd eq 'chattr') {
die "Invalid options\n" if (scalar(@ARGV) > 0);
chattr();
} elsif ($cmd eq 'escape') {
die "Invalid options\n" if (scalar(@ARGV) != 1);
print escape($ARGV[0]);
} elsif ($cmd eq 'find') {
die "Invalid options\n" if (!GetOptions(\%opts,
"create-tar", "dereference|L", "exclude=s@", "extract-tar",
"find-files=i", "ignore-times", "include=s@", "newer=i", "older=i",
"preserve", "sync",
"find-files=i", "ignore-times", "include=s@", "newer=s", "older=s",
"preserve=s", "sync",
));
die "Invalid options\n" if (scalar(@ARGV) > 0);
find();
@ -107,7 +112,8 @@ if (!$cmd) {
'window' => 4194304,
);
die "Invalid options\n" if (!GetOptions(\%opts,
"buffer-size=i", "ports=s", "streams=i", "tcp", "verify", "window=i",
"buffer-size=i", "ports=s", "secure", "streams=i", "tcp", "verify",
"window=i",
));
die "Invalid options\n" if (scalar(@ARGV) > 0);
$opts{'buffer-size'} <<= 20;
@ -118,14 +124,28 @@ if (!$cmd) {
} elsif ($cmd eq 'sum') {
%opts = (
'buffer-size' => 4,
'hash-type' => "md5",
'split-size' => 1024,
'threads' => 4,
);
die "Invalid options\n" if (!GetOptions(\%opts,
"buffer-size=i", "c", "split-size=i", "threads=i",
"buffer-size=i", "c", "hash-type=s", "split-size=i", "threads=i",
));
die "Invalid options\n" if (!$opts{c});
my ($type, $bits) = split(/_/, $opts{'hash-type'});
# untaint arguments
$type = $1 if ($type =~ /(.*)/);
$bits = $1 if ($bits =~ /(.*)/s);
my $mod = "Digest::" . uc($type);
$opts{hash_ctx} = eval "require $mod; $mod->new($bits)";
$opts{hash_size} = length $opts{hash_ctx}->hexdigest;
sum();
} elsif ($cmd eq 'unescape') {
if (scalar(@ARGV)) {
print unescape($_) foreach (@ARGV);
} else {
print unescape($_) while (<STDIN>);
}
}
################
@ -135,14 +155,11 @@ if (!$cmd) {
sub chattr {
# check for existence of commands
my %have;
foreach my $bin (qw(fallocate lfs setfacl setfattr)) {
foreach my $path (split(/:/, $ENV{PATH})) {
if (-x "$path/$bin") {
$have{$bin} = 1;
last;
}
}
foreach my $bin (qw(fallocate lfs setfacl setfattr shift-bin)) {
$have{$bin} = first {-x "$_/$bin"} (split(/:/, $ENV{PATH}));
}
# offload to shift-bin for more efficient processing if possible
exec("shift-bin") if ($have{'shift-bin'});
while (my $line = <STDIN>) {
chomp $line;
@ -154,7 +171,6 @@ sub chattr {
$file = $1 if ($file =~ /(.*)/s);
$attrs = $1 if ($attrs =~ /(.*)/);
my $ufile = unescape($file);
my $uattrs = unescape($attrs);
# make sure parent directory exists
my $dir = $ufile =~ s/\/$// ? $ufile : dirname($ufile);
@ -162,22 +178,21 @@ sub chattr {
my ($cin, @copts, $in, $out);
if ($cmd eq 'fallocate') {
@copts = ("-n", "-l", $uattrs, $ufile);
} elsif ($cmd eq 'lfs') {
@copts = ("-n", "-l", unescape($attrs), $ufile);
} elsif ($cmd eq 'setstripe') {
$cmd = "lfs";
my ($count, $size, $pool) = split(/\s+/, unescape($attrs));
$count = 0 if (!$count);
$size = 0 if (!$size);
# stripes > 160 may fail due to max of lustre < 2.4
$count = 160 if ($count > 160);
@copts = ("setstripe", "-c", $count, "-S", $size, $ufile);
splice(@copts, -1, 0, "-p", $pool) if ($pool);
} elsif ($cmd eq 'setfacl') {
$uattrs =~ s/,/\n/g;
$cin = $uattrs;
@copts = ("-PM-", $ufile);
$attrs =~ s/,/\n/g;
$cin = unescape($attrs);
@copts = ("-M-", $ufile);
} elsif ($cmd eq 'setfattr') {
$uattrs =~ s/,/\n/g;
$cin = "# file: $ufile\n$uattrs";
$attrs =~ s/,/\n/g;
$cin = "# file: $ufile\n" . unescape($attrs);
@copts = ("-h", "--restore=-");
}
my $pid = IPC::Open3::open3($in, $out, $out, $cmd, @copts);
@ -222,8 +237,15 @@ sub find {
my $opt = pop @args;
$opts{$1} = $2 if ($opt =~ /(\w+)=(\S+)/);
}
find1(map {unescape($_)} @args);
my @uargs = map {unescape($_)} @args;
if ($opts{'extract-tar'}) {
# process local tar files
find_tar(@uargs);
} else {
find1(@uargs);
}
}
open3_wait($opts{fhpid}) if (defined $opts{fhpid});
if ($opts{dmtmp}) {
close $opts{dmfh};
@ -234,8 +256,9 @@ sub find {
delete $opts{dmfh};
delete $opts{dmtmp};
} else {
my $extra = $opts{'create-tar'} ? " -a" : "";
# ignore errors since files will be automatically retrieved anyway
open3_get([$opts{dmtmp}, -1, -1], "dmget -nq");
open3_get([$opts{dmtmp}, -1, -1], "dmget -nq$extra");
unlink $opts{dmtmp};
POSIX::_exit(0);
}
@ -248,25 +271,17 @@ sub find {
# output list of files/dirs beneath given paths with stat info
sub find1 {
my ($shost, $spath, $dst) = @_;
# process local tar files
if ($opts{'extract-tar'}) {
find_tar($shost, $spath, $dst);
return;
}
# check for existence of various commands
my %have;
foreach my $bin (qw(dmget lfs getfacl getfattr)) {
foreach my $path (split(/:/, $ENV{PATH})) {
if (-x "$path/$bin") {
$have{$bin} = 1;
last;
}
if (!defined $opts{have}) {
# check for existence of various commands
$opts{have} = {};
foreach my $bin (qw(dmget lfs getfacl getfattr shift-bin)) {
$opts{have}->{$bin} = first {-x "$_/$bin"} (split(/:/, $ENV{PATH}));
}
$opts{fhpid} = open3_run([undef, undef, -1], "shift-bin")
if ($opts{have}->{'shift-bin'});
}
my $dmf = $have{dmget} && $opts{srcfs} =~ /,dmi/ ? 1 : 0;
my $dmf = $opts{have}->{dmget} && $opts{srcfs} =~ /,dmf/ ? 1 : 0;
my $sdir = dirname($spath);
$sdir = "" if ($sdir eq '/');
@ -381,9 +396,33 @@ sub find1 {
}
next if (!$found);
}
# newer/older files (must be after dir processing)
next if (defined $opts{newer} && $stat[9] < $opts{newer});
next if (defined $opts{older} && $stat[9] >= $opts{older});
my %ti = ('a' => 8, 'm' => 9, 'c' => 10);
if (defined $opts{newer}) {
if ($opts{newer} =~ /^([^:]+):(\S+)/) {
my ($type, $time) = ($1, $2);
$type =~ s/\|/1||/g;
$type =~ s/([amc])/$stat[$ti{$1}]>=$time&&/g;
$type =~ s/([AMC])/$stat[$ti{lc($1)}]<$time&&/g;
$type .= "1";
next if (!eval $type);
} elsif ($stat[9] < $opts{newer}) {
next;
}
}
if (defined $opts{older}) {
if ($opts{older} =~ /^([^:]+):(\S+)/) {
my ($type, $time) = ($1, $2);
$type =~ s/\|/1||/g;
$type =~ s/([amc])/$stat[$ti{$1}]<$time&&/g;
$type =~ s/([AMC])/$stat[$ti{lc($1)}]>=$time&&/g;
$type .= "1";
next if (!eval $type);
} elsif ($stat[9] >= $opts{older}) {
next;
}
}
# dereference before stat
# resolve uid/gid if possible
@ -398,39 +437,68 @@ sub find1 {
my @lattrs;
my @xattrs;
# try to get acls
if ($have{getacl} && !$opts{'create-tar'} && $opts{preserve} &&
if (($opts{have}->{'shift-bin'} || $opts{have}->{getfacl}) &&
!$opts{'create-tar'} &&
($opts{preserve} == 1 || $opts{preserve} =~ /acl/) &&
(!$opts{srcfs} || $opts{srcfs} =~ /,acl/)) {
open(FILE, '-|', "getfacl", "-cPps", "--", $file);
while (<FILE>) {
chomp;
next if (!$_);
push(@acls, escape($_));
if (defined $opts{fhpid}) {
$opts{fhpid}->[0]->print("getfacl $file\n");
my $text = $opts{fhpid}->[1]->getline;
my @cols = split(/\s+/, $text);
push(@acls, $cols[2]) if ($file eq $cols[1] && $cols[-1] eq '0');
} else {
my $fhpid = open3_run([-1, undef, -1],
"getfacl", "-cps", "--", $file);
while (defined $fhpid && defined ($_ = $fhpid->[1]->getline)) {
chomp;
next if (!$_);
push(@acls, escape($_));
}
open3_wait($fhpid);
}
close FILE;
}
# try to get xattrs
if ($have{getfattr} && !$opts{'create-tar'} && $opts{preserve} &&
if (($opts{have}->{'shift-bin'} || $opts{have}->{getfattr}) &&
!$opts{'create-tar'} &&
($opts{preserve} == 1 || $opts{preserve} =~ /xattr/) &&
(!$opts{srcfs} || $opts{srcfs} =~ /,xattr/)) {
open(FILE, '-|', "getfattr", "-dhe", "base64", $file);
while (<FILE>) {
chomp;
next if (!$_ || /^\s*#/);
push(@xattrs, escape($_));
if (defined $opts{fhpid}) {
$opts{fhpid}->[0]->print("getfattr $file\n");
my $text = $opts{fhpid}->[1]->getline;
my @cols = split(/\s+/, $text);
push(@xattrs, $cols[2]) if ($file eq $cols[1] && $cols[-1] eq '0');
} else {
my $fhpid = open3_run([-1, undef, -1],
"getfattr", "-dhe", "base64", $file);
while (defined $fhpid && defined ($_ = $fhpid->[1]->getline)) {
chomp;
next if (!$_ || /^\s*#/);
push(@xattrs, escape(decode_base64($_)));
}
open3_wait($fhpid);
}
close FILE;
}
# try to get lustre striping
if ($have{lfs} && !S_ISLNK($mode) && !$opts{'create-tar'} &&
$opts{preserve} && $opts{srcfs} =~ /^lustre/) {
if (($opts{have}->{'shift-bin'} || $opts{have}->{lfs}) &&
!S_ISLNK($mode) && !$opts{'create-tar'} &&
($opts{preserve} == 1 || $opts{preserve} =~ /stripe/) &&
$opts{srcfs} =~ /^lustre/) {
# ignore symlinks as link to fifo can hang forever
open(FILE, '-|', "lfs", "getstripe", "-d", $file);
while (<FILE>) {
$lattrs[0] = $1 if (/stripe_count:\s*(-?\d+)/);
$lattrs[1] = $1 if (/stripe_size:\s*(-?\d+)/);
if (defined $opts{fhpid}) {
$opts{fhpid}->[0]->print("getstripe $file\n");
my $text = $opts{fhpid}->[1]->getline;
my @cols = split(/\s+/, $text);
@lattrs = split(/,/, $cols[2]) if ($file eq $cols[1] && $cols[-1] eq '0');
} else {
open(FILE, '-|', "lfs", "getstripe", "-d", $file);
while (<FILE>) {
$lattrs[0] = $1 if (/stripe_count:\s*(-?\d+)/);
$lattrs[1] = $1 if (/stripe_size:\s*(-?\d+)/);
}
close FILE;
}
close FILE;
}
$lattrs[0] = 0 if (!defined $lattrs[0] && defined $lattrs[1]);
$lattrs[1] = 0 if (!defined $lattrs[1] && defined $lattrs[0]);
@ -665,7 +733,8 @@ sub find_tar {
print " size=$attrs[4] attrs=", join(",", @attrs[1,2,3,5,5],
escape($attrs[11]), escape($attrs[12]), @attrs[4,4]);
my $bytes = $offset . "-" . ($offset + $attrs[4]);
print " bytes=$bytes tar_bytes=$bytes\n";
print " bytes=$bytes tar_bytes=$bytes";
print " tar_name=", escape(tar_canonpath($attrs[0])), "\n";
$nfiles++;
}
if (length($head) < 512) {
@ -692,11 +761,15 @@ sub fish {
# default is to indicate running
my $rc = "### 200";
my ($port, $sock, $key);
if (!$have_threads && $opts{tcp}) {
my ($cert, $key, $port, $sock);
if ($opts{tcp} && !$perl{threads}) {
# indicate that threads are not supported
$rc = "### 500 nothread";
$opts{tcp} = 0;
} elsif ($opts{tcp} && $opts{secure} && !$perl{ssl}) {
# indicate that ssl is not supported
$rc = "### 500 nossl";
$opts{tcp} = 0;
} elsif ($opts{tcp}) {
my ($port1, $port2) = split(/:/, $opts{ports});
foreach (sort {(-1,1)[rand 2]} ($port1..$port2)) {
@ -719,7 +792,20 @@ sub fish {
#TODO: de-hardcode 60 second timeout
$sock->sockopt(SO_RCVTIMEO, pack('L!L!', +60, 0));
$key = "" . rand();
$rc = "$port $key\n$rc";
my $scert;
if ($opts{secure}) {
require IO::Socket::SSL::Utils;
($scert, my $skey) = IO::Socket::SSL::Utils::CERT_create(
CA => 1,
subject => {CN => $key},
);
$scert = PEM_cert2string($scert) . PEM_key2string($skey);
(my $fh, $cert) = tempfile();
print $fh $scert;
close $fh;
$scert = " " . escape($scert);
}
$rc = "$port $key$scert\n$rc";
}
}
$out->write($rc . "\n");
@ -743,28 +829,45 @@ sub fish {
}
}
return if (!$opts{tcp});
require Digest::HMAC_SHA1;
require Digest::HMAC;
require Digest::SHA::PurePerl;
my @threads = map {threads->create(sub {
my ($tsock, $trc0);
$tsock = $sock->accept;
if ($opts{secure}) {
# this can only be reached if ssl is available
IO::Socket::SSL->start_SSL($tsock,
SSL_server => 1,
SSL_use_cert => 1,
SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_PEER(),
SSL_verifycn_name => $key,
SSL_hostname => $key,
SSL_ca_file => $cert,
SSL_cert_file => $cert,
SSL_key_file => $cert,
);
}
my $trc = fish_return($tsock, "rinit");
my ($nonce, $hmac) = split(/\s+/, $trc);
my $my_hmac = Digest::HMAC_SHA1::hmac_sha1_hex($nonce, $key);
my $my_hmac = Digest::HMAC::hmac_hex($nonce, $key,
\&Digest::SHA::PurePerl::sha512);
if ($hmac ne $my_hmac) {
# remote side cannot be authenticated
close $tsock;
return;
}
my $nonce2 = "" . rand();
my $hmac2 = Digest::HMAC_SHA1::hmac_sha1_hex($nonce . $nonce2, $key);
my $hmac2 = Digest::HMAC::hmac_hex($nonce . $nonce2, $key,
\&Digest::SHA::PurePerl::sha512);
$tsock->print($nonce2 . " " . $hmac2 . "\n### 100\n");
while (1) {
my $trc = fish_return($tsock, "rauth");
my ($fi, $fi_hmac) = split(/\s+/, $trc);
my $my_fi_hmac = Digest::HMAC_SHA1::hmac_sha1_hex(
$fi . $nonce2++, $key);
my $my_fi_hmac = Digest::HMAC::hmac_hex($fi . $nonce2++, $key,
\&Digest::SHA::PurePerl::sha512);
if ($fi_hmac ne $my_fi_hmac) {
# remote side cannot be authenticated
close $tsock;
@ -776,6 +879,7 @@ sub fish {
close $tsock;
})} (1 .. $opts{streams});
$_->join foreach (@threads);
unlink $cert if ($cert);
close $sock;
}
@ -931,31 +1035,43 @@ sub fqdn {
# output the set of remotely mounted file systems
sub mount {
my $host = fqdn(hostname);
my %mnt;
$mnt{host} = $host;
$mnt{args} = "mount";
my %mnt = (
host => $host,
args => "mount",
);
my %fstab;
if (open(FILE, "/etc/fstab")) {
while (<FILE>) {
s/^\s+|\s+$//g;
next if (/^#/);
my ($dev, $local, $type) = split(/\s+/);
next if (!$type);
$fstab{$local} = [$dev, $type];
}
close FILE;
}
# check for existence of getfacl
my $acl;
foreach my $path (split(/:/, $ENV{PATH})) {
if (-x "$path/getfacl") {
$acl = 1;
last;
}
}
my $acl = first {-x "$_/getfacl"} (split(/:/, $ENV{PATH}));
# gather file system information from mount
my $fhpid = open3_run([-1, undef, -1], "mount");
while (defined $fhpid && defined ($_ = $fhpid->[1]->getline)) {
$mnt{opts} = /[\(,]ro[\),]/ ? "ro" : "rw";
# acl support is the default unless explicitly disabled
$mnt{opts} .= ",acl" if (/[\(,]acl[\),]/ || $acl && !/[\(,]noacl[\),]/);
$mnt{opts} .= ",dmi" if (/[\(,]dmi[\),]/);
$mnt{opts} .= ",dmf" if (/[\(,]dm(ap)?i[\),]/);
$mnt{opts} .= ",xattr" if (/[\(,]user_xattr[\),]/);
#TODO: need to escape local and remote?
(my $dev, $mnt{local}, $mnt{type}) = ($1, $2, $3)
(my $dev, $mnt{local}, my $type) = ($1, $2, $3)
if (/(\S+)\s+on\s+(\S+)\s+type\s+(\S+)/);
if ($mnt{local}) {
if ($dev eq 'systemd-1') {
# systemd mounts must be read from fstab
my $fstab = $fstab{$mnt{local}};
($dev, $type) = ($fstab->[0], $fstab->[1]);
next if (!$type);
}
# try to avoid NFS hangs by resolving dir but not base
my ($base, $dir) = fileparse($mnt{local});
$dir = abs_path($dir);
@ -964,16 +1080,17 @@ sub mount {
}
if (/server_list=\(([^\)]+)\)/) {
# cxfs appears as xfs but with server_list set
$mnt{servers} = join(",", map {$_ = fqdn($_)} split(/,/, $1));
$mnt{type} = "cxfs";
$mnt{servers} = join("|", map {$_ = fqdn($_)} split(/,/, $1));
$mnt{remote} = $mnt{local};
} elsif (/^(\S+):(\S+)/) {
$type = "cxfs";
} elsif ($dev =~ /^(\S+):([^:]+)$/) {
# typical form for nfs
$mnt{remote} = $2;
$mnt{servers} = $1;
$mnt{servers} =~ s/@.*//;
$mnt{servers} = fqdn($mnt{servers});
} elsif ($mnt{type} eq 'gpfs') {
# lustre may have extra @id and multiple colon-separated servers
$mnt{servers} =~ s/@\w*//g;
$mnt{servers} = join("|", map {$_ = fqdn($_)} split(/:/, $mnt{servers}));
} elsif ($type eq 'gpfs') {
# gpfs servers do not appear in mount output so call mmlsmgr
my $srv = open3_get([-1, undef, -1], "mmlsmgr $dev");
# try a default location if not in path
@ -985,14 +1102,17 @@ sub mount {
$mnt{remote} = "/$1";
$mnt{servers} = fqdn($2);
}
} elsif ($mnt{opts} =~ /,dmi/) {
# always report dmi file systems even if local
} elsif ($mnt{opts} =~ /,dmf/) {
# always report dmf file systems even if local
$mnt{servers} = $mnt{host};
$mnt{remote} = $mnt{local};
} else {
# ignore local file systems
next;
}
$mnt{opts} = "$type,$mnt{opts}";
# ensure servers are in same order across all hosts
$mnt{servers} = join("|", sort(split(/\|/, $mnt{servers})));
# print hash in single line with space-separated key=val form
print join(" ", map {"$_=$mnt{$_}"} sort(keys(%mnt))), "\n";
}
@ -1109,8 +1229,7 @@ sub sum {
if ($opts{'split-size'} < $opts{'buffer-size'});
my ($qi, $q, $qret, @sums);
if ($have_threads && $opts{threads} > 1) {
require Thread::Queue;
if ($perl{threads} && $opts{threads} > 1) {
$q = Thread::Queue->new;
$qret = Thread::Queue->new;
$qi = 0;
@ -1139,7 +1258,7 @@ sub sum {
$start = 0 if (!defined $start);
# use file end if no end given
$stop = (stat($file))[7] if (!defined $stop);
if (!$have_threads || $opts{threads} <= 1) {
if (!$perl{threads} || $opts{threads} <= 1) {
my $hash = sum1($file, $start, $stop);
sum_check($file, $hash0, $hash, $start, $stop, $partial);
next;
@ -1155,7 +1274,7 @@ sub sum {
}
}
return if (!$have_threads || $opts{threads} <= 1);
return if (!$perl{threads} || $opts{threads} <= 1);
# choose min of specified threads and amount of work
my $nthr = min($opts{threads}, $q->pending);
my @threads = map {threads->create(sub {
@ -1189,14 +1308,14 @@ sub sum1 {
if (open($fh, '<', $file)) {
if ($start == $stop) {
# compute empty hex hash
$hash .= unpack("H*", md5(""));
$hash .= unpack("H*", $opts{hash_ctx}->digest);
} else {
# compute concatenated list of hex hashes for each split
for (my $x1 = $start; $x1 < $stop; $x1 += $opts{'split-size'}) {
my $x2 = min($x1 + $opts{'split-size'}, $stop);
sysseek($fh, $x1, 0) or
print STDERR "Unable to seek '$file': $!\n";
my ($buf, $md5, $total) = ("", Digest::MD5->new, 0);
my ($buf, $ctx, $total) = ("", $opts{hash_ctx}->clone, 0);
while ($total < $x2 - $x1) {
# read data into buffer
my $n = sysread($fh, $buf,
@ -1205,10 +1324,10 @@ sub sum1 {
last if (!$n);
last if (!$n);
# add data to hash
$md5->add($buf);
$ctx->add($buf);
$total += $n;
}
$hash .= unpack("H*", $md5->digest);
$hash .= unpack("H*", $ctx->digest);
}
}
close $fh;
@ -1238,7 +1357,8 @@ sub sum_check {
my $i = 0;
for (my $x1 = $start; $x1 < $stop; $x1 += $opts{'split-size'}) {
my $x2 = min($x1 + $opts{'split-size'}, $stop);
if (substr($hash, $i * 32, 32) ne substr($hash0, $i * 32, 32)) {
if (substr($hash, $i * $opts{hash_size}, $opts{hash_size}) ne
substr($hash0, $i * $opts{hash_size}, $opts{hash_size})) {
print ",", $x1, "-", $x2;
}
$i++;
@ -1352,9 +1472,9 @@ sub verify_buffer_leaf {
#####################
sub verify_init {
my %sopts = (
buffer_size => $opts{buffer} ? $opts{buffer} >> 20 : 4,
buffer_size => $opts{buffer},
hash_ctx => Digest::MD5->new,
split_size => 1024,
split_size => $opts{'split-size'},
stack => [],
@_,
);
@ -1386,16 +1506,8 @@ $fatpacked{"Digest/HMAC.pm"} = '#line '.(1+__LINE__).' "'.__FILE__."\"\n".<<'DIG
package Digest::HMAC;$VERSION="1.03";use strict;sub new {my($class,$key,$hasher,$block_size)=@_;$block_size ||=64;$key=$hasher->new->add($key)->digest if length($key)> $block_size;my$self=bless {},$class;$self->{k_ipad}=$key ^ (chr(0x36)x $block_size);$self->{k_opad}=$key ^ (chr(0x5c)x $block_size);$self->{hasher}=$hasher->new->add($self->{k_ipad});$self}sub reset {my$self=shift;$self->{hasher}->reset->add($self->{k_ipad});$self}sub add {my$self=shift;$self->{hasher}->add(@_);$self}sub addfile {my$self=shift;$self->{hasher}->addfile(@_);$self}sub _digest {my$self=shift;my$inner_digest=$self->{hasher}->digest;$self->{hasher}->reset->add($self->{k_opad},$inner_digest)}sub digest {shift->_digest->digest}sub hexdigest {shift->_digest->hexdigest}sub b64digest {shift->_digest->b64digest}require Exporter;*import=\&Exporter::import;use vars qw(@EXPORT_OK);@EXPORT_OK=qw(hmac hmac_hex);sub hmac {my($data,$key,$hash_func,$block_size)=@_;$block_size ||=64;$key=&$hash_func($key)if length($key)> $block_size;my$k_ipad=$key ^ (chr(0x36)x $block_size);my$k_opad=$key ^ (chr(0x5c)x $block_size);&$hash_func($k_opad,&$hash_func($k_ipad,$data))}sub hmac_hex {unpack("H*",&hmac)}1;
DIGEST_HMAC
$fatpacked{"Digest/HMAC_MD5.pm"} = '#line '.(1+__LINE__).' "'.__FILE__."\"\n".<<'DIGEST_HMAC_MD5';
package Digest::HMAC_MD5;$VERSION="1.01";use strict;use Digest::MD5 qw(md5);use Digest::HMAC qw(hmac);use vars qw(@ISA);@ISA=qw(Digest::HMAC);sub new {my$class=shift;$class->SUPER::new($_[0],"Digest::MD5",64)}require Exporter;*import=\&Exporter::import;use vars qw(@EXPORT_OK);@EXPORT_OK=qw(hmac_md5 hmac_md5_hex);sub hmac_md5 {hmac($_[0],$_[1],\&md5,64)}sub hmac_md5_hex {unpack("H*",&hmac_md5)}1;
DIGEST_HMAC_MD5
$fatpacked{"Digest/HMAC_SHA1.pm"} = '#line '.(1+__LINE__).' "'.__FILE__."\"\n".<<'DIGEST_HMAC_SHA1';
package Digest::HMAC_SHA1;$VERSION="1.03";use strict;use Digest::SHA::PurePerl qw(sha1);use Digest::HMAC qw(hmac);use vars qw(@ISA);@ISA=qw(Digest::HMAC);sub new {my$class=shift;$class->SUPER::new($_[0],"Digest::SHA",64)}require Exporter;*import=\&Exporter::import;use vars qw(@EXPORT_OK);@EXPORT_OK=qw(hmac_sha1 hmac_sha1_hex);sub hmac_sha1 {hmac($_[0],$_[1],\&sha1,64)}sub hmac_sha1_hex {unpack("H*",&hmac_sha1)}1;
DIGEST_HMAC_SHA1
$fatpacked{"Digest/SHA/PurePerl.pm"} = '#line '.(1+__LINE__).' "'.__FILE__."\"\n".<<'DIGEST_SHA_PUREPERL';
package Digest::SHA::PurePerl;require 5.003000;use strict;use warnings;use vars qw($VERSION @ISA @EXPORT @EXPORT_OK);use Fcntl;use integer;use Carp qw(croak);$VERSION='5.95';require Exporter;@ISA=qw(Exporter);@EXPORT_OK=();eval {require Digest::base;push(@ISA,'Digest::base')};my$MAX32=0xffffffff;my$uses64bit=(((1 << 16)<< 16)<< 16)<< 15;my@H01=(0x67452301,0xefcdab89,0x98badcfe,0x10325476,0xc3d2e1f0);my@H0224=(0xc1059ed8,0x367cd507,0x3070dd17,0xf70e5939,0xffc00b31,0x68581511,0x64f98fa7,0xbefa4fa4);my@H0256=(0x6a09e667,0xbb67ae85,0x3c6ef372,0xa54ff53a,0x510e527f,0x9b05688c,0x1f83d9ab,0x5be0cd19);my(@H0384,@H0512,@H0512224,@H0512256);sub _c_SL32 {my($x,$n)=@_;"($x << $n)"}sub _c_SR32 {my($x,$n)=@_;my$mask=(1 << (32 - $n))- 1;"(($x >> $n) & $mask)"}sub _c_Ch {my($x,$y,$z)=@_;"($z ^ ($x & ($y ^ $z)))"}sub _c_Pa {my($x,$y,$z)=@_;"($x ^ $y ^ $z)"}sub _c_Ma {my($x,$y,$z)=@_;"(($x & $y) | ($z & ($x | $y)))"}sub _c_ROTR {my($x,$n)=@_;"(" ._c_SR32($x,$n)." | " ._c_SL32($x,32 - $n).")"}sub _c_ROTL {my($x,$n)=@_;"(" ._c_SL32($x,$n)." | " ._c_SR32($x,32 - $n).")"}sub _c_SIGMA0 {my($x)=@_;"(" ._c_ROTR($x,2)." ^ " ._c_ROTR($x,13)." ^ " ._c_ROTR($x,22).")"}sub _c_SIGMA1 {my($x)=@_;"(" ._c_ROTR($x,6)." ^ " ._c_ROTR($x,11)." ^ " ._c_ROTR($x,25).")"}sub _c_sigma0 {my($x)=@_;"(" ._c_ROTR($x,7)." ^ " ._c_ROTR($x,18)." ^ " ._c_SR32($x,3).")"}sub _c_sigma1 {my($x)=@_;"(" ._c_ROTR($x,17)." ^ " ._c_ROTR($x,19)." ^ " ._c_SR32($x,10).")"}sub _c_M1Ch {my($a,$b,$c,$d,$e,$k,$w)=@_;"$e += " ._c_ROTL($a,5)." + " ._c_Ch($b,$c,$d)." + $k + $w; $b = " ._c_ROTL($b,30).";\n"}sub _c_M1Pa {my($a,$b,$c,$d,$e,$k,$w)=@_;"$e += " ._c_ROTL($a,5)." + " ._c_Pa($b,$c,$d)." + $k + $w; $b = " ._c_ROTL($b,30).";\n"}sub _c_M1Ma {my($a,$b,$c,$d,$e,$k,$w)=@_;"$e += " ._c_ROTL($a,5)." + " ._c_Ma($b,$c,$d)." + $k + $w; $b = " ._c_ROTL($b,30).";\n"}sub _c_M11Ch {my($k,$w)=@_;_c_M1Ch('$a','$b','$c','$d','$e',$k,$w)}sub _c_M11Pa {my($k,$w)=@_;_c_M1Pa('$a','$b','$c','$d','$e',$k,$w)}sub _c_M11Ma {my($k,$w)=@_;_c_M1Ma('$a','$b','$c','$d','$e',$k,$w)}sub _c_M12Ch {my($k,$w)=@_;_c_M1Ch('$e','$a','$b','$c','$d',$k,$w)}sub _c_M12Pa {my($k,$w)=@_;_c_M1Pa('$e','$a','$b','$c','$d',$k,$w)}sub _c_M12Ma {my($k,$w)=@_;_c_M1Ma('$e','$a','$b','$c','$d',$k,$w)}sub _c_M13Ch {my($k,$w)=@_;_c_M1Ch('$d','$e','$a','$b','$c',$k,$w)}sub _c_M13Pa {my($k,$w)=@_;_c_M1Pa('$d','$e','$a','$b','$c',$k,$w)}sub _c_M13Ma {my($k,$w)=@_;_c_M1Ma('$d','$e','$a','$b','$c',$k,$w)}sub _c_M14Ch {my($k,$w)=@_;_c_M1Ch('$c','$d','$e','$a','$b',$k,$w)}sub _c_M14Pa {my($k,$w)=@_;_c_M1Pa('$c','$d','$e','$a','$b',$k,$w)}sub _c_M14Ma {my($k,$w)=@_;_c_M1Ma('$c','$d','$e','$a','$b',$k,$w)}sub _c_M15Ch {my($k,$w)=@_;_c_M1Ch('$b','$c','$d','$e','$a',$k,$w)}sub _c_M15Pa {my($k,$w)=@_;_c_M1Pa('$b','$c','$d','$e','$a',$k,$w)}sub _c_M15Ma {my($k,$w)=@_;_c_M1Ma('$b','$c','$d','$e','$a',$k,$w)}sub _c_W11 {my($s)=@_;'$W[' .(($s + 0)& 0xf).']'}sub _c_W12 {my($s)=@_;'$W[' .(($s + 13)& 0xf).']'}sub _c_W13 {my($s)=@_;'$W[' .(($s + 8)& 0xf).']'}sub _c_W14 {my($s)=@_;'$W[' .(($s + 2)& 0xf).']'}sub _c_A1 {my($s)=@_;my$tmp=_c_W11($s)." ^ " ._c_W12($s)." ^ " ._c_W13($s)." ^ " ._c_W14($s);"((\$tmp = $tmp), (" ._c_W11($s)." = " ._c_ROTL('$tmp',1)."))"}my$sha1_code='
package Digest::SHA::PurePerl;require 5.003000;use strict;use warnings;use vars qw($VERSION @ISA @EXPORT_OK $errmsg);use Fcntl qw(O_RDONLY);use integer;use Carp qw(croak);$VERSION='6.01';require Exporter;@ISA=qw(Exporter);@EXPORT_OK=('$errmsg');eval {require Digest::base;push(@ISA,'Digest::base')};my$MAX32=0xffffffff;my$uses64bit=(((1 << 16)<< 16)<< 16)<< 15;my@H01=(0x67452301,0xefcdab89,0x98badcfe,0x10325476,0xc3d2e1f0);my@H0224=(0xc1059ed8,0x367cd507,0x3070dd17,0xf70e5939,0xffc00b31,0x68581511,0x64f98fa7,0xbefa4fa4);my@H0256=(0x6a09e667,0xbb67ae85,0x3c6ef372,0xa54ff53a,0x510e527f,0x9b05688c,0x1f83d9ab,0x5be0cd19);my(@H0384,@H0512,@H0512224,@H0512256);sub _c_SL32 {my($x,$n)=@_;"($x << $n)"}sub _c_SR32 {my($x,$n)=@_;my$mask=(1 << (32 - $n))- 1;"(($x >> $n) & $mask)"}sub _c_Ch {my($x,$y,$z)=@_;"($z ^ ($x & ($y ^ $z)))"}sub _c_Pa {my($x,$y,$z)=@_;"($x ^ $y ^ $z)"}sub _c_Ma {my($x,$y,$z)=@_;"(($x & $y) | ($z & ($x | $y)))"}sub _c_ROTR {my($x,$n)=@_;"(" ._c_SR32($x,$n)." | " ._c_SL32($x,32 - $n).")"}sub _c_ROTL {my($x,$n)=@_;"(" ._c_SL32($x,$n)." | " ._c_SR32($x,32 - $n).")"}sub _c_SIGMA0 {my($x)=@_;"(" ._c_ROTR($x,2)." ^ " ._c_ROTR($x,13)." ^ " ._c_ROTR($x,22).")"}sub _c_SIGMA1 {my($x)=@_;"(" ._c_ROTR($x,6)." ^ " ._c_ROTR($x,11)." ^ " ._c_ROTR($x,25).")"}sub _c_sigma0 {my($x)=@_;"(" ._c_ROTR($x,7)." ^ " ._c_ROTR($x,18)." ^ " ._c_SR32($x,3).")"}sub _c_sigma1 {my($x)=@_;"(" ._c_ROTR($x,17)." ^ " ._c_ROTR($x,19)." ^ " ._c_SR32($x,10).")"}sub _c_M1Ch {my($a,$b,$c,$d,$e,$k,$w)=@_;"$e += " ._c_ROTL($a,5)." + " ._c_Ch($b,$c,$d)." + $k + $w; $b = " ._c_ROTL($b,30).";\n"}sub _c_M1Pa {my($a,$b,$c,$d,$e,$k,$w)=@_;"$e += " ._c_ROTL($a,5)." + " ._c_Pa($b,$c,$d)." + $k + $w; $b = " ._c_ROTL($b,30).";\n"}sub _c_M1Ma {my($a,$b,$c,$d,$e,$k,$w)=@_;"$e += " ._c_ROTL($a,5)." + " ._c_Ma($b,$c,$d)." + $k + $w; $b = " ._c_ROTL($b,30).";\n"}sub _c_M11Ch {my($k,$w)=@_;_c_M1Ch('$a','$b','$c','$d','$e',$k,$w)}sub _c_M11Pa {my($k,$w)=@_;_c_M1Pa('$a','$b','$c','$d','$e',$k,$w)}sub _c_M11Ma {my($k,$w)=@_;_c_M1Ma('$a','$b','$c','$d','$e',$k,$w)}sub _c_M12Ch {my($k,$w)=@_;_c_M1Ch('$e','$a','$b','$c','$d',$k,$w)}sub _c_M12Pa {my($k,$w)=@_;_c_M1Pa('$e','$a','$b','$c','$d',$k,$w)}sub _c_M12Ma {my($k,$w)=@_;_c_M1Ma('$e','$a','$b','$c','$d',$k,$w)}sub _c_M13Ch {my($k,$w)=@_;_c_M1Ch('$d','$e','$a','$b','$c',$k,$w)}sub _c_M13Pa {my($k,$w)=@_;_c_M1Pa('$d','$e','$a','$b','$c',$k,$w)}sub _c_M13Ma {my($k,$w)=@_;_c_M1Ma('$d','$e','$a','$b','$c',$k,$w)}sub _c_M14Ch {my($k,$w)=@_;_c_M1Ch('$c','$d','$e','$a','$b',$k,$w)}sub _c_M14Pa {my($k,$w)=@_;_c_M1Pa('$c','$d','$e','$a','$b',$k,$w)}sub _c_M14Ma {my($k,$w)=@_;_c_M1Ma('$c','$d','$e','$a','$b',$k,$w)}sub _c_M15Ch {my($k,$w)=@_;_c_M1Ch('$b','$c','$d','$e','$a',$k,$w)}sub _c_M15Pa {my($k,$w)=@_;_c_M1Pa('$b','$c','$d','$e','$a',$k,$w)}sub _c_M15Ma {my($k,$w)=@_;_c_M1Ma('$b','$c','$d','$e','$a',$k,$w)}sub _c_W11 {my($s)=@_;'$W[' .(($s + 0)& 0xf).']'}sub _c_W12 {my($s)=@_;'$W[' .(($s + 13)& 0xf).']'}sub _c_W13 {my($s)=@_;'$W[' .(($s + 8)& 0xf).']'}sub _c_W14 {my($s)=@_;'$W[' .(($s + 2)& 0xf).']'}sub _c_A1 {my($s)=@_;my$tmp=_c_W11($s)." ^ " ._c_W12($s)." ^ " ._c_W13($s)." ^ " ._c_W14($s);"((\$tmp = $tmp), (" ._c_W11($s)." = " ._c_ROTL('$tmp',1)."))"}my$sha1_code='
my($K1, $K2, $K3, $K4) = ( # SHA-1 constants
0x5a827999, 0x6ed9eba1, 0x8f1bbcdc, 0xca62c1d6
@ -1587,11 +1699,11 @@ $fatpacked{"Digest/SHA/PurePerl.pm"} = '#line '.(1+__LINE__).' "'.__FILE__."\"\n
for (@_) { _hmacWrite(\$_, length($_), $state) }
_hmacfinish($state);
_hmac' .$suffix_intern[$i].'($state);
}';eval($fcn);push(@EXPORT_OK,'hmac_sha' .$alg .$suffix_extern[$i])}}sub hashsize {my$self=shift;_shadsize($self)<< 3}sub algorithm {my$self=shift;$self->{alg}}sub add {my$self=shift;for (@_){_shaWrite(\$_,length($_),$self)}$self}sub digest {my$self=shift;_shafinish($self);my$rsp=_shadigest($self);_sharewind($self);$rsp}sub hexdigest {my$self=shift;_shafinish($self);my$rsp=_shahex($self);_sharewind($self);$rsp}sub b64digest {my$self=shift;_shafinish($self);my$rsp=_shabase64($self);_sharewind($self);$rsp}sub new {my($class,$alg)=@_;$alg =~ s/\D+//g if defined$alg;if (ref($class)){if (!defined($alg)|| ($alg==$class->algorithm)){_sharewind($class);return($class)}my$self=_shaopen($alg)or return;return(_shacpy($class,$self))}$alg=1 unless defined$alg;my$self=_shaopen($alg)or return;bless($self,$class);$self}sub clone {my$self=shift;my$copy=_shadup($self)or return;bless($copy,ref($self))}BEGIN {*reset=\&new}sub add_bits {my($self,$data,$nbits)=@_;unless (defined$nbits){$nbits=length($data);$data=pack("B*",$data)}$nbits=length($data)* 8 if$nbits > length($data)* 8;_shawrite($data,$nbits,$self);return($self)}sub _bail {my$msg=shift;$msg .= ": $!";croak$msg}sub _addfile {my ($self,$handle)=@_;my$n;my$buf="";while (($n=read($handle,$buf,4096))){$self->add($buf)}_bail("Read failed")unless defined$n;$self}{my$_can_T_filehandle;sub _istext {local*FH=shift;my$file=shift;if (!defined$_can_T_filehandle){local $^W=0;my$istext=eval {-T FH};$_can_T_filehandle=$@ ? 0 : 1;return$_can_T_filehandle ? $istext : -T $file}return$_can_T_filehandle ? -T FH : -T $file}}sub addfile {my ($self,$file,$mode)=@_;return(_addfile($self,$file))unless ref(\$file)eq 'SCALAR';$mode=defined($mode)? $mode : "";my ($binary,$UNIVERSAL,$BITS,$portable)=map {$_ eq $mode}("b","U","0","p");local*FH;$file eq '-' and open(FH,'< -')or sysopen(FH,$file,O_RDONLY)or _bail('Open failed');if ($BITS){my ($n,$buf)=(0,"");while (($n=read(FH,$buf,4096))){$buf =~ s/[^01]//g;$self->add_bits($buf)}_bail("Read failed")unless defined$n;close(FH);return($self)}binmode(FH)if$binary || $portable || $UNIVERSAL;if ($UNIVERSAL && _istext(*FH,$file)){while (<FH>){s/\015\012/\012/g;s/\015/\012/g;$self->add($_)}}elsif ($portable && _istext(*FH,$file)){while (<FH>){s/\015?\015\012/\012/g;s/\015/\012/g;$self->add($_)}}else {$self->_addfile(*FH)}close(FH);$self}sub getstate {my$self=shift;return _shadump($self)}sub putstate {my$class=shift;my$state=shift;if (ref($class)){my$self=_shaload($state)or return;return(_shacpy($class,$self))}my$self=_shaload($state)or return;bless($self,$class);return($self)}sub dump {my$self=shift;my$file=shift;my$state=$self->getstate or return;$file="-" if (!defined($file)|| $file eq "");local*FH;open(FH,"> $file")or return;print FH$state;close(FH);return($self)}sub load {my$class=shift;my$file=shift;$file="-" if (!defined($file)|| $file eq "");local*FH;open(FH,"< $file")or return;my$str=join('',<FH>);close(FH);$class->putstate($str)}1;
}';eval($fcn);push(@EXPORT_OK,'hmac_sha' .$alg .$suffix_extern[$i])}}sub hashsize {my$self=shift;_shadsize($self)<< 3}sub algorithm {my$self=shift;$self->{alg}}sub add {my$self=shift;for (@_){_shaWrite(\$_,length($_),$self)}$self}sub digest {my$self=shift;_shafinish($self);my$rsp=_shadigest($self);_sharewind($self);$rsp}sub hexdigest {my$self=shift;_shafinish($self);my$rsp=_shahex($self);_sharewind($self);$rsp}sub b64digest {my$self=shift;_shafinish($self);my$rsp=_shabase64($self);_sharewind($self);$rsp}sub new {my($class,$alg)=@_;$alg =~ s/\D+//g if defined$alg;if (ref($class)){if (!defined($alg)|| ($alg==$class->algorithm)){_sharewind($class);return($class)}my$self=_shaopen($alg)or return;return(_shacpy($class,$self))}$alg=1 unless defined$alg;my$self=_shaopen($alg)or return;bless($self,$class);$self}sub clone {my$self=shift;my$copy=_shadup($self)or return;bless($copy,ref($self))}BEGIN {*reset=\&new}sub add_bits {my($self,$data,$nbits)=@_;unless (defined$nbits){$nbits=length($data);$data=pack("B*",$data)}$nbits=length($data)* 8 if$nbits > length($data)* 8;_shawrite($data,$nbits,$self);return($self)}sub _bail {my$msg=shift;$errmsg=$!;$msg .= ": $!";croak$msg}sub _addfile {my ($self,$handle)=@_;my$n;my$buf="";while (($n=read($handle,$buf,4096))){$self->add($buf)}_bail("Read failed")unless defined$n;$self}{my$_can_T_filehandle;sub _istext {local*FH=shift;my$file=shift;if (!defined$_can_T_filehandle){local $^W=0;my$istext=eval {-T FH};$_can_T_filehandle=$@ ? 0 : 1;return$_can_T_filehandle ? $istext : -T $file}return$_can_T_filehandle ? -T FH : -T $file}}sub addfile {my ($self,$file,$mode)=@_;return(_addfile($self,$file))unless ref(\$file)eq 'SCALAR';$mode=defined($mode)? $mode : "";my ($binary,$UNIVERSAL,$BITS)=map {$_ eq $mode}("b","U","0");local*FH;$file eq '-' and open(FH,'< -')or sysopen(FH,$file,O_RDONLY)or _bail('Open failed');if ($BITS){my ($n,$buf)=(0,"");while (($n=read(FH,$buf,4096))){$buf =~ tr/01//cd;$self->add_bits($buf)}_bail("Read failed")unless defined$n;close(FH);return($self)}binmode(FH)if$binary || $UNIVERSAL;if ($UNIVERSAL && _istext(*FH,$file)){while (<FH>){s/\015\012/\012/g;s/\015/\012/g;$self->add($_)}}else {$self->_addfile(*FH)}close(FH);$self}sub getstate {my$self=shift;return _shadump($self)}sub putstate {my$class=shift;my$state=shift;if (ref($class)){my$self=_shaload($state)or return;return(_shacpy($class,$self))}my$self=_shaload($state)or return;bless($self,$class);return($self)}sub dump {my$self=shift;my$file=shift;my$state=$self->getstate or return;$file="-" if (!defined($file)|| $file eq "");local*FH;open(FH,"> $file")or return;print FH$state;close(FH);return($self)}sub load {my$class=shift;my$file=shift;$file="-" if (!defined($file)|| $file eq "");local*FH;open(FH,"< $file")or return;my$str=join('',<FH>);close(FH);$class->putstate($str)}1;
DIGEST_SHA_PUREPERL
$fatpacked{"IPC/Open3.pm"} = '#line '.(1+__LINE__).' "'.__FILE__."\"\n".<<'IPC_OPEN3';
package IPC::Open3;use strict;no strict 'refs';our ($VERSION,@ISA,@EXPORT);require Exporter;use Carp;use Symbol qw(gensym qualify);$VERSION='1.18';@ISA=qw(Exporter);@EXPORT=qw(open3);our$Me='open3 (bug)';sub xpipe {pipe $_[0],$_[1]or croak "$Me: pipe($_[0], $_[1]) failed: $!"}sub xopen {open $_[0],$_[1],@_[2..$#_]and return;local $"=', ';carp "$Me: open(@_) failed: $!"}sub xclose {$_[0]=~ /\A=?(\d+)\z/ ? do {my$fh;open($fh,$_[1].'&=' .$1)and close($fh)}: close $_[0]or croak "$Me: close($_[0]) failed: $!"}sub xfileno {return $1 if $_[0]=~ /\A=?(\d+)\z/;return fileno $_[0]}use constant FORCE_DEBUG_SPAWN=>0;use constant DO_SPAWN=>$^O eq 'os2' || $^O eq 'MSWin32' || FORCE_DEBUG_SPAWN;sub _open3 {local$Me=shift;splice @_,0,1,undef if \$_[0]==\undef;splice @_,1,1,undef if \$_[1]==\undef;unless (eval {$_[0]=gensym unless defined $_[0]&& length $_[0];$_[1]=gensym unless defined $_[1]&& length $_[1];1}){$@ =~ s/(?<=value attempted) at .*//s;croak "$Me: $@"}my@handles=({mode=>'<',handle=>\*STDIN },{mode=>'>',handle=>\*STDOUT },{mode=>'>',handle=>\*STDERR },);for (@handles){$_->{parent}=shift;$_->{open_as}=gensym}if (@_ > 1 and $_[0]eq '-'){croak "Arguments don't make sense when the command is '-'"}$handles[2]{parent}||=$handles[1]{parent};$handles[2]{dup_of_out}=$handles[1]{parent}eq $handles[2]{parent};my$package;for (@handles){$_->{dup}=($_->{parent}=~ s/^[<>]&//);if ($_->{parent}!~ /\A=?(\d+)\z/){$package=caller 1 if (!defined$package);$_->{parent}=qualify $_->{parent},$package}next if $_->{dup}or $_->{dup_of_out};if ($_->{mode}eq '<'){xpipe $_->{open_as},$_->{parent}}else {xpipe $_->{parent},$_->{open_as}}}my$kidpid;if (!DO_SPAWN){xpipe my$stat_r,my$stat_w;$kidpid=fork;croak "$Me: fork failed: $!" unless defined$kidpid;if ($kidpid==0){eval {untie*STDIN;untie*STDOUT;untie*STDERR;close$stat_r;require Fcntl;my$flags=fcntl$stat_w,&Fcntl::F_GETFD,0;croak "$Me: fcntl failed: $!" unless$flags;fcntl$stat_w,&Fcntl::F_SETFD,$flags|&Fcntl::FD_CLOEXEC or croak "$Me: fcntl failed: $!";if (!$handles[2]{dup_of_out}&& $handles[2]{dup}&& xfileno($handles[2]{parent})==fileno \*STDOUT){my$tmp=gensym;xopen($tmp,'>&',$handles[2]{parent});$handles[2]{parent}=$tmp}for (@handles){if ($_->{dup_of_out}){xopen \*STDERR,">&STDOUT" if defined fileno STDERR && fileno STDERR!=fileno STDOUT}elsif ($_->{dup}){xopen $_->{handle},$_->{mode}.'&',$_->{parent}if fileno $_->{handle}!=xfileno($_->{parent})}else {xclose $_->{parent},$_->{mode};xopen $_->{handle},$_->{mode}.'&=',fileno $_->{open_as}}}return 1 if ($_[0]eq '-');exec @_ or do {local($")=(" ");croak "$Me: exec of @_ failed"}}and do {close$stat_w;return 0};my$bang=0+$!;my$err=$@;utf8::encode$err if $] >= 5.008;print$stat_w pack('IIa*',$bang,length($err),$err);close$stat_w;eval {require POSIX;POSIX::_exit(255)};exit 255}else {close$stat_w;my$to_read=length(pack('I',0))* 2;my$bytes_read=read($stat_r,my$buf='',$to_read);if ($bytes_read){(my$bang,$to_read)=unpack('II',$buf);read($stat_r,my$err='',$to_read);waitpid$kidpid,0;if ($err){utf8::decode$err if $] >= 5.008}else {$err="$Me: " .($!=$bang)}$!=$bang;die($err)}}}else {my@close;for (@handles){if ($_->{dup_of_out}){$_->{open_as}=$handles[1]{open_as}}elsif ($_->{dup}){$_->{open_as}=$_->{parent}=~ /\A[0-9]+\z/ ? $_->{parent}: \*{$_->{parent}};push@close,$_->{open_as}}else {push@close,\*{$_->{parent}},$_->{open_as}}}require IO::Pipe;$kidpid=eval {spawn_with_handles(\@handles,\@close,@_)};die "$Me: $@" if $@}for (@handles){next if $_->{dup}or $_->{dup_of_out};xclose $_->{open_as},$_->{mode}}xclose$handles[0]{parent},$handles[0]{mode}if$handles[0]{dup};select((select($handles[0]{parent}),$|=1)[0]);$kidpid}sub open3 {if (@_ < 4){local $"=', ';croak "open3(@_): not enough arguments"}return _open3 'open3',@_}sub spawn_with_handles {my$fds=shift;my$close_in_child=shift;my ($fd,%saved,@errs);for$fd (@$fds){$fd->{tmp_copy}=IO::Handle->new_from_fd($fd->{handle},$fd->{mode});$saved{fileno$fd->{handle}}=$fd->{tmp_copy}if$fd->{tmp_copy}}for$fd (@$fds){bless$fd->{handle},'IO::Handle' unless eval {$fd->{handle}->isa('IO::Handle')};my$open_as=$fd->{open_as};my$fileno=fileno($open_as);$fd->{handle}->fdopen(defined($fileno)? $saved{$fileno}|| $open_as : $open_as,$fd->{mode})}unless ($^O eq 'MSWin32'){require Fcntl;for$fd (@$close_in_child){next unless fileno$fd;fcntl($fd,Fcntl::F_SETFD(),1)or push@errs,"fcntl $fd: $!" unless$saved{fileno$fd}}}my$pid;unless (@errs){if (FORCE_DEBUG_SPAWN){pipe my$r,my$w or die "Pipe failed: $!";$pid=fork;die "Fork failed: $!" unless defined$pid;if (!$pid){{no warnings;exec @_}print$w 0 + $!;close$w;require POSIX;POSIX::_exit(255)}close$w;my$bad=<$r>;if (defined$bad){$!=$bad;undef$pid}}else {$pid=eval {system 1,@_}}push@errs,"IO::Pipe: Can't spawn-NOWAIT: $!" if!$pid || $pid < 0}for$fd (reverse @$fds){$fd->{handle}->fdopen($fd->{tmp_copy},$fd->{mode})}for (values%saved){$_->close or croak "Can't close: $!"}croak join "\n",@errs if@errs;return$pid}1;
package IPC::Open3;use strict;no strict 'refs';our ($VERSION,@ISA,@EXPORT);require Exporter;use Carp;use Symbol qw(gensym qualify);$VERSION='1.20';@ISA=qw(Exporter);@EXPORT=qw(open3);our$Me='open3 (bug)';sub xpipe {pipe $_[0],$_[1]or croak "$Me: pipe($_[0], $_[1]) failed: $!"}sub xopen {open $_[0],$_[1],@_[2..$#_]and return;local $"=', ';carp "$Me: open(@_) failed: $!"}sub xclose {$_[0]=~ /\A=?(\d+)\z/ ? do {my$fh;open($fh,$_[1].'&=' .$1)and close($fh)}: close $_[0]or croak "$Me: close($_[0]) failed: $!"}sub xfileno {return $1 if $_[0]=~ /\A=?(\d+)\z/;return fileno $_[0]}use constant FORCE_DEBUG_SPAWN=>0;use constant DO_SPAWN=>$^O eq 'os2' || $^O eq 'MSWin32' || FORCE_DEBUG_SPAWN;sub _open3 {local$Me=shift;splice @_,0,1,undef if \$_[0]==\undef;splice @_,1,1,undef if \$_[1]==\undef;unless (eval {$_[0]=gensym unless defined $_[0]&& length $_[0];$_[1]=gensym unless defined $_[1]&& length $_[1];1}){$@ =~ s/(?<=value attempted) at .*//s;croak "$Me: $@"}my@handles=({mode=>'<',handle=>\*STDIN },{mode=>'>',handle=>\*STDOUT },{mode=>'>',handle=>\*STDERR },);for (@handles){$_->{parent}=shift;$_->{open_as}=gensym}if (@_ > 1 and $_[0]eq '-'){croak "Arguments don't make sense when the command is '-'"}$handles[2]{parent}||=$handles[1]{parent};$handles[2]{dup_of_out}=$handles[1]{parent}eq $handles[2]{parent};my$package;for (@handles){$_->{dup}=($_->{parent}=~ s/^[<>]&//);if ($_->{parent}!~ /\A=?(\d+)\z/){$package=caller 1 if (!defined$package);$_->{parent}=qualify $_->{parent},$package}next if $_->{dup}or $_->{dup_of_out};if ($_->{mode}eq '<'){xpipe $_->{open_as},$_->{parent}}else {xpipe $_->{parent},$_->{open_as}}}my$kidpid;if (!DO_SPAWN){xpipe my$stat_r,my$stat_w;$kidpid=fork;croak "$Me: fork failed: $!" unless defined$kidpid;if ($kidpid==0){eval {untie*STDIN;untie*STDOUT;untie*STDERR;close$stat_r;require Fcntl;my$flags=fcntl$stat_w,&Fcntl::F_GETFD,0;croak "$Me: fcntl failed: $!" unless$flags;fcntl$stat_w,&Fcntl::F_SETFD,$flags|&Fcntl::FD_CLOEXEC or croak "$Me: fcntl failed: $!";if (!$handles[2]{dup_of_out}&& $handles[2]{dup}&& xfileno($handles[2]{parent})==fileno \*STDOUT){my$tmp=gensym;xopen($tmp,'>&',$handles[2]{parent});$handles[2]{parent}=$tmp}for (@handles){if ($_->{dup_of_out}){xopen \*STDERR,">&STDOUT" if defined fileno STDERR && fileno STDERR!=fileno STDOUT}elsif ($_->{dup}){xopen $_->{handle},$_->{mode}.'&',$_->{parent}if fileno $_->{handle}!=xfileno($_->{parent})}else {xclose $_->{parent},$_->{mode};xopen $_->{handle},$_->{mode}.'&=',fileno $_->{open_as}}}return 1 if ($_[0]eq '-');exec @_ or do {local($")=(" ");croak "$Me: exec of @_ failed: $!"}}and do {close$stat_w;return 0};my$bang=0+$!;my$err=$@;utf8::encode$err if $] >= 5.008;print$stat_w pack('IIa*',$bang,length($err),$err);close$stat_w;eval {require POSIX;POSIX::_exit(255)};exit 255}else {close$stat_w;my$to_read=length(pack('I',0))* 2;my$bytes_read=read($stat_r,my$buf='',$to_read);if ($bytes_read){(my$bang,$to_read)=unpack('II',$buf);read($stat_r,my$err='',$to_read);waitpid$kidpid,0;if ($err){utf8::decode$err if $] >= 5.008}else {$err="$Me: " .($!=$bang)}$!=$bang;die($err)}}}else {my@close;for (@handles){if ($_->{dup_of_out}){$_->{open_as}=$handles[1]{open_as}}elsif ($_->{dup}){$_->{open_as}=$_->{parent}=~ /\A[0-9]+\z/ ? $_->{parent}: \*{$_->{parent}};push@close,$_->{open_as}}else {push@close,\*{$_->{parent}},$_->{open_as}}}require IO::Pipe;$kidpid=eval {spawn_with_handles(\@handles,\@close,@_)};die "$Me: $@" if $@}for (@handles){next if $_->{dup}or $_->{dup_of_out};xclose $_->{open_as},$_->{mode}}xclose$handles[0]{parent},$handles[0]{mode}if$handles[0]{dup};select((select($handles[0]{parent}),$|=1)[0]);$kidpid}sub open3 {if (@_ < 4){local $"=', ';croak "open3(@_): not enough arguments"}return _open3 'open3',@_}sub spawn_with_handles {my$fds=shift;my$close_in_child=shift;my ($fd,%saved,@errs);for$fd (@$fds){$fd->{tmp_copy}=IO::Handle->new_from_fd($fd->{handle},$fd->{mode});$saved{fileno$fd->{handle}}=$fd->{tmp_copy}if$fd->{tmp_copy}}for$fd (@$fds){bless$fd->{handle},'IO::Handle' unless eval {$fd->{handle}->isa('IO::Handle')};my$open_as=$fd->{open_as};my$fileno=fileno($open_as);$fd->{handle}->fdopen(defined($fileno)? $saved{$fileno}|| $open_as : $open_as,$fd->{mode})}unless ($^O eq 'MSWin32'){require Fcntl;for$fd (@$close_in_child){next unless fileno$fd;fcntl($fd,Fcntl::F_SETFD(),1)or push@errs,"fcntl $fd: $!" unless$saved{fileno$fd}}}my$pid;unless (@errs){if (FORCE_DEBUG_SPAWN){pipe my$r,my$w or die "Pipe failed: $!";$pid=fork;die "Fork failed: $!" unless defined$pid;if (!$pid){{no warnings;exec @_}print$w 0 + $!;close$w;require POSIX;POSIX::_exit(255)}close$w;my$bad=<$r>;if (defined$bad){$!=$bad;undef$pid}}else {$pid=eval {system 1,@_}}if($@){push@errs,"IO::Pipe: Can't spawn-NOWAIT: $@"}elsif(!$pid || $pid < 0){push@errs,"IO::Pipe: Can't spawn-NOWAIT: $!"}}for$fd (reverse @$fds){$fd->{handle}->fdopen($fd->{tmp_copy},$fd->{mode})}for (values%saved){$_->close or croak "Can't close: $!"}croak join "\n",@errs if@errs;return$pid}1;
IPC_OPEN3
s/^ //mg for values %fatpacked;

2534
perl/shift-mgr
View File

File diff suppressed because one or more lines are too long

1379
perl/shiftc
View File

File diff suppressed because one or more lines are too long