Shift Installation and Configuration ==================================== 1. Deployments Shift consists of four executable components: o shiftc - the Shift client, which is invoked by users and other client instances and must exist on hosts that initiate transfers and hosts on which a transfer is to be parallelized (henceforth called "client hosts") o shift-mgr - the Shift manager, which is invoked by client instances and must exist either on all client hosts or on a host accessible via ssh hostbased or publickey authentication (henceforth called "manager hosts") o shift-aux - the Shift auxiliary utility, which is invoked by client instances and is optional for basic functionality, but must be installed on any hosts accessed as the source or destination of a transfer initiated from a different host (henceforth called "remote hosts") for efficient remote file initialization, higher speed transfers via the built-in fish/fish-tcp protocols, remote acl/xattr preservation, remote lustre striping, and remote verification (if msum from mutil is not installed) o shift-bin - the Shift binary helper, which is an optional component invoked by shiftc and shift-aux to perform file operations that cannot be implemented directly in pure perl and/or cannot be executed efficiently in batches by standard command line utilities. The main consideration for a Shift deployment is deciding which hosts will be the manager hosts. In the simplest case, all client hosts are manager hosts. If there is more than one client host, then the manager must be configured to store its metadata on a shared file system that supports locking. In a more complex multi-user, multi-host environment without a shared file system across all potential client hosts, one or more hosts must be designated as manager hosts. Since the manager is not an active server and is simply a passive executable invoked by clients, the manager can be located on any host with ssh access. These hosts must support either hostbased or publickey authentication, however, since Shift is an automated framework that performs actions without the user present. The manager has a synchronization mechanism allowing multiple hosts without a shared file system to be used for redundancy. 2. Prerequisites 2.1. Required o perl >= 5.8.5 (>= 5.10.1 is required for multi-threading support) o ssh (and sftp) access to manager/remote hosts via hostbased or publickey authentication 2.2. Optional o Data::MessagePack (compiled version) - increase metadata performance (*highly recommended* for very large multi-host transfers) (https://metacpan.org/pod/Data::MessagePack) o IO::Socket::SSL - allows fish-tcp encryption with --secure (https://metacpan.org/pod/IO::Socket::SSL) o bbftp - high speed remote copy (http://doc.in2p3.fr/bbftp) o gnuplot >= 5.0 - allow display of --plot output (http://gnuplot.info) o mcp/msum >= 1.76.7 - high speed local copy/sum (http://mutil.sf.net) o mesh - lightweight single sign-on via ssh publickey authentication (http://mesh.sf.net) o rsync - bandwidth-efficient local/remote copy (http://rsync.samba.org) 2.3. Invoked (standard on most systems or when specific file systems in use) o beegfs-ctl - get beegfs server information o chown - change symlink ownership o crontab - install cron jobs to recover from host/process failures o df - get file system utilization o dm{get,put} - recall and migrate DMF-managed files o fallocate - preallocate files o {get,set}facl - get and set file ACLs o {get,set}fattr - get and set file extended attributes o lfs - get and set Lustre striping o lspci - find 10GE adapters o mount - get file system information o ping - determine network latency o ps - find clients and PBS processes, and determine client CPU load o su - become non-root process to access manager during root transfers o sysctl - determine number of cpus on BSD o touch - change symlink modification time 3. Build (optional - linux only) An optional binary component called "shift-bin" can be built on linux systems to significantly improve the efficiency of some file operations. These operations include equivalents to those provided by posix_fadvise, fallocate, {get,set}facl, {get,set}fattr, and lfs. Development headers for acls and xattrs are required on the system. This utility must be built differently depending on whether the system mounts lustre (and has access to the corresponding liblustreapi library) or not. 3.1. Systems mounting lustre file systems cd c make lustre 3.2. Systems not mounting lustre file systems cd c make nolustre 4. Installation Note that the following shows the exact files needed on each type of host. Since the number of files is small, however, there is minimal penalty to simply installing them all on every host. The installation commands assume that the optional shift-bin component has been built as needed for client and remote hosts. 4.1. Single-user installation Note that the user's home directory is used as the default install prefix in all examples, but can be changed to any other desired location as long as the corresponding bin directory is in the user's path. 4.1.1. Client hosts install -m 700 perl/shiftc ~/bin/ install -m 700 c/shift-bin ~/bin/ install -m 600 doc/shiftc.1 ~/man/man1/ 4.1.2. Manager hosts install -m 700 perl/shift-mgr ~/bin/ install -m 600 etc/shiftrc ~/.shiftrc 4.1.3. Remote hosts (optional but recommended when possible) install -m 700 perl/shift-aux ~/bin/ install -m 700 c/shift-bin ~/bin/ 4.2. Multi-user installation Note that /usr/local is used as the default install prefix in all examples, but can be changed to any other desired location as long as the corresponding bin directory is in the default system path. 4.2.1. Client hosts install -m 755 perl/shiftc /usr/local/bin/ install -m 755 c/shift-bin /usr/local/bin/ install -m 644 doc/shiftc.1 /usr/local/man/man1/ 4.2.2. Manager hosts install -m 755 perl/shift-mgr /usr/local/bin/ install -m 644 etc/shiftrc /etc/ 4.2.3. Remote hosts (optional but recommended when possible) install -m 755 perl/shift-aux /usr/local/bin/ install -m 755 c/shift-bin /usr/local/bin/ 5. Configuration 5.1. Client hosts 5.1.1. ~/.ssh/id_rsa (or similar) If hostbased authentication is not supported by client hosts, manager hosts, and/or remote hosts, publickey authentication must be used. Clients must have access to private keys to access these systems, which may either be referenced explicitly via the --mgr-identity and --identity options for manager and remote hosts, respectively, or be added to an ssh agent on the client(s). Note that the private keys used for these options must not be protected by a passphrase. The use of an ssh agent to access manager or remote hosts may be preferable from a security standpoint but comes with a drop in reliability as any failure of the agent or agent host leaves any associated transfers with no way to recover. 5.1.2. ~/.ssh/authorized_keys If hostbased authentication is not supported to other client hosts for parallelization, publickey authentication must be used. In this case, the public key(s) corresponding to the private key(s) used by clients (those named ~/.ssh/id* loaded into an ssh agent) must be added to the invoking user's authorized_keys file on other client hosts. 5.1.3. ~/bin/shiftc (single-user) or /usr/local/bin/shiftc (multi-user) If the manager hosts differ from the client hosts, the manager host(s) can be hardcoded within the shiftc program in the "site-specific options" section. This allows the client to be used without specifying the --mgr option every time. For example, the line: $opts{"mgr"} = "mgr.example.com"; would be equivalent to specifying the option "--mgr=mgr.example.com". In general, any shiftc command-line option --X can be hardcoded using: $opts{"X"} = "hardcoded_value"; Those familiar with perl can add more complex logic (e.g. choosing which manager host out of a set of hosts will be used for each invocation). 5.2. Manager hosts 5.2.1. ~/.shiftrc (single-user) or /etc/shiftrc (multi-user) All items in the default config file should be reviewed. The only required setting is: user_dir If there are multiple manager hosts, the configured directory must either be a file system shared across all manager hosts that supports file locking or the setting: sync_host must be configured to sync the transfer metadata across two manager hosts. The transport options should definitely be reviewed to enable any higher performance transports that may be available. Higher performance remote transports based on TCP connections (fish-tcp) and SSH connections (fish) are built-in, but require that the shift-aux executable exist on remote hosts. Note that transports can be enabled on a case-by-case basic using the client --local and --remote options. To allow Shift to make parallelization decisions and enable functionality specific to certain remote file systems (e.g. Lustre striping), it is desirable to configure: mount_db with a database of host and file system information from within the local environment. A template for producing this database is provided in the file "etc/shift-mounts.pl". After the items indicated in the script are configured, it can be run once/periodically to initialize/update the file system database. To optimize access to remote hosts on the LAN, the setting: select_hook may be configured to specify how remote hosts should be selected when more than one is available. This may include deciding which hosts are up, which are least loaded, which have the best connectivity to the given client host, etc. A skeleton for a suitable selection hook is provided in the file "etc/shift-select.hook", but must be fleshed out based on site-specific knowledge and/or calls to the local load balancing infrastructure. If not configured, hosts will be chosen randomly after a successful sshd ping test. 5.2.2. ~/.ssh/authorized_keys If hostbased authentication is not supported to manager hosts, publickey authentication must be used. In this case, the public key(s) corresponding to the private key(s) used by clients (either loaded into an ssh agent or specified via --mgr-identity) must be added to the appropriate user's authorized_keys file on manager hosts. This user will either be the invoking user or the one specified by --mgr-user. 5.3. Remote hosts 5.3.1. ~/.ssh/authorized_keys If hostbased authentication is not supported to remote hosts, publickey authentication must be used. In this case, the public key(s) corresponding to the private key(s) used by clients (either loaded into an ssh agent or specified via --identity) must be added to the appropriate user's authorized_keys file on remote hosts. This user will either be the invoking user or the one specified by --user. 6. Usage 6.1. shiftc Client usage is detailed in the man page "doc/shiftc.1", which can be viewed with: nroff -man doc/shiftc.1 |less -r if not already in a manpath directory. Basic usage is drop-in compatible with cp/scp with special consideration for the authentication options --mgr, --mgr-identity, --mgr-user, --identity, and --user. 6.2. shift-mgr Normal users need not invoke shift-mgr directly as all relevant manager functionality is accessed indirectly through the client. Additional capabilities are available for administrators, however, to aid in collecting usage stats and debugging information. Below is a sampling of some of the most useful administrative shift-mgr commands. To see the status of a particular user X's transfers, run: shift-mgr --user=X --status To see the history of a particular user X's transfers, run: shift-mgr --user=X --history To see the status of running transfers across all users, run: shift-mgr --status --state=run To see detailed stats across the transfers of all users including a sampling of error messages, run: shift-mgr --stats This can be added to a cron job (normally at the interval of the "data_expire" setting in shiftrc) with a redirection to a file to periodically save usage stats. To see the metadata associated with a given user X's transfer T, run: shift-mgr --user=X --id=T --meta The metadata is log structured, so it is also possible to view the metadata at any number of steps S back in time using: shift-mgr --user=X --id=T --meta=S