mirror of https://github.com/pkolano/shift.git
387 lines
24 KiB
Plaintext
387 lines
24 KiB
Plaintext
CHANGES
|
|
=======
|
|
|
|
* Shift 3.0 (01/09/14)
|
|
- First public release
|
|
- Note that --no-offline is currently enabled by default due to
|
|
DMF corruption issues found when releasing files immediately
|
|
after they are copied
|
|
|
|
* Shift 3.1 (01/13/14)
|
|
- Fixed bad metadata counts during --sync preventing dirs from completing
|
|
- Removed some site-specific config that was mistakenly included in shiftc
|
|
|
|
* Shift 3.2 (02/07/14)
|
|
- Added note about mcp/msum >= 1.76.7 since earlier versions do not
|
|
support all required functionality
|
|
- Added warning message when --no-check option specified
|
|
- Fixed leftover temp files after --status
|
|
- Fixed hung transfers with one but not both of --no-check/--no-preserve
|
|
- Fixed not an array reference exception during some rsync scenarios
|
|
- Fixed use of undefined value exception during some tar scenarios
|
|
|
|
* Shift 3.3 (10/15/14)
|
|
- Added file name to fish error messages
|
|
- Changed final file size checks to reread src size when dst size differs
|
|
- Changed status totals to use total+ instead of - during initialization
|
|
- Changed mkdir and dir chattr ops to be limited by --files setting
|
|
- Fixed tar file renaming when multiple unsplit tar files created via stdin
|
|
- Fixed remote tar extraction due to missing fields in shift-aux tar parsing
|
|
- Fixed parsing of @ in remote addresses (user name still dropped however)
|
|
- Fixed local lustre striping when stripes > 160 for lustre < 2.4
|
|
- Fixed unescape of src for tar link validation
|
|
- Fixed display of ssize in --stats when --verify not used
|
|
- Fixed addition to meta file if --meta=0 specified
|
|
- Fixed reversion of metadata corruption after abrupt termination
|
|
- Fixed truncation of existing file during file corruption recovery
|
|
- Fixed possible truncation of directory prefix during tar creation
|
|
- Fixed writability check of remote files
|
|
- Fixed hang when getting striping of link to fifo (due to lfs bug)
|
|
|
|
* Shift 3.4 (01/07/15)
|
|
- Added support for GNU long names/links during tar creation
|
|
- Added inclusion/exclusion of file names based on regular expressions
|
|
- Added --io[rw] and --net[rw] options to throttle by read/write only
|
|
- Added global throttling to throttle read/writes/both by fs/host/user
|
|
- Added --sync-fast to synchronize files without verifying integrity
|
|
- Added built-in metadata sync mechanism for improved scalability
|
|
- Added estimated time until completion in status
|
|
- Added ability to specify some transport options in manager config
|
|
- Added background spawn of dmget/dmput to avoid intermittent hangs
|
|
- Changed disk throttling based on high/low threshold for suspend/resume
|
|
- Fixed inadvertent tar file renaming when transfer completes with errors
|
|
- Fixed error parsing of mcp/msum based on coreutils 8.x
|
|
- Fixed table of contents truncation during remote tar creation
|
|
|
|
* Shift 4.0 (07/23/15)
|
|
- Note that metadata is not backward compatible with previous versions
|
|
- Added backgroundable, parallelizable, and restartable initialization
|
|
- Added --newer and --older options for incremental backups
|
|
- Added --preallocate option to preallocate files below given sparsity
|
|
- Added --no-verify to disable verification, which is now enabled by default
|
|
- Added preservation of extended attributes during non-tar transfers
|
|
- Added backoff to alternate transports/checksums during retries
|
|
- Added error handling when transfer metadata dir can't be created
|
|
- Added ability to use --state w/o --id for brief status of xfers in state
|
|
- Added situational attribute handling for optimized transfer initialization
|
|
- Added ability to specify default tar split size in manager config
|
|
- Changed handling of embedded modules to use App::FatPacker
|
|
- Changed detailed status to show source file during tar creation
|
|
- Changed intra-source symlink dereferencing due to initialization changes
|
|
(location of dereferenced links may differ depending on traversal order)
|
|
- Changed status output to omit some completed transfers beyond threshold
|
|
- Changed --meta manager option to require --id
|
|
- Fixed verification of files ending in whitespace
|
|
- Fixed time estimate to reflect mgr overhead, parallelism, and chattr speed
|
|
- Fixed sum handling of names containing trailing carriage returns
|
|
- Fixed handling of pathological umasks
|
|
- Fixed overrun of metadata directories when linux subdir limit reached
|
|
- Fixed automatic striping of tar files on lustre
|
|
- Fixed tar extraction of files with trailing spaces
|
|
- Fixed preservation of symlink times
|
|
- Fixed --no-cron propagation across parallel clients
|
|
- Fixed tar header validation of symlinks containing escaped characters
|
|
- Fixed bbftp inability to handle vt character
|
|
- Fixed rsync inability to handle cr/lf characters (due to --files-from)
|
|
- Fixed rsync unexpected rename behavior when one file in --files-from list
|
|
- Fixed fish handling of errors with linefeeds
|
|
- Removed --sync-fast since it is now equivalent to --sync --no-verify
|
|
- Removed file/dir meter during initialization since init now backgrounded
|
|
|
|
* Shift 5.0 (07/12/16)
|
|
- Added better detection of lan transfers in stats
|
|
- Added error handling for malformed tar headers
|
|
- Added --buffer option to adjust buffer size used by underlying transports
|
|
- Added --streams option to adjust tcp streams used by tcp-based transports
|
|
- Added --window option to set tcp window size used by tcp-based transports
|
|
- Added --ports option to set the remote ports used by tcp-based transports
|
|
- Added --threads option to set number of threads used by local transports
|
|
- Added --bandwidth option to set bandwidth for stream/window calculation
|
|
- Added adjustment of tcp window/streams based on b/w heuristics and latency
|
|
- Added ability to set small file breakeven points for local/lan/wan cases
|
|
- Added minimum split setting to prevent file system overload with metadata
|
|
- Added rescan of mesh keys between batches to pick up newly generated keys
|
|
- Added support for bbcp as underlying transport
|
|
- Added support for all remote transports to be used as local transports
|
|
- Added manager setting for lustre default stripe count
|
|
- Added multi-threading of single/multi-file batches to built-in transports
|
|
- Added multi-threading of single/multi-file batches to built-in hashing
|
|
- Added multi-threaded tcp-based remote transport based on fish protocol
|
|
- Added support for --verify-fast during fish gets
|
|
- Added output across transfers of all users when --status invoked as root
|
|
- Added zero-padding to duration and estimated time when --status=pad used
|
|
- Changed status emails to limit length of original command sent
|
|
- Changed brief status so at least one completed transfer is always shown
|
|
- Changed --encrypt to --secure, which also changes ssh cipher/mac selection
|
|
- Changed dmput handling so -r is no longer used in automatic offlining
|
|
- Changed help output into functional units
|
|
- Changed external invocations to eliminate all extra shell processes
|
|
- Changed client selection to use selection hook instead of random policy
|
|
- Changed shift-aux sums so file issues are errors and not bad checksums
|
|
- Changed handling of estimated completion to reflect actual operation rates
|
|
- Changed extraction of tar files to remove relative path components
|
|
- Changed --stats output to omit rows without non-empty values
|
|
- Fixed gridftp support using unbuffer utility to interlace stderr/stdout
|
|
- Fixed umask for root transfers so won't inadvertently expose files
|
|
- Fixed status after --wait, which sometimes did not appear when redirected
|
|
- Fixed getting/setting of acls and xattrs on symlinks
|
|
- Fixed existence check of target path when using openssh 7.x
|
|
- Fixed infinite loop in built-in hashing when source file shrinks
|
|
- Fixed reported rate when operations report in after transfer stopped
|
|
- Fixed improper dst truncation in some non-tar corruption recovery cases
|
|
- Fixed infinite loop when extracting tar files less than 512 bytes
|
|
- Fixed host/process failures due to dmgets on every command line file
|
|
- Fixed --include/--exclude options to handle malformed regular expressions
|
|
- Fixed truncation of built-in local copies when dst larger than src
|
|
- Fixed toc file empty blocks of increasing size during split tar creation
|
|
- Fixed built-in hashing chopping off range when file has backslash/newline
|
|
- Fixed correction of corruption in multiple byte ranges during tar creation
|
|
- Fixed distribution of clients to remote hosts when more clients than hosts
|
|
- Fixed built-in transport detection when user $PATH is empty
|
|
- Fixed sum file rename when no regular files during tar creation
|
|
- Fixed -d with unwritable or trailing slash dst (bug report by J. Otey)
|
|
- Fixed exception in fish protocol when input stream is invalid
|
|
- Fixed transport selection order when first transport not suitable
|
|
- Fixed crontab handling with csh variants
|
|
- Fixed exception when fish input stream is unreadable
|
|
- Fixed mkdir errors during parallelization in some scenarios
|
|
- Fixed sum file rename when transfer w/o regular files grouped with regular
|
|
- Fixed abort due to embedded use of Time::HiRes in some perl versions
|
|
- Removed use of File::Copy for built-in local copies
|
|
|
|
* Shift 5.1 (12/21/17)
|
|
- Added --plot to visualize transfer performance when piped to gnuplot
|
|
- Added support for setting stripe size and stripe pool via --stripe
|
|
- Added code location in fish/fish-tcp invalid protocol returns
|
|
- Added colorized status using --status=color
|
|
- Added total time throttled to --stats output
|
|
- Added --no-recall to prevent automated dmget handling
|
|
- Added ability to prevent restart using no_restart file in mgr meta dir
|
|
- Added disk overrun protection when cp would exceed throttling threshold
|
|
- Added support for wildcard expansion when args piped in via stdin
|
|
- Added check of overwrite of same file at transfer initiation
|
|
- Changed rates in --stats to ignore --sync transfers to prevent skewed max
|
|
- Changed tar preallocation to allocate during first tar file operation
|
|
- Fixed mgr sync when user has many transfers/many clients in one transfer
|
|
- Fixed high estimated completion when few small files copied before large
|
|
- Fixed --status/--stats in overflow dir when no transfers in parent dir
|
|
- Fixed failed auth in remote --wait due to agent killed by child process
|
|
- Fixed error handling in remote --wait to show error output instead of ref
|
|
- Fixed collection of os type and perl version from non-origin hosts
|
|
- Fixed backout from fish-tcp when client does not have threading support
|
|
- Fixed duplication of built-in transport error messages in thread splits
|
|
- Fixed empty file name in detailed status during tar renames
|
|
- Fixed metadata recovery after host/process failures preventing retries
|
|
- Fixed cleanup of temporary files during transfer and at completion
|
|
- Fixed detailed status line wrap when one large file and some smaller files
|
|
- Fixed stall due to repeated timeout when very large dir hierarchy
|
|
- Fixed --stripe forcing nearest power of 2 instead of specified
|
|
- Fixed overspawn of dmget processes causing process failures
|
|
- Fixed incorrect dereferencing of dangling symlinks during tar creation
|
|
- Fixed status of transfers that only have errors in final tar rename
|
|
- Fixed exception when unable to open sum file
|
|
- Fixed zero-length files when copying dangling symlinks to lustre
|
|
- Fixed file system load recording for throttling during sum/cksum
|
|
- Fixed aux/mgr taint sanitization of insecure env variables on some OS's
|
|
- Fixed mgr exit delay and exit code during background sync
|
|
- Fixed tar creation when mount point name differs during parallel find
|
|
- Fixed tar corruption of dir entries with specific (rare) name format
|
|
(IMPORTANT - SEE "BUGS")
|
|
- Removed deprecated options
|
|
|
|
* Shift 5.2 (04/23/18)
|
|
- Added ability to show stats in CSV format using --stats=csv
|
|
- Fixed lfs setstripe calls to use -S instead of -s (lustre >= 2.3.0)
|
|
- Fixed descent into directories matching --exclude during initialization
|
|
- Fixed hash errors when remote source mapped to local file system
|
|
|
|
* Shift 6.0 (01/16/19)
|
|
- Note that metadata is not backward compatible with previous versions
|
|
- Added depth-first file stage processing using --pipeline
|
|
- Added CSV history output using --history=csv
|
|
- Added disablement of email status for states given in --no-mail
|
|
- Added disablement of preservation for attributes given in --no-preserve
|
|
- Added continuous real-time display of status using --monitor
|
|
- Added error handling for manager disk exhaustion
|
|
- Added overwrite of read-only files using --force
|
|
- Added ability to ignore unrecoverable errors using --restart=ignore
|
|
- Added faster get/set of external file attributes using shift-bin
|
|
- Added plotting by client using --plot=client
|
|
- Added alert state for items that must be externally checked/fixed
|
|
- Added encrypted data streams to fish-tcp transport using --secure
|
|
- Added better support for scp-style "user@" syntax
|
|
- Added silent corruption detection for previously completed transfers
|
|
- Added --last-sum option to query silent corruption database
|
|
- Added fadvise of source after transfer when shift-bin available
|
|
- Added per file dynamic striping via perl-based expressions
|
|
- Added manager config for hash algorithm and hash leaf size
|
|
- Added advanced --older/--newer expressions using atime/ctime/mtime
|
|
- Added --doing manager option to see past/present doing logs
|
|
- Added abort during tar creation if first split file already exists
|
|
- Added same file detection across remote hosts
|
|
- Changed --no-check option to --no-sanity
|
|
- Changed manager --meta output to handle deeper data structures
|
|
- Changed manager db_file config setting to mount_db
|
|
- Changed dmget handling to use -a during tar creation to keep files online
|
|
- Changed manager meta file format from base64 to yEnc
|
|
- Changed manager log structured files to seekable compressed format
|
|
- Changed manager doing logs to log structured format
|
|
- Changed HMAC authentication from SHA1 to SHA512 in fish-tcp
|
|
- Changed manager sync mechanism to better support multiple failovers
|
|
- Changed estimated completion to use running avgs instead of elapsed time
|
|
- Fixed mount detection for DMF file systems reported as dmapi
|
|
- Fixed mount detection for lustre file systems with multiple MDS's
|
|
- Fixed overwrite of symlinks when exist at destination
|
|
- Fixed deprecated File::Glob usage on newer perls
|
|
- Fixed Net::Ping invocations during latency measurements on redhat/centos
|
|
- Fixed detection of automounted file systems
|
|
- Fixed hang on PBS-controlled hosts during file system detection
|
|
- Fixed --sync on DMF to copy file instead of summing when time/size match
|
|
- Fixed DMF recalls during file overwrite when using built-in transports
|
|
- Fixed incorrect striping when --stripe=0 given and transport is mcp
|
|
- Fixed warning output in some --status=color scenarios
|
|
- Fixed exception in detailed status when very large file with small files
|
|
- Fixed race condition between check and creation of user metadata dir
|
|
- Fixed tracking of corruption stats during --sync
|
|
- Fixed stalls after host/process failures due to bad metadata recovery
|
|
- Fixed throttling deadlock due to file not counting own preallocation
|
|
- Fixed error detection in some gridftp scenarios
|
|
- Removed manager lustre_default_stripe config setting
|
|
- Removed manager min_split config setting
|
|
- Removed previously allowed inverse options --preserve and --verify
|
|
|
|
* Shift 6.1 (02/07/19)
|
|
- Added manager config to globally disable use of cron for restarts
|
|
- Fixed corruption tracking causing negative sizes/rates in --status
|
|
- Fixed inadvertent call of test shift-mgr during mgr synchronization
|
|
- Removed warning message when --no-cron specified
|
|
|
|
* Shift 6.2 (06/24/19)
|
|
- Added multi-threading of directory traversal
|
|
- Added multi-threading of attr preservation when external programs invoked
|
|
- Added --interval to dynamically adjust batch size around fixed frequency
|
|
- Added use of gzip index files to increase metadata read performance
|
|
- Added --no-silent to disable silent corruption detection
|
|
- Added ability to escape multiple lines over stdin in shift-aux
|
|
- Added syntax checking of numeric options that can take units
|
|
- Changed --files and --size to be minimums instead of absolutes
|
|
- Changed time remaining to more accurately reflect manager overhead
|
|
- Fixed remote host selection that was running per file instead of per batch
|
|
- Fixed detection of operations already performed by another client
|
|
- Fixed slow processing of doing logs due to small backward read size
|
|
- Fixed local/remote fadvise that had old initial implementation
|
|
- Fixed --no-recall handling of remote files
|
|
- Fixed possible exceptions during some thread joins
|
|
- Fixed Net::Ping exceptions when Time::HiRes not available
|
|
- Fixed exception when stat of remote file fails
|
|
- Fixed use of setfacl when shift-bin available
|
|
- Fixed run state when --restart=ignore ignores final operations
|
|
- Fixed manager --doing=n so it retrieves nth doing in past
|
|
- Fixed output of error table in stats due to tainted file names
|
|
- Fixed option abbreviation so deprecated options don't invoke other options
|
|
- Fixed double lustre striping during attribute preservation stage
|
|
- Fixed high memory utilization during traversal of very large directories
|
|
- Fixed stalls due to inadvertent newlines left in logs after restarts
|
|
- Fixed preservation of lustre striping due to bad conditional
|
|
|
|
* Shift 6.3 (09/18/19)
|
|
- Fixed erroneous crontab entries for extra clients when --clients > 1
|
|
- Fixed slash prepended to relative paths during tar creation
|
|
- Fixed Tie::DB_FileLock exceptions that can abort manager updates
|
|
- Fixed tainted file names preventing manager synchronization
|
|
- Fixed missing lock during writes of user mount db
|
|
- Fixed use of dequeue_timed preventing directory traversal in older perls
|
|
- Fixed double locks causing deadlock during manager synchronization
|
|
- Fixed thread race condition resulting in zombie shift-aux chattr processes
|
|
- Fixed rounding of lustre stripe count causing allocated stripes off by 1
|
|
- Fixed potential noop in shift-aux chattr when thread creation fails
|
|
- Fixed stalls due to deletion of local mount information during restarts
|
|
- Fixed asymmetric fish-tcp performance during some wan scenarios
|
|
- Fixed potential duplicate error messages in various transports
|
|
- Fixed duplication of corrupt byte ranges resulting in negative rates/sizes
|
|
- Fixed infinite retry of corruption rectification when no progress made
|
|
- Fixed missing write error handling in fish and fish-tcp transports
|
|
- Fixed metadata counts during tar creation renames with --restart=ignore
|
|
- Fixed bad newline handling in shift-aux escape/unescape commands
|
|
|
|
* Shift 7.0 (10/04/19)
|
|
- Note that metadata is not backward compatible with previous versions
|
|
- Fixed vulnerability in root --stats/--status due to use of Storable
|
|
when metadata directly accessible by users (bug report by J. Neff)
|
|
- Fixed race condition causing permission denied during some remote mkdirs
|
|
- Fixed built-in checksums when thread creation fails
|
|
- Fixed chattrs when thread creation fails
|
|
- Fixed leftover shift-bin processes due to unneeded chattr thread spawns
|
|
- Fixed exception when setting binmode on remote file handles
|
|
- Fixed local dmget calls during traversal when source is not local
|
|
- Fixed duplicated source tar lines in dmget input files
|
|
- Fixed deceptive thread creation errors during fish-tcp initialization
|
|
|
|
* Shift 7.1 (03/05/21)
|
|
- Note that the shiftc included with 7.0 was the wrong version (still 6.3)
|
|
- Added latency collection on windows systems
|
|
- Added tcp buffer size collection on osx
|
|
- Added periodic latency measurement to track network issues
|
|
- Added support of \Q metacharacter in --include/--exclude
|
|
- Added manager setting for maximum files per batch
|
|
- Added manager setting for time between latency measurement
|
|
- Added origin IP address to transfer metadata
|
|
- Added save of manager exceptions to associated user debug file
|
|
- Added --plot by file system
|
|
- Changed tracking of built-in operations for stats output
|
|
- Changed file metadata operations to report metadata rate
|
|
- Changed corruption tracking to not count file sizes differing
|
|
- Fixed globs while locating ssh agents to reduce system audit messages
|
|
- Fixed potential for multiple simultaneous keepalive invocations
|
|
- Fixed ephemeral port usage during remote bbftpd invocation
|
|
- Fixed missing handling of --no-recall in some scenarios
|
|
- Fixed accuracy of rsync rates during partial file transfers
|
|
- Fixed failure to detect completion properly in some tar scenarios
|
|
- Fixed parallelization due to overwrite of per user mount metadata
|
|
- Fixed manager exception during read of corrupt/incomplete gzi files
|
|
- Fixed handling of manager disk exhaustion to prevent finalized metadata
|
|
- Fixed potential divide by zero exceptions during throttling
|
|
- Fixed lustre handling in shift-bin to current lustre API
|
|
|
|
* Shift 8.0 (09/16/21)
|
|
- Note that metadata is not backward compatible with previous versions
|
|
- Added plots by tool, subnet, and batch size
|
|
- Added detection of BeeGFS file systems
|
|
- Changed plots from line to heatmap for improved scalability
|
|
- Changed processing so batches only contain single operation type
|
|
- Changed monitoring to support manager hosts across shared file system
|
|
- Changed naming of built-in tools in --stats and new --plot by tool
|
|
- Fixed detection of DMF file systems when using xdsm mount option
|
|
- Fixed detection of GPFS file systems when mmlsmgr not user-accessible
|
|
- Fixed --wait status retrieval when integrated with Mesh framework
|
|
- Fixed shift-bin striping with lustre progressive layouts when --stripe=0
|
|
- Fixed use of XS Data::MessagePack when embedded pure perl version differs
|
|
- Fixed scalability of built-in yEnc encoding/decoding
|
|
- Fixed unnecessary metadata traversal when showing detailed status
|
|
- Fixed application of lustre striping expression to directories
|
|
- Removed bbcp and gridftp support since reported fatal bugs never fixed
|
|
|
|
* Shift 8.1 (01/08/24)
|
|
- Added silent corruption database search using --search with --last-sum
|
|
- Added custom lustre striping of directories using DR in expressions
|
|
- Added alternative metadata locking for broken NFSv3 locks
|
|
- Added debugging output when manager unable to map local/remote files
|
|
- Changed embedded mesh handling to latest mesh version
|
|
- Fixed fish-tcp --secure SSL handling with more recent IO::Socket::SSL
|
|
- Fixed mount collection when file server is an alias to many IPs
|
|
- Fixed handling of @kfi forms in lustre mount collection
|
|
- Fixed remote mount point in GPFS mount collection
|
|
- Fixed mcp default stripe handling overriding --stripe=0
|
|
- Fixed handling of --mgr so no longer always overrides with default
|
|
- Fixed inherited retry count of operations in run state during restart
|
|
- Fixed clean of files associated with old monitoring processes
|
|
- Fixed exception during silent corruption processing of some tar operations
|
|
- Fixed escape of commas in email command line
|
|
- Fixed synchronization of monitor files between managers
|
|
- Fixed error handling of manager invocations
|
|
|
|
* Shift 8.2 (03/15/24)
|
|
- Fixed collection and application of default acls in shift-bin
|
|
- Added additional documentation in shiftc
|
|
|