github-mirror/rclone
2
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-12-11 19:53:46 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
669 Commits 179 Branches 114 Tags 347 MiB
1a87b69376
Commit Graph

669 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
klauspost
345c98ed62 Update to AWS SDK 0.10.0 
Tested with S3 and Dreamhost

Here is a link to the release notes:

http://aws.amazon.com/releasenotes/5476699172355228
 2015-11-02 19:52:11 +00:00
klauspost
b872ff0237 Add option to disable server certificate verification. 
The option name mirrors the 'wget' option (also `--no-check-certificate`). The cURL equivalent is called `--insecure`, which is a bit unclear.

Put in the "developers" section in documentation with proper warnings.

Fixes #168
 2015-10-29 16:42:25 +01:00
Nick Craig-Wood
1b95718460 Fix typos in filter docs and unit test assertions 2015-10-20 09:16:47 +01:00
klauspost
6a3580c556 Show status of master branch, so it doesn't show the status of the last pushed branch, 2015-10-19 18:56:03 +01:00
klauspost
16c9fba5de Fix tests failing on Windows. 
* ":" is kept when part of a drive.
  * Create tests.
  * Fix test framework.
 2015-10-19 17:36:15 +01:00
Nick Craig-Wood
4e952af614 Allow spaces in remotes and check remote names for validity at creation time - fixes #171 2015-10-12 17:54:09 +01:00
Klaus Post
6344c3051c Add async readahead buffer 
This adds an async read buffer of 4x4MB when copying files >10MB.

This fixes #164 and reduces the number of IO operations for copy/move.
 2015-10-12 08:30:27 +01:00
klauspost
ab9f521cbd Allow '&' and disallow ':' in Windows filenames. 
Fixes #161
 2015-10-05 11:04:25 +02:00
Nick Craig-Wood
3a900e5bb7 Version v1.23 2015-10-03 16:24:07 +01:00
Nick Craig-Wood
b4d7741611 Improve output of --dump-headers 2015-10-03 16:04:51 +01:00
Nick Craig-Wood
95fd79faf9 swift: use Content-Length on uploads - fixes #125 2015-10-03 16:04:51 +01:00
Nick Craig-Wood
b79dc01016 swift: stop chunked operations logging "Failed to read info: Object Not Found" 2015-10-03 16:04:51 +01:00
Nick Craig-Wood
bf562d7373 Tweak wording on client/secret ids in acd, gcs and drive - fixes #155 2015-10-03 16:04:45 +01:00
Nick Craig-Wood
2e9f2ea3d3 oauthutil: fix headless config for drive and gcs - fixes #158 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
177dbbc29a Implement rclone size for measuring remotes - fixes #153 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
4712043e26 Fix race condition in pacer tests - fixes #156 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
852acd5e4e oauthutil: tell the user they should try again if the webserver method failed 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
9f1daabb2c s3: allow anonymous access to public repositories - fixes #154 
When setting up the remote, leave both the access key and secret key
blank.
 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
938dd24cc9 Fix typo 2015-09-28 22:51:33 +01:00
Nick Craig-Wood
57aad81b68 Version v1.22 2015-09-28 19:38:20 +01:00
Nick Craig-Wood
a91bcaaeb0 Implement rsync like include and exclude - fixes #27 
* Implement include/exclude
  * Implement rsync compatible file globbing
  * Implement command line filtering flags
    * --delete-excluded - Delete files on dest excluded from sync
    * --filter - Add a file-filtering rule
    * --filter-from - Read filtering patterns from a file
    * --exclude - Exclude files matching pattern
    * --exclude-from - Read exclude patterns from file
    * --include - Include files matching pattern
    * --include-from - Read include patterns from file
    * --files-from - Read list of source-file nam
    * --min-size - Don't transfer any file smaller than this in k or suffix k|M|G
    * --max-size - Don't transfer any file larger than this in k or suffix k|M|G
  * Document
 2015-09-28 19:18:21 +01:00
Nick Craig-Wood
d04c21b198 Add Sergey Tolmachev to contributors 2015-09-28 19:12:35 +01:00
Nick Craig-Wood
4a0a42c2f1 swift: large file upload fixes 
* Read metadata in file listing for 0 length files to fix syncs
  * Ignore non-existent files in isManifestFile to fix errors on copy
  * remove nsToSwiftFloatString - experiments with the swift program
    indicate that it puts a variable number of points after the
    decimal, so might as well use the one in the swift library.
  * Make sure segments get deleted properly when move from segmented
    to non segmented and vice versa
  * Use internal list routine to detect errors on listing
  * Remove the _segments container if possible
  * Remove manifest first when deleting
 2015-09-26 17:58:04 +01:00
Sergey Tolmachev
cc7b9af50e swift: support files > 5GB - fixes #46 
* Write segments to ..._segments container
  * Choice of container and segment names compatible with swift tool
    * See http://docs.openstack.org/developer/swift/overview_large_objects.html
  * Controlled by command line flag --swift-chunk-size
  * Segments removed on delete
 2015-09-26 13:03:58 +01:00
Nick Craig-Wood
68fef49c55 swift: fetch headers as the only source of metadata 2015-09-26 13:03:58 +01:00
Nick Craig-Wood
5d4b149884 Version v1.21 2015-09-22 21:05:11 +01:00
Nick Craig-Wood
5f20ae707d Make lsl output times in localtime and fix tests - fixes #141 2015-09-22 19:04:12 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2ed158aba3 Fixes from go vet and errcheck 2015-09-22 07:31:12 +01:00
Nick Craig-Wood
05050d53ad Implement compliant pacing scheme for Amazon Cloud Drive 
* Implement switchable pacing algorithm
  * Add tests for pacer
 2015-09-21 21:49:54 +01:00
Nick Craig-Wood
e391311512 gofmt 2015-09-17 18:42:39 +01:00
klauspost
3234c28f7c Minor adjustments to Unc path conversion function. 2015-09-17 18:42:10 +01:00
klauspost
6fbd9cf24b Make Google Drive Directory reads concurrent for increased speed. 
If you use the -v flag then this will show the progress of directory
listing.  Fixes #143.
 2015-09-17 18:42:10 +01:00
klauspost
bc5b63ffef Move ACD Connection-limiter into Pacer. 2015-09-17 18:42:10 +01:00
klauspost
788ef76f1c Show more of the filename and align output. 
Print more of the file name, and make the output aligned, so it is nicer on frequent updates.
 2015-09-17 15:53:45 +02:00
Nick Craig-Wood
0872ec3204 Allow user to override credentials again in drive, gcs and acd - fixes #139 2015-09-16 20:08:40 +01:00
klauspost
0a5870208e Display individual transfer progress 
Improve progress printing by displaying individual file progress, as well
as a moving average speed with ETA. Example output:

2015/09/15 16:38:21
Transferred:    183599104 Bytes (4646.49 kByte/s)
Errors:                 0
Checks:                 1
Transferred:            0
Elapsed time:       38.5s
Transferring:
 * 01_06_14.mp3: 33% done. avg: 1280.5, cur: 1288.8 kByte/s. ETA: 1m12s
 * 01_12_15.mp3: 33% done. avg: 1002.2, cur:  943.4 kByte/s. ETA: 1m17s
 * 01_13_14.mp3: 48% done. avg: 1456.8, cur: 1425.2 kByte/s. ETA: 39s
 * 01_19_15.mp3: 28% done. avg: 1226.9, cur: 1114.4 kByte/s. ETA: 1m37s
 2015-09-16 19:42:46 +01:00
klauspost
3219334c3e Add "-race" test to travis. 2015-09-16 12:26:28 +02:00
klauspost
79fd662676 Protect concurrent read/writes to pacer. 
Protects all variables in the pacer from concurrent modifications. It is now safe to modify pacer settings while it is running.

I decided to not go for an RWMutex, since all accesses are very short, so the overhead of an RWMutex isn't worth it.

Fixes #138.
 2015-09-16 12:25:55 +02:00
Nick Craig-Wood
34193fd8d9 Version v1.20 2015-09-15 07:42:31 +01:00
Nick Craig-Wood
2203766f77 Retry listing in tests to work around eventual consistency 2015-09-14 21:01:25 +01:00
Nick Craig-Wood
235cbe0e57 amazon cloud drive: retry 409 errors too 2015-09-14 21:00:44 +01:00
klauspost
f50f353b5d local: always use UNC paths on Windows - fixes #124, fixes #130, fixes #90 
* Convert all paths to UNC paths on Windows.
  * Update local filesystem to always use UNC paths.
  * Change tests, so they can work with Windows character replacements.
  * Remove "/" suffix on paths.
  * Always use path/filepath
 2015-09-14 19:58:03 +01:00
Klaus Post
00afe6cc96 ACD: Shorten minimum and maximum sleep. 2015-09-14 18:22:21 +01:00
Klaus Post
dd48e62b7e Limit concurrent directory listings. 
Never run more than "fs.Config.Checkers" directory listings at once.
 2015-09-14 18:11:59 +01:00
Klaus Post
a1a780e847 Read folders in separate goroutines. 
As proposed in the FIXME, read folders in parallel.
This appears to fix "Next token is expired" on very big directories.
The only downside is that this doesn't abort at once if an error is found.
I added some logging, so there is some output for "-v".
 2015-09-14 18:11:59 +01:00
Klaus Post
fa87077211 Retry if a timeout occurs. 
This happens more rarely than 500, but can still be encountered.
 2015-09-14 18:02:35 +01:00
Klaus Post
6ac7145d2d Retry when we get a "500 Internal server error". 
It seems like this happens randomly, and retrying works fine for that.
 2015-09-14 18:02:35 +01:00
Nick Craig-Wood
f1226f19b2 drive, googlecloudstorage: optionally use auto config for the oauth token 2015-09-12 14:17:39 +01:00
Klaus Post
3ecbf2af25 Fix missing link text. 2015-09-12 11:22:17 +02:00