Commit Graph

129 Commits

Author SHA1 Message Date
Nick Craig-Wood
870c58f7f8 sftp: fail soft with a debug on hash failure #1474
If md5sum/sha1sum fails we debug what it outputed on stderr and return
an empty hash indicating we didn't have a hash, rather than
hash.ErrUnsupported indicating that we don't support this hash type.

This fixes lots of ERROR messages for sftp and synology NAS which,
while it supports md5sum the SFTP paths and the SSH paths are
different so md5sum doesn't work.

We also stop disabling md5sum/sha1sum on errors since typically Hashes
is only checked at the start of a sync run and isn't expected to
change dynamically.
2018-04-21 09:02:53 +01:00
Nick Craig-Wood
311a962011 s3: Look in S3 named profile files for credentials - fixes #2243 2018-04-21 09:00:20 +01:00
Nick Craig-Wood
da7a77ef2e ftp: Fix no error on listing non-existent directory 2018-04-20 23:22:46 +01:00
hensur
8fe3037301 webdav: support SharePoint cookie authentication
This enables the use of the SharePoint webdav endpoint provided by
OneDrive for Business or Office365 Education Accounts. It enables
unverified accounts to be accessed with rclone via webdav as it isn't
possible through the normal onedrive backend.

This integrates the https://github.com/hensur/onedrive-cookie-test
package to fetch the required cookies to authorize against the
SharePoint webdav endpoint.
2018-04-20 21:43:54 +01:00
Nick Craig-Wood
dc59836021 webdav: strip leading and trailing / off root - fixes #2257 2018-04-20 21:43:54 +01:00
Nick Craig-Wood
1a3fb21a77 onedrive: add QuickXorHash support for OneDrive for business - fixes #2262 2018-04-20 21:03:03 +01:00
Nick Craig-Wood
57a5b72d60 onedrive: implement quickXorHash algorithm #2262 2018-04-20 21:02:37 +01:00
Chris Redekop
a35e62e15c s3: Add an option to disable checksum uploading - fixes #2213 2018-04-20 20:51:12 +01:00
Michael G. Noll
d1ca8b8959 sftp: update docs to match code, fix typos and clarify disable_hashcheck prompt 2018-04-20 20:49:49 +01:00
Nick Craig-Wood
a0c65deca8 box: Parse file/directory size as a floating point number
Very large directories can have their sizes returned as floating point
numbers, eg `1.0034576985781e+14` from the box API.

Before this change this would fail to parse as an int64.

This change parses the size as a float64 instead which will be
perfectly accurate for sizes up to 2**56 which is about 9 PB.

It is unknown whether box themselves use a float64 as an intermediate
representation in the API or not - it seems likely.

Fixes #2261
2018-04-19 21:04:52 +01:00
Nick Craig-Wood
1f255a8567 Add a mega.nz remote #163
Not supported yet:
  * Hash
  * ModTime
  * Server Side Copy

Otherwise fully functional and passing all the tests.
2018-04-18 21:09:54 +01:00
Nick Craig-Wood
1ac6dacf0f about: complete other providers and re-work internals
* Implement about for:
    * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
  * Implement `--json` and `---full` flag for `rclone about`
  * change About interface to return a Usage structure
  * Remove operations.About as it is too thin an interface
  * Implement Integration test

Relates to #1138 and #1564
2018-04-17 21:53:27 +01:00
a-roussos
94e277d759 about: add new command 'about' to get quota info from a remote
Implemented for drive only.

Relates to #1138 and #1564.
2018-04-17 21:50:14 +01:00
Nick Craig-Wood
b83814082b backend/http: if HEAD didn't return Content-Length use -1 as size
This means that the files will be treated as an unknown length and
will download properly.

Fixes #2247
2018-04-16 19:40:02 +01:00
Nick Craig-Wood
3d5106e52b drive: fix DirMove leaving a hardlinked directory behind #2245
This bug was introduced by the v3 API conversion in 07f20dd1fd.

The problem was that dircache.FindPath doesn't work for the root directory.

This adds an internal error for dircache.FindPath being called with
the root directory.  This makes a failing test, which the fix to the
drive backend fixes.

This also improves the DirCache integration test.
2018-04-15 10:12:21 +01:00
Nick Craig-Wood
dc247d21ff s3: add in config for all the supported S3 providers #2140
These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other.

This configures endpoints where known and makes sure config doesn't
appear where it isn't valid where possible.
2018-04-13 16:33:26 +01:00
Giri Badanahatti
acd5d4377e config,s3: hierarchical configuration support #2140
This introduces a method of making provider specific configuration
within a remote.  This is useful particularly in s3.

This commit does the basic configuration in S3 for IBM COS.
2018-04-13 16:05:35 +01:00
Craig Rachel
0e6faa2313 s3: add One Zone Infrequent Access storage class - fixes #2240 2018-04-13 13:36:25 +01:00
Peter Baumgartner
1db68571fd s3,swift: Add --use-server-modtime
`--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
2018-04-13 13:32:17 +01:00
Nick Craig-Wood
ef412c1985 drive: fix misplaced log in dedupe MergeDirs 2018-04-13 13:23:55 +01:00
Nick Craig-Wood
e82452ce9a drive: check Open calls for google error messages
This should also enable Open calls to retry properly
2018-04-11 20:55:58 +01:00
Nick Craig-Wood
d5b2ec32f1 local: add --local-no-check-updated to disable update checks #2206
This disables the `can't copy - source file is being updated` checks.
2018-04-09 15:27:58 +01:00
Nick Craig-Wood
05e32cfcf9 dropbox: Fix crypt+obfuscate on dropbox - fixes #2191
Before this change we lowercased the dropbox root directory.  This was
likely a leftover from when we used to build a dictionary to translate
the cases of dropbox files.  Now with the v2 API we can rely on
dropbox to do that for us, so we no longer need to lowercase the root.

This fixes issues using crypt with name obfuscation on dropbox.
2018-04-09 11:53:41 +01:00
Nick Craig-Wood
0fa700b3cf Make integration tests use go1.7+ nested tests #2154
* Removed generated code and code generator
  * Updated docs on how to write integration tests
  * Tidied up the actual integration tests
2018-04-08 15:04:14 +01:00
Nick Craig-Wood
42f0963bf9 local: retry remove on Windows sharing violation error #2202
Before this change asynchronous closes in cmount could cause sharing
violations under Windows on Remove which manifest themselves
frequently as test failures.

This change lets the Remove be retried on a sharing violation under
Windows.
2018-04-07 17:36:26 +01:00
Nick Craig-Wood
be54fd8f70 Remove builds conditional on go1.7 since that is now guaranteed #2154
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
2018-04-07 11:42:55 +01:00
Nick Craig-Wood
e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood
80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Nick Craig-Wood
c749cf8d99 dropbox: fix repeatedly uploading the same files - fixes #2218
In #2134 and dfd0f4c5a4 some testing
changes got committed by accident which caused this regression.

This patch reverts it to how it was before.
2018-04-06 15:34:56 +01:00
Nick Craig-Wood
0cb5c4aa73 gcs: detect bucket presence by listing it - fixes #2193
Doing it like this enables the use of a service account that only has
the "Storage Object Admin" role.
2018-04-06 12:45:15 +01:00
Nick Craig-Wood
92c5aa3786 s3: add --s3-chunk-size option - fixes #2203 2018-04-05 15:40:08 +01:00
Nick Craig-Wood
fbe1c7f1ea dropbox: remove unused code 2018-04-05 15:23:23 +01:00
Nick Craig-Wood
c4531daa43 local: work on spurious "can't copy - source file is being updated" errors #2206
Update all the time comparisons to use time.Time.Equal instead of ==

Improve the logging for that error so we can see exactly what has changed
2018-04-05 14:57:30 +01:00
remusb
6e11a25df5 cache: flush the memory cache after close 2018-04-04 23:25:53 +03:00
remusb
1dea99ab20 cache: purge file data on notification 2018-04-03 23:24:45 +03:00
remusb
e42cee5e02 cache: always forget parent dir for notifications - for #2117 2018-03-31 12:44:09 +03:00
Stefan
a8267d1628
link: allow creating public link to files and folders - closes #1562 2018-03-29 09:10:19 +02:00
Nick Craig-Wood
9df266a6b4 onedrive: Fix socket leak in multipart session upload
This had gone unnoticed until recently when we changed to uploading
all files with a multipart session.
2018-03-28 21:03:19 +01:00
Stefan Breunig
4d553ef701 drive: when initialized with a filepath, optional features used incorrect root path – see #2182 2018-03-28 20:33:39 +02:00
remusb
b52e34ef5e cache: add info log on notification - for #2150 2018-03-23 22:41:01 +02:00
remusb
0ed0d9a7bc cache: integrate with Plex websocket 2018-03-22 21:21:03 +02:00
seuffert
d9c13bff83 add rc cache/stats 2018-03-22 21:16:16 +02:00
Nick Craig-Wood
5ba5be9b37 gcs: ignore zero length directory markers at the root too 2018-03-21 20:10:00 +00:00
Nick Craig-Wood
e9a2cbec37 s3: ignore zero length directory markers at the root too 2018-03-21 20:09:37 +00:00
Nick Craig-Wood
f6020f1308 gcs: ignore zero length directory markers 2018-03-19 17:42:27 +00:00
Nick Craig-Wood
a46f2a9eb7 s3: ignore zero length directory markers - fixes #1621 2018-03-19 17:41:46 +00:00
Nick Craig-Wood
911a78ce6d sftp: require go1.8+ after github.com/pkg/sftp update 2018-03-19 16:37:40 +00:00
remusb
4924ac2f17 cache: reduce log level for plex api - for #2102 2018-03-17 11:57:36 +02:00
Nick Craig-Wood
d4cca8d9f9 onedrive: fix upload of zero length files #1716
Unfortunately multi part upload can't upload zero length files so
bring back the single part upload for zero length files only.

This was broken when we made all uploads multipart uploads.
2018-03-17 09:48:28 +00:00
Oliver Heyme
645cf5ec0f onedrive: fix wrong upload endpoint and createDate #1716
This fixes the problem introduced by 7f744033d8
2018-03-16 19:18:51 +00:00