Nick Craig-Wood
c2050172aa
qingstor: upgrade to v3 SDK and fix listing loop
2019-08-14 16:15:34 +01:00
Nick Craig-Wood
a72ef7ca0e
vendor: update github.com/yunify/qingstor-sdk-go to v3
2019-08-14 16:15:34 +01:00
Nick Craig-Wood
b84cc0cae7
vendor: run go tidy and go vendor
2019-08-14 16:15:34 +01:00
Nick Craig-Wood
93228dfcc9
operations: debug successful hashes as well as failures #3419
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
eb087a3b04
operations: disable multi thread copy for local to local copies #3419
...
...unless --multi-thread-streams has been set explicitly
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
ec8e0a6c58
fstest/mockobject: add SetFs method so it can have a valid Fs() #3419
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
f0e0d6cc3c
fs: add IsLocal feature to identify local backend #3419
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
752d43d6fa
fs: Implement UnWrapObject and UnWrapFs
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
7c146e2618
operations: check transfer hashes when using --size-only mode #3419
...
Before this change we didn't calculate or check hashes of transferred
files if --size-only mode was explicitly set.
This problem was introduced in 20da3e6352
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
f9ceade9b4
operations: don't calculate checksums when using --ignore-checksum #3419
...
Before this change we calculated the checkums when using
--ignore-checksum but ignored them at the end.
Now we don't calculate the checksums at all which is more efficient.
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
ae9c0e56c8
operations: run hashing operations in parallel #3419
...
Before this change for a post copy Hash check we would run the hashes sequentially.
Now we run the hashes in parallel for a useful speedup.
Note that this refactors the hash check in Copy to use the standard
hash checking routine.
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
402aaca7fe
local: don't calculate any hashes by default #3419
...
Before this change, if the caller didn't provide a hint, we would
calculate all hashes for reads and writes.
The new whirlpool hash is particularly expensive and that has become noticeable.
Now we don't calculate any hashes on upload or download unless hints are provided.
This means that some operations may run slower and these will need to be discovered!
It does not affect anything calling operations.Copy which already puts
the corrects hints in.
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
106cf1852d
Add ginvine to contributors
2019-08-14 13:40:15 +01:00
Nick Craig-Wood
50b8f15b5d
Add another email for Laura Hausmann to contributors
2019-08-14 13:40:07 +01:00
ginvine
1e7bc359be
drive: Add error for purge with --drive-trashed-only - fixes #3407
...
Purge should not be used with --drive-trashed-only flag as it leads to
unexpected behavior. After this commit if TrashedOnly option is set to
true, error message is returned.
See also: https://forum.rclone.org/t/drive-trashed-only-weird-occurrence/11066/14
2019-08-14 13:34:52 +01:00
Nick Craig-Wood
23a0332185
config: don't offer hidden values for editing in the config - fixes #3416
2019-08-14 08:40:22 +01:00
buengese
6812844b3d
march: Fix checking sub-directories when using --no-traverse
2019-08-13 19:30:56 +01:00
buengese
3a04d0d1a9
march: rework testcases to better reflect real use
2019-08-13 19:30:56 +01:00
buengese
6f4b86e569
jottacloud: use new api for retrieving internal username - fixes #3434
2019-08-13 17:18:14 +00:00
Laura Hausmann
9aa889bfa2
fichier: fix character encoding for file names, fixes rclone#3298
2019-08-13 16:56:59 +01:00
Nick Craig-Wood
8247c8a6af
rc: add anchor tags to the docs so links are consistent
2019-08-13 11:57:01 +01:00
Nick Craig-Wood
535f5f3c99
rc: fix --loopback with rc/list and others
...
Before this change `rclone rc --loopback` would give the error "bad
JSON".
This was because the output of the `rc/list` command was not serialzed
through JSON.
This serializes it through JSON and fixes that (and probably other)
command.
2019-08-13 11:51:16 +01:00
Nick Craig-Wood
7f7946564d
error: make "bad record MAC" a retriable error - Fixes #3338
...
The error "tls: bad record MAC" is very likely to be caused by
hardware issues. It indicates that a packet got corrupted somewhere.
As a work around, this change treats it as retriable error which
allows the chunk to get retried and the transfer to continue.
2019-08-12 20:37:10 +01:00
Chaitanya
bbb8d43716
rc: (docs) Add new parameters --rc-web-gui and --rc-allow-origin, --rc-web-fetch-url and rc-web-gui-update to documentation.
2019-08-12 19:04:12 +01:00
Nick Craig-Wood
5e0a30509c
http: add --http-headers flag for setting arbitrary headers
2019-08-12 18:04:24 +01:00
Nick Craig-Wood
cd7ca2a320
googlephotos: implement optional features UserInfo and Disconnect
...
As part of rclone's UX review it was required that rclone had a means
of disconnecting from google photos and showing which user is
connected.
2019-08-12 13:49:23 +01:00
Nick Craig-Wood
a808e98fe1
config: add reconnect, userinfo and disconnect subcommands.
...
- reconnect runs through the oauth flow again.
- userinfo shows the connected user info if available
- disconnect revokes the token
2019-08-12 13:49:23 +01:00
Nick Craig-Wood
3ebcb555f4
fs: add optional features UserInfo and Disconnect
2019-08-12 13:49:23 +01:00
Nick Craig-Wood
a1263e70cf
premiumizeme: new backend for premiumize.me - Fixes #3063
2019-08-10 19:17:51 +01:00
Nick Craig-Wood
f47e5220a2
Add Abhinav Sharma to contributors
2019-08-10 17:31:25 +01:00
Abhinav Sharma
4db742dc77
oauthutil: note that the same version is recommended for remote auth
2019-08-10 17:31:08 +01:00
Nick Craig-Wood
3ecbd603ab
rc: move job expire flags to rc to fix initalization problem
...
See: https://forum.rclone.org/t/rc-rc-job-expire-interval-bug/11188
rclone was ignoring the --rc-job-expire-duration and --rc-job-interval
flags. This turned out to be an initialization order problem and was
fixed by moving those flags out of global config into rc config.
2019-08-10 17:12:22 +01:00
Nick Craig-Wood
0693deea1c
rc: fix unmarshalable http.AuthFn in options and put in test for marshalability
2019-08-10 16:22:17 +01:00
Nick Craig-Wood
99eaa76dc8
Add Macavirus to contributors
2019-08-10 14:13:24 +01:00
Macavirus
ba3b0a175e
docs: Add rsync.net stub link to SFTP page
2019-08-10 14:13:15 +01:00
Macavirus
01c0c0b009
docs: Add C14 Cold Storage to homepage and SFTP backend
2019-08-10 14:13:15 +01:00
Nick Craig-Wood
7d85ccb11e
fs/cache: test for fix cached values pointing to files #3424
2019-08-10 08:39:56 +01:00
buengese
0c1eaf1bcb
cache: correctly handle fs.ErrorIsFile in GetFn - fixes #3424
2019-08-09 21:45:46 +00:00
Chaitanya
873e87fc38
rc: WebGUI should check for new update only when rc-web-gui-update is specified or not already downloaded.
...
rc: WebGUI should check for new update only when rc-web-gui-update is specified or not already downloaded.
rc: change permission to 0755 instead of 755 to prevent unexpected behaviour.
2019-08-09 15:14:52 +01:00
Chaitanya
33677ff367
rc: Added command line parameter to control the cross origin resource sharing (CORS) in the rcd. (Security Improvement)
...
rc: Import statements
Fixing the problem with test
2019-08-09 15:14:52 +01:00
Nick Craig-Wood
5195075677
Add Michał Matczuk to contributors
2019-08-08 23:42:03 +01:00
Michał Matczuk
f396550934
backend/local: Avoid polluting page cache when uploading local files to remote backends
...
This patch makes rclone keep linux page cache usage under control when
uploading local files to remote backends. When opening a file it issues
FADV_SEQUENTIAL to configure read ahead strategy. While reading
the file it issues FADV_DONTNEED every 128kB to free page cache from
already consumed pages.
```
fadvise64(5, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(5, "\324\375\251\376\213\361\240\224>\t5E\301\331X\274^\203oA\353\303.2'\206z\177N\27fB"..., 32768) = 32768
read(5, "\361\311\vW!\354_\317hf\276t\307\30L\351\272T\342C\243\370\240\213\355\210\v\221\201\177[\333"..., 32768) = 32768
read(5, ":\371\337Gn\355C\322\334 \253f\373\277\301;\215\n\240\347\305\6N\257\313\4\365\276ANq!"..., 32768) = 32768
read(5, "\312\243\360P\263\242\267H\304\240Y\310\367sT\321\256\6[b\310\224\361\344$Ms\234\5\314\306i"..., 32768) = 32768
fadvise64(5, 0, 131072, POSIX_FADV_DONTNEED) = 0
read(5, "m\251\7a\306\226\366-\v~\"\216\353\342~0\fht\315DK0\236.\\\201!A#\177\320"..., 32768) = 32768
read(5, "\7\324\207,\205\360\376\307\276\254\250\232\21G\323n\255\354\234\257P\322y\3502\37\246\21\334^42"..., 32768) = 32768
read(5, "e{*\225\223R\320\212EG:^\302\377\242\337\10\222J\16A\305\0\353\354\326P\336\357A|-"..., 32768) = 32768
read(5, "n\23XA4*R\352\234\257\364\355Y\204t9T\363\33\357\333\3674\246\221T\360\226\326G\354\374"..., 32768) = 32768
fadvise64(5, 131072, 131072, POSIX_FADV_DONTNEED) = 0
read(5, "SX\331\251}\24\353\37\310#\307|h%\372\34\310\3070YX\250s\2269\242\236\371\302z\357_"..., 32768) = 32768
read(5, "\177\3500\236Y\245\376NIY\177\360p!\337L]\2726\206@\240\246pG\213\254N\274\226\303\357"..., 32768) = 32768
read(5, "\242$*\364\217U\264]\221Y\245\342r\t\253\25Hr\363\263\364\336\322\t\325\325\f\37z\324\201\351"..., 32768) = 32768
read(5, "\2305\242\366\370\203tM\226<\230\25\316(9\25x\2\376\212\346Q\223 \353\225\323\264jf|\216"..., 32768) = 32768
fadvise64(5, 262144, 131072, POSIX_FADV_DONTNEED) = 0
```
Page cache consumption per file can be checked with tools like [pcstat](https://github.com/tobert/pcstat ).
This patch does not have a performance impact. Please find below results
of an experiment comparing local copy of 1GB file with and without this
patch.
With the patch:
```
(mmt/fadvise)$ pcstat 1GB.bin.1
+-----------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|-----------+----------------+------------+-----------+---------|
| 1GB.bin.1 | 1073741824 | 262144 | 0 | 000.000 |
+-----------+----------------+------------+-----------+---------+
(mmt/fadvise)$ taskset -c 0 /usr/bin/time -v ./rclone copy 1GB.bin.1 /var/empty/rclone
Command being timed: "./rclone copy 1GB.bin.1 /var/empty/rclone"
User time (seconds): 13.19
System time (seconds): 1.12
Percent of CPU this job got: 96%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.81
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 27660
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 2212
Voluntary context switches: 5755
Involuntary context switches: 9782
Swaps: 0
File system inputs: 4155264
File system outputs: 2097152
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
(mmt/fadvise)$ pcstat 1GB.bin.1
+-----------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|-----------+----------------+------------+-----------+---------|
| 1GB.bin.1 | 1073741824 | 262144 | 0 | 000.000 |
+-----------+----------------+------------+-----------+---------+
```
Without the patch:
```
(master)$ taskset -c 0 /usr/bin/time -v ./rclone copy 1GB.bin.1 /var/empty/rclone
Command being timed: "./rclone copy 1GB.bin.1 /var/empty/rclone"
User time (seconds): 14.46
System time (seconds): 0.81
Percent of CPU this job got: 93%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:16.41
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 27600
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 2228
Voluntary context switches: 7190
Involuntary context switches: 1980
Swaps: 0
File system inputs: 2097152
File system outputs: 2097152
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
(master)$ pcstat 1GB.bin.1
+-----------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|-----------+----------------+------------+-----------+---------|
| 1GB.bin.1 | 1073741824 | 262144 | 262144 | 100.000 |
+-----------+----------------+------------+-----------+---------+
```
2019-08-08 23:41:52 +01:00
Nick Craig-Wood
6f87267b34
accounting: fix locking in Transfer to avoid deadlock with --progress
...
Before this change, using -P occasionally deadlocked on the transfer
mutex and the stats mutex since they call each other via the progress
printing.
This is fixed by shortening the locking windows and converting the
mutex to a RW mutex.
2019-08-08 15:46:46 +01:00
Nick Craig-Wood
9d1fb2f4e7
Revert "cmd: shorten the locking window when using --progress to avoid deadlock"
...
This reverts commit fdef567da6
2019-08-08 15:19:41 +01:00
Nick Craig-Wood
99b3154abd
Revert "filter: Add BoundedRecursion method"
...
This reverts commit 047f00a411
2019-08-08 14:15:50 +01:00
Nick Craig-Wood
6c38bddf3e
walk: fix listing with filters listing whole remote
...
Prior to this fix, a request such as
rclone lsf -R --include "/dir/**" remote:
Would use ListR which is very inefficient as it lists the whole remote
for one directory.
This changes it to use recursive walking if the filters imply any
directory filtering. So `--include *.jpg` and `--exclude *.jpg` will
still use ListR wheras `--include "/dir/**` will not.
2019-08-08 14:15:50 +01:00
Nick Craig-Wood
a00a0471a8
filter: Add UsesDirectoryFilters method
2019-08-08 14:15:50 +01:00
Nick Craig-Wood
9e81fc343e
swift: fix upload when using no_chunk to return the correct size
...
When using the VFS with swift and --swift-no-chunk, PutStream was
returning objects with size -1 which was causing corrupted transfer
messages.
This was fixed by counting the bytes transferred in a streamed file
and updating the metadata with that.
2019-08-08 12:41:46 +01:00
Nick Craig-Wood
fdef567da6
cmd: shorten the locking window when using --progress to avoid deadlock
...
Before this change, using -P occasionally deadlocked on the progress
mutex and the stats mutex since they call each other.
This is fixed by shortening the locking window in the progress routine
so as not to include the stats calculation.
2019-08-08 12:37:50 +01:00
Nick Craig-Wood
d377842395
vfs: make write without cache more efficient
...
This updates the out of sequence write code to be more efficient using
a conditional lock with a timeout.
2019-08-08 12:37:50 +01:00
