Nick Craig-Wood
25786cafd3
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier
...
- Read the storage class for each object
- Implement SetTier/GetTier
- Check the storage class on the **object** before using SetModTime
This updates the fix in 1a2fb52Fixes #3522
2019-09-14 09:18:55 +01:00
Nick Craig-Wood
23dc313fa5
azureblob: add missing type assertions for GetTier/SetTier
2019-09-14 09:18:55 +01:00
Nick Craig-Wood
1a16849df0
http: fix race introduced in 7982aaf151
2019-09-14 08:48:13 +01:00
Nick Craig-Wood
3b68340eac
http: add --http-no-head to stop rclone doing HEAD in listings #3523
2019-09-14 00:17:39 +01:00
Nick Craig-Wood
7982aaf151
http: HEAD directory entries in parallel to speedup #3523
2019-09-14 00:16:44 +01:00
Nick Craig-Wood
29b4f211ab
gcs: add context to SDK calls #3257
2019-09-09 23:27:07 +01:00
Nick Craig-Wood
bd863f8868
drive: add context to SDK calls #3257
2019-09-09 23:27:07 +01:00
Nick Craig-Wood
66c23723e3
Add context to all http.NewRequest #3257
...
When we drop support for go1.12 we can use http.NewRequestWithContext
2019-09-09 23:27:07 +01:00
Nick Craig-Wood
58a531a203
rest: add context propagation to rest library #3257
...
This fixes up the calling and propagates the contexts for the backends
which use lib/rest.
2019-09-09 23:27:07 +01:00
Ivan Andreev
ba1daea072
mailru: backend for mail.ru
2019-09-09 21:56:16 +01:00
Ivan Andreev
bdcd0b4c64
Add mailru hash (mrhash)
2019-09-09 21:34:15 +01:00
Nick Craig-Wood
ffa1dac10b
build: apply gofmt from go1.13 to change case of number literals
2019-09-05 13:59:06 +01:00
Nick Craig-Wood
20a57aaccb
gcs: fix need for elevated permissions on SetModTime - fixes #3493
...
Before this change we used PATCH on the object to update the metadata.
Apparently this requires the "full_control" scope which Google were
unhappy with in their oauth review.
This changes it to update the metadata by copying the object ontop of
itself (which is the way s3 works). This can be done with normal
permissions.
2019-09-02 09:26:33 +01:00
Nick Craig-Wood
0edbc9578d
googlephotos,onedrive: fix crash on error response - fixes #3491
...
This fixes a crash on the google photos backend when an error is
returned from the rest.Call function.
This turned out to be a mis-understanding of the rest docs so
- improved rest.Call docs
- fixed mis-understanding in google photos backend
- fixed similar mis-understading in onedrive backend
2019-08-28 12:11:03 +01:00
Nick Craig-Wood
73e010aff9
docs: make the config walkthroughs consistent for each backend
2019-08-26 10:47:17 +01:00
Nick Craig-Wood
193c30d570
Review random string/password generation
...
- factor password generation into lib/random.Password
- call from appropriate places
- choose appropriate use of random.String vs random.Password
2019-08-25 11:09:19 +01:00
Cenk Alti
a1ca65bd80
putio: add new backend
2019-08-23 12:11:36 +01:00
Nick Craig-Wood
6f16588123
s3,b2,googlecloudstorage,swift,qingstor,azureblob: fixes after code review #3421
...
- change the interface of listBuckets() removing dir parameter and adding context
- add makeBucket() and use in place of Mkdir("")
- this fixes some corner cases in Copy/Update
- mark all the listed buckets OK in ListR
Thanks to @yparitcher for the review.
2019-08-22 23:06:59 +01:00
Nick Craig-Wood
b619430bcf
qingstor: make all operations work from the root #3421
2019-08-17 10:30:41 +01:00
Nick Craig-Wood
8a0775ce3c
azureblob: make all operations work from the root #3421
2019-08-17 10:30:41 +01:00
Nick Craig-Wood
d8e9b1a67c
gcs: make all operations work from the root #3421
2019-08-17 10:30:41 +01:00
Nick Craig-Wood
e0e0e0c7bd
b2: make all operations work from the root #3421
2019-08-17 10:30:41 +01:00
Nick Craig-Wood
eaaf2ded94
s3: make all operations work from the root #3421
2019-08-17 10:30:41 +01:00
Nick Craig-Wood
eaeef4811f
swift: make all operations work from the root #3421
2019-08-17 10:30:41 +01:00
buengese
acc9dadcdc
jottacloud: refactor configuration and minor cleanup
2019-08-15 11:46:56 +00:00
Michał Matczuk
712f7e38f7
backend/local: fadvise run syscall on a dedicated go routine
...
Before we issued an additional syscall periodically on a hot path.
This patch offloads the fadvise syscall to a dedicated go routine.
2019-08-14 21:01:39 +01:00
Nick Craig-Wood
fa539b9d9b
sftp: save the md5/sha1 command in use to the config file
2019-08-14 20:54:08 +01:00
Nick Craig-Wood
3ea82032e7
sftp: support md5/sha1 with rsync.net #3254
...
rsync.net uses the freebsd equivalent of sha1sum and md5sum so adapt
to that.
2019-08-14 20:54:08 +01:00
Nick Craig-Wood
c2050172aa
qingstor: upgrade to v3 SDK and fix listing loop
2019-08-14 16:15:34 +01:00
Nick Craig-Wood
f0e0d6cc3c
fs: add IsLocal feature to identify local backend #3419
2019-08-14 15:07:38 +01:00
Nick Craig-Wood
402aaca7fe
local: don't calculate any hashes by default #3419
...
Before this change, if the caller didn't provide a hint, we would
calculate all hashes for reads and writes.
The new whirlpool hash is particularly expensive and that has become noticeable.
Now we don't calculate any hashes on upload or download unless hints are provided.
This means that some operations may run slower and these will need to be discovered!
It does not affect anything calling operations.Copy which already puts
the corrects hints in.
2019-08-14 15:07:38 +01:00
ginvine
1e7bc359be
drive: Add error for purge with --drive-trashed-only - fixes #3407
...
Purge should not be used with --drive-trashed-only flag as it leads to
unexpected behavior. After this commit if TrashedOnly option is set to
true, error message is returned.
See also: https://forum.rclone.org/t/drive-trashed-only-weird-occurrence/11066/14
2019-08-14 13:34:52 +01:00
buengese
6f4b86e569
jottacloud: use new api for retrieving internal username - fixes #3434
2019-08-13 17:18:14 +00:00
Laura Hausmann
9aa889bfa2
fichier: fix character encoding for file names, fixes rclone#3298
2019-08-13 16:56:59 +01:00
Nick Craig-Wood
5e0a30509c
http: add --http-headers flag for setting arbitrary headers
2019-08-12 18:04:24 +01:00
Nick Craig-Wood
cd7ca2a320
googlephotos: implement optional features UserInfo and Disconnect
...
As part of rclone's UX review it was required that rclone had a means
of disconnecting from google photos and showing which user is
connected.
2019-08-12 13:49:23 +01:00
Nick Craig-Wood
3ebcb555f4
fs: add optional features UserInfo and Disconnect
2019-08-12 13:49:23 +01:00
Nick Craig-Wood
a1263e70cf
premiumizeme: new backend for premiumize.me - Fixes #3063
2019-08-10 19:17:51 +01:00
Michał Matczuk
f396550934
backend/local: Avoid polluting page cache when uploading local files to remote backends
...
This patch makes rclone keep linux page cache usage under control when
uploading local files to remote backends. When opening a file it issues
FADV_SEQUENTIAL to configure read ahead strategy. While reading
the file it issues FADV_DONTNEED every 128kB to free page cache from
already consumed pages.
```
fadvise64(5, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(5, "\324\375\251\376\213\361\240\224>\t5E\301\331X\274^\203oA\353\303.2'\206z\177N\27fB"..., 32768) = 32768
read(5, "\361\311\vW!\354_\317hf\276t\307\30L\351\272T\342C\243\370\240\213\355\210\v\221\201\177[\333"..., 32768) = 32768
read(5, ":\371\337Gn\355C\322\334 \253f\373\277\301;\215\n\240\347\305\6N\257\313\4\365\276ANq!"..., 32768) = 32768
read(5, "\312\243\360P\263\242\267H\304\240Y\310\367sT\321\256\6[b\310\224\361\344$Ms\234\5\314\306i"..., 32768) = 32768
fadvise64(5, 0, 131072, POSIX_FADV_DONTNEED) = 0
read(5, "m\251\7a\306\226\366-\v~\"\216\353\342~0\fht\315DK0\236.\\\201!A#\177\320"..., 32768) = 32768
read(5, "\7\324\207,\205\360\376\307\276\254\250\232\21G\323n\255\354\234\257P\322y\3502\37\246\21\334^42"..., 32768) = 32768
read(5, "e{*\225\223R\320\212EG:^\302\377\242\337\10\222J\16A\305\0\353\354\326P\336\357A|-"..., 32768) = 32768
read(5, "n\23XA4*R\352\234\257\364\355Y\204t9T\363\33\357\333\3674\246\221T\360\226\326G\354\374"..., 32768) = 32768
fadvise64(5, 131072, 131072, POSIX_FADV_DONTNEED) = 0
read(5, "SX\331\251}\24\353\37\310#\307|h%\372\34\310\3070YX\250s\2269\242\236\371\302z\357_"..., 32768) = 32768
read(5, "\177\3500\236Y\245\376NIY\177\360p!\337L]\2726\206@\240\246pG\213\254N\274\226\303\357"..., 32768) = 32768
read(5, "\242$*\364\217U\264]\221Y\245\342r\t\253\25Hr\363\263\364\336\322\t\325\325\f\37z\324\201\351"..., 32768) = 32768
read(5, "\2305\242\366\370\203tM\226<\230\25\316(9\25x\2\376\212\346Q\223 \353\225\323\264jf|\216"..., 32768) = 32768
fadvise64(5, 262144, 131072, POSIX_FADV_DONTNEED) = 0
```
Page cache consumption per file can be checked with tools like [pcstat](https://github.com/tobert/pcstat ).
This patch does not have a performance impact. Please find below results
of an experiment comparing local copy of 1GB file with and without this
patch.
With the patch:
```
(mmt/fadvise)$ pcstat 1GB.bin.1
+-----------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|-----------+----------------+------------+-----------+---------|
| 1GB.bin.1 | 1073741824 | 262144 | 0 | 000.000 |
+-----------+----------------+------------+-----------+---------+
(mmt/fadvise)$ taskset -c 0 /usr/bin/time -v ./rclone copy 1GB.bin.1 /var/empty/rclone
Command being timed: "./rclone copy 1GB.bin.1 /var/empty/rclone"
User time (seconds): 13.19
System time (seconds): 1.12
Percent of CPU this job got: 96%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.81
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 27660
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 2212
Voluntary context switches: 5755
Involuntary context switches: 9782
Swaps: 0
File system inputs: 4155264
File system outputs: 2097152
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
(mmt/fadvise)$ pcstat 1GB.bin.1
+-----------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|-----------+----------------+------------+-----------+---------|
| 1GB.bin.1 | 1073741824 | 262144 | 0 | 000.000 |
+-----------+----------------+------------+-----------+---------+
```
Without the patch:
```
(master)$ taskset -c 0 /usr/bin/time -v ./rclone copy 1GB.bin.1 /var/empty/rclone
Command being timed: "./rclone copy 1GB.bin.1 /var/empty/rclone"
User time (seconds): 14.46
System time (seconds): 0.81
Percent of CPU this job got: 93%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:16.41
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 27600
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 2228
Voluntary context switches: 7190
Involuntary context switches: 1980
Swaps: 0
File system inputs: 2097152
File system outputs: 2097152
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
(master)$ pcstat 1GB.bin.1
+-----------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|-----------+----------------+------------+-----------+---------|
| 1GB.bin.1 | 1073741824 | 262144 | 262144 | 100.000 |
+-----------+----------------+------------+-----------+---------+
```
2019-08-08 23:41:52 +01:00
Nick Craig-Wood
9e81fc343e
swift: fix upload when using no_chunk to return the correct size
...
When using the VFS with swift and --swift-no-chunk, PutStream was
returning objects with size -1 which was causing corrupted transfer
messages.
This was fixed by counting the bytes transferred in a streamed file
and updating the metadata with that.
2019-08-08 12:41:46 +01:00
Nick Craig-Wood
e502be475a
azureblob/b2/dropbox/gcs/koofr/qingstor/s3: fix 0 length files
...
In 0386d22cc9
2019-08-06 15:18:08 +01:00
Nick Craig-Wood
5065c422b4
lib/random: unify random string generation into random.String
...
This was factored from fstest as we were including the testing
enviroment into the main binary because of it.
This was causing opening the browser to fail because of 8243ff8bc8
2019-08-06 12:44:08 +01:00
Nick Craig-Wood
0be14120e4
swift: use FixRangeOption to fix 0 length files via the VFS
2019-08-03 18:25:44 +01:00
Nick Craig-Wood
629b7eacd8
b2: fix integration tests after accounting changes
...
In 53a1a0e3ef
2019-08-03 13:30:31 +01:00
yparitcher
d3149acc32
b2: link sharing
2019-08-03 13:30:31 +01:00
Nick Craig-Wood
5be968c0ca
drive: update API for teamdrive use - fixes #3348
2019-08-02 16:06:23 +01:00
Nick Craig-Wood
57d5de6fba
build: fix up package paths after repo move
...
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
2019-07-28 18:47:38 +01:00
Aleksandar Jankovic
53a1a0e3ef
accounting: add reference to completed transfers
...
Add core/transferred call that lists completed transfers and their
status.
2019-07-28 14:48:19 +01:00
Aleksandar Jankovic
8243ff8bc8
accounting: isolate stats to groups
...
Introduce stats groups that will isolate accounting for logically
different transferring operations. That way multiple accounting
operations can be done in parallel without interfering with each other
stats.
Using groups is optional. There is dedicated global stats that will be
used by default if no group is specified. This is operating mode for CLI
usage which is just fire and forget operation.
For running rclone as rc http server each request will create it's own
group. Also there is an option to specify your own group.
2019-07-28 14:48:19 +01:00
yparitcher
ccc416e62b
b2: Fix link sharing #3314
2019-07-28 11:47:31 +01:00
