Commit Graph

5091 Commits

Author SHA1 Message Date
Nick Craig-Wood
752997c5e8 Add Yury Stankevich to contributors 2021-01-08 08:58:50 +00:00
Yury Stankevich
71edc75ca6 HDFS (Hadoop Distributed File System) implementation - #42
This includes an HDFS docker image to use with the integration tests.

Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2021-01-07 09:48:51 +00:00
Ivan Andreev
768e4c4735 build: Temporary fix for Windows build errors
Applies a temporary fix similar to https://github.com/grafana/grafana/pull/28557
before go 1.15.6+ fixes https://github.com/golang/go/issues/40795
2021-01-07 09:48:51 +00:00
Nick Craig-Wood
c553ad5158 serve sftp: fix authentication on one connection blocking others - fixes #4882
Before this change, if one connection was authenticating this would
block any others from authenticating.

This was due to ssh.NewServerConn not being called in a go routine
after the Accept call.

This is fixed by running the ssh authentication in a go routine.

Thanks to @FiloSottile for advice on how to fix this.

See: https://github.com/golang/go/issues/43521
2021-01-06 15:34:07 +00:00
Alex Chen
c66b901320
onedrive: (business only) workaround to replace existing file on server-side copy (#4904) 2021-01-06 10:50:37 +08:00
Nick Craig-Wood
dd67a3d5f5 operations: add size if known to skipped items and JSON log - fixes #4624 2021-01-05 19:44:25 +00:00
Nick Craig-Wood
e972f2c98a log: make it easier to add parameters to JSON logging 2021-01-05 19:44:25 +00:00
Nick Craig-Wood
acbcb1ea9d Add Ilyess Bachiri to contributors 2021-01-05 19:44:25 +00:00
Nick Craig-Wood
d4444375ac Add Kerry Su to contributors 2021-01-05 19:44:25 +00:00
Cnly
00bf40a8ef onedrive: fix server-side copy completely disabled on OneDrive for Business
This fixes a little problem in PR #4903, which is a fix for #4342
2021-01-06 02:57:51 +08:00
Ilyess Bachiri
5d1f947f32
docs: fix a typo in the dedupe docs 2021-01-05 15:46:57 +00:00
Alex Chen
b594cb9430
onedrive: fall back to normal copy if server-side copy unavailable (#4903)
Fixes #4342 by:
* Disabling server-side copy if either drive isn't OneDrive Personal
* Falling back to normal copy if server-side copy fails
2021-01-05 21:26:00 +08:00
Kerry Su
add7a35e55 b2: docs for download_url with private buckets
The current authentication scheme works without creating
a public download endpoint for a private bucket as in the B2 official blog.
On the contrary, if the existing authorization header gets duplicated
in the Cloudflare Workers script, one might receive 401 Unauthorized errors.
2021-01-02 11:33:48 +00:00
Nick Craig-Wood
2af7b61fc3 cmd: fix wording for no retries message
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326/
2021-01-02 11:20:02 +00:00
Nick Craig-Wood
cb97c2b0d3 azureblob: fix crash on startup
This was introduced by accidental code deletion in

08b9ede217 azureblob: add support for managed identities
2020-12-31 18:39:09 +00:00
Nick Craig-Wood
35da38e93f operations: fix --immutable error message
This was accidentally changed in

53a1a0e3ef accounting: add reference to completed transfers
2020-12-31 18:16:51 +00:00
Nick Craig-Wood
963c0f28b9 sync: Only print "There was nothing to transfer" if no errors
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
2020-12-31 18:16:51 +00:00
Nick Craig-Wood
b3815dc0c2 sync: fix --immutable errors retrying many times
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
2020-12-31 18:16:51 +00:00
Nick Craig-Wood
8053fc4e16 fs: correct default implementation of fs.CountError 2020-12-31 18:16:51 +00:00
buengese
66c3f2f31f new backend: zoho workdrive - fixes #4533 2020-12-30 17:56:08 +00:00
Nick Craig-Wood
62c9074132 compress: fix icon in docs 2020-12-30 17:56:08 +00:00
Nick Craig-Wood
a854cb9617 webdav: add "Depth: 0" to GET requests to fix bitrix
See: https://forum.rclone.org/t/bitrix24-de-remote-support/21112/
2020-12-30 10:14:50 +00:00
Nick Craig-Wood
fbf9942fe7 Add Ingo Weiss to contributors 2020-12-30 10:14:50 +00:00
Nick Craig-Wood
f425950a52 fs: Always show stats when using --dry-run or --interactive #4624 2020-12-29 21:11:12 +00:00
Ingo Weiss
1d40bc1901 fs: Accumulate stats when using --dry-run
Fixes #4624
2020-12-29 21:11:12 +00:00
Nick Craig-Wood
ba51409c3c sftp: implement keyboard interactive authentication - fixes #4177
Some ssh servers are set up with keyboard interactive authentication
which previously the sftp backkend was ignoring.
2020-12-29 19:48:09 +00:00
Nick Craig-Wood
a64fc05385 Add Benjamin Gustin to contributors 2020-12-29 19:48:09 +00:00
Benjamin Gustin
4d54454900
fs/log: don't compile systemd log integration for non unix systems 2020-12-28 23:07:12 +00:00
Nick Craig-Wood
5601652d65 ncdu: add ! (errror) and . (unreadable) file flags to go with e (empty) 2020-12-28 17:25:46 +00:00
Adam Plánský
b218bc5bed ncdu: add empty folder flag into ncdu browser
Implemented empty folder flag for ncdu browser interface. If there is
empty folder in the list the flag e is prepended before size. If there
is no empty folder this flag is ommited. It has the same behaviour as
original ncdu browser. (https://dev.yorhel.nl/ncdu/man)
2020-12-28 17:25:46 +00:00
Nick Craig-Wood
65eee674b9 webdav: fix Open Range requests to fix 4shared mount
Before this change the webdav backend didn't truncate Range requests
to the size of the object. Most webdav providers are OK with this (it
is RFC compliant), but it causes 4shared to return 500 internal error.

Because Range requests are used in mounting, this meant that mounting
didn't work for 4shared.

This change truncates the Range request to the size of the object.

See: https://forum.rclone.org/t/cant-copy-use-files-on-webdav-mount-4shared-that-have-foreign-characters/21334/
2020-12-28 15:45:40 +00:00
Nick Craig-Wood
72eb74e94a Add Claudio Bantaloukas to contributors 2020-12-28 15:45:40 +00:00
Nick Craig-Wood
6bfec25165 Add Mitsuo Heijo to contributors 2020-12-28 15:45:39 +00:00
Nick Craig-Wood
1c61d51448 Add Brad Ackerman to contributors 2020-12-28 15:45:39 +00:00
Claudio Bantaloukas
f7fe1d766b
cmd/ncdu: highlight read errors instead of aborting - fixes #4014
When a directory cannot be walk-ed because of a permissions error
- or any error for that matter -, ncdu mode keeps track of the error
and highlights directories that could not be read.

Previously, the error would cause ncdu to abort.

Now, directories with unreadable sub-directories are displayed in yellow and
a message warns that the total may be underestimated.

Unreadable directories themselves are displayed in red along with the error message
2020-12-28 14:08:12 +00:00
albertony
55aec19389 mount: docs: add section about windows filesystem permissions 2020-12-28 13:59:34 +00:00
albertony
9db51117dc vfs: docs: add note on os specific option 2020-12-28 13:59:34 +00:00
albertony
a9c9467210 vfs: docs: add note about use of --transfers in section about performance 2020-12-28 13:59:34 +00:00
albertony
f50e15c77c mount: docs: add note about mounted file system size 2020-12-28 13:59:34 +00:00
albertony
e3191d096f mount: just set default options without checking if customized by user, because it will be overridden anyway 2020-12-28 13:59:34 +00:00
albertony
07c40780b3 mount: also detect if uid or gid are set in same option string: -o uid=123,gid=456 2020-12-28 13:59:34 +00:00
albertony
67b82b4a28 mount: docs: update documentation according to new syntax on windows 2020-12-28 13:59:34 +00:00
albertony
5f47e1e034 mount: cleanup os specific option handling and documentation 2020-12-28 13:59:34 +00:00
albertony
e92cb9e8f8 mount: more user friendly mounting as network drive on windows
Add --network-mode option to activate mounting as network drive without having to set volume prefix.
Add support for automatic drive letter assignment (not specific to network drive mounting).
Allow full network share unc path in --volname, which will also implicitely activate network drive mounting.
Allow full network share unc path as mountpoint, which will also implicitely activate network drive mounting, and the specified path will be used as volume prefix and the remote will be mounted on an automatically assigned drive letter instead.
2020-12-28 13:59:34 +00:00
Mitsuo Heijo
9ea990d5a2 azureblob: update azure-storage-blob-go to v0.12.0
See https://github.com/Azure/azure-storage-blob-go/blob/master/ChangeLog.md#version-0120
2020-12-28 13:29:38 +00:00
Brad Ackerman
08b9ede217 azureblob: add support for managed identities
Fixes #3213
2020-12-28 13:23:35 +00:00
Nguyễn Hữu Luân
6342499c47
swift: fix deletion of parts of Static Large Object (SLO)
Before this change, deleting SLO objects could leave the parts of the object behind.
2020-12-28 13:21:11 +00:00
Nick Craig-Wood
f347a198f7 azureblob: delete archive tier blobs before update if --azureblob-archive-tier-delete
Before this change, attempting to update an archive tier blob failed
with a 409 error message:

    409 This operation is not permitted on an archived blob.

This change detects if we are overwriting a blob and either generates
the error (if `--azureblob-archive-tier-delete` is not set):

    can't update archive tier blob without --azureblob-archive-tier-delete

Or deletes the blob first before uploading it again (if
`--azureblob-archive-tier-delete` is set).

Fixes #4819
2020-12-28 12:31:24 +00:00
Nick Craig-Wood
060642ad14 flags: improve error message when reading environment vars #4888
The message now includes the flag name to help the user work out what
is happening.

    Invalid value for environment variable "RCLONE_VERSION" when setting default
    for --version: strconv.ParseBool: parsing "yes": invalid syntax
2020-12-28 12:26:23 +00:00
Nick Craig-Wood
629c0d0f65 serve http: fix serving files of unknown length
Before this change serving files of unknown length were always
returned as 0 length files.

This change serves them correctly, but does not support Range:
requests on them.

See: https://forum.rclone.org/t/serve-http-behavior-when-the-size-is-unknown/21319
2020-12-27 22:01:41 +00:00