Commit Graph

5846 Commits

Author SHA1 Message Date
Nick Craig-Wood
1e4ef4b4d5 Add Felix Bünemann to contributors 2021-11-03 21:44:09 +00:00
Nick Craig-Wood
8d92f7d697 s3: fallback to ListObject v1 on unsupported providers
This implements a quirks system for providers and notes which
providers we have tested to support ListObjectsV2.

For those providers which don't support ListObjectsV2 we use the
original ListObjects call.
2021-11-03 19:13:50 +00:00
Felix Bünemann
fd56abc5f2 s3: Use ListObjectsV2 for faster listings
Using ListObjectsV2 with a continuation token is about 5-6x faster than
ListObjectsV2 with a marker.
2021-11-03 19:13:50 +00:00
Ivan Andreev
b323bf34e2 sync/test: skip test ConcurrentTruncate on uptobox (take 2)
The test is not applicable to uptobox which can't upload empty files.
The test was not skipped as intended because the direct error was compared.
This fix will compare error Cause because Sync wraps the error.
2021-11-02 19:24:23 +03:00
Ivan Andreev
e78e73eae7 lib/encoder: fix benchmarks
Some day in the past the Slash encode option was added to Onedrive
encoder so it began to encode slashes in file names rather then treat
them as path separators.
This patch adapts benchmark test cases accordingly.

Fixes #5659
2021-11-02 19:23:16 +03:00
Nick Craig-Wood
f51a5eca2e fstests: add encoding test for URL encoded path name #5768
Add an encoding test to make sure backends can deal with a URL encoded
path name. This is a fairly common failing in backends and has been an
intermittent problem with onedrive itself.
2021-11-02 15:59:36 +00:00
albertony
39e2af7974
config: allow dot in remote names (#5606) 2021-11-01 20:50:06 +01:00
Ivan Andreev
b3217adf08 Add Chris Nelson to contributors 2021-11-01 21:24:06 +03:00
Ivan Andreev
074234119a bisync: documentation #5164
Co-authored-by: Chris Nelson <stuff@cjnaz.com>
2021-11-01 21:00:27 +03:00
Ivan Andreev
6210e22ab5 bisync: implementation #5164
Fixes #118

Co-authored-by: Chris Nelson <stuff@cjnaz.com>
2021-11-01 21:00:27 +03:00
Ivan Andreev
940e99a929 bisync: test scenarios #5164
Co-authored-by: Chris Nelson <stuff@cjnaz.com>
2021-11-01 21:00:27 +03:00
Ivan Andreev
79b6866b57 rc: export NewErrParamInvalid #5164 2021-11-01 21:00:27 +03:00
Ivan Andreev
c142e3edcc filter: export GlobToRegexp #5164 2021-11-01 21:00:27 +03:00
Nick Craig-Wood
5c646dff9a Start v1.58.0-DEV development 2021-11-01 16:54:17 +00:00
Nick Craig-Wood
19dfaf7440 docs: fix shortcode rendering on download page 2021-11-01 16:50:52 +00:00
Nick Craig-Wood
169990e270 Version v1.57.0 2021-11-01 15:45:40 +00:00
Nick Craig-Wood
e781bcbba1 Add David Liu to contributors 2021-11-01 15:45:40 +00:00
Nick Craig-Wood
409916b0c5 Add trevyn to contributors 2021-11-01 15:45:40 +00:00
albertony
d9c5be32e7 Add thomae to contributors 2021-11-01 15:10:14 +01:00
thomae
57074be9b3 serve sftp: fix typo 2021-11-01 15:07:13 +01:00
Joda Stößer
bae5c7c81b docs: remove application type "other" from drive.md
The application type "other" is not an option anymore.
2021-11-01 13:15:31 +00:00
albertony
05321f4aef docs/sftp: more detailed explanation of pubkey file and certificate 2021-11-01 13:14:17 +00:00
albertony
c9d7248d85 docs/jottacloud: update description of whitelabel services 2021-11-01 12:57:51 +00:00
albertony
da8f9be84b docs: describe the --human-readable option in more detail 2021-11-01 12:55:52 +00:00
David Liu
b806166147 docs: swift: Update OCI url
Oracle cloud storage now rebranded also as OCI bucket with new entry point
2021-11-01 12:54:23 +00:00
acsfer
20f936c9d4 Add note about S3 compatible services 2021-11-01 12:47:18 +00:00
albertony
91cdaffcc1
docs: add faq section explaining why rclone changes fullwidth characters in file names 2021-11-01 12:46:23 +00:00
trevyn
33bf9b4923 Add mention of Rust bindings for librclone 2021-11-01 12:43:31 +00:00
albertony
b4944f4520 docs/librclone: document that strings are utf8 encoded 2021-11-01 12:39:00 +00:00
albertony
286b152e7b librclone: free strings in python example 2021-11-01 12:36:52 +00:00
Nick Craig-Wood
f7764a0c9d premiumizeme: fix server side directory move after API changes
Apparently moving a directory using the id "0" as the root no longer
works, so this reads the real root ID when it is listed and uses that.

This fixes the DirMove problem.

See: https://forum.rclone.org/t/premiumize-cant-move-files/27169
See: #5734
2021-10-31 19:18:55 +00:00
Nick Craig-Wood
07fcba888c dircache: add SetRootIDAlias to update RootID from FindLeaf 2021-10-31 19:18:55 +00:00
Nick Craig-Wood
af705c754c premiumizeme: fix server side move after API change
See: https://forum.rclone.org/t/premiumize-cant-move-files/27169
See: #5734
2021-10-31 19:18:55 +00:00
Nick Craig-Wood
f85e3209b3 premiumizeme: fix directory listing after API changes
The API doesn't seem to accept a value of "0" any more for the root
directory ID, giving the error "Could not decode folder id".

However omitting it seems to work fine.
2021-10-31 19:18:55 +00:00
Nick Craig-Wood
e77dcb7f52 test_all: remove stray debug 2021-10-31 19:18:55 +00:00
Nick Craig-Wood
4ab842198a Revert "premiumizeme: attempt to fix integration tests"
This reverts commit 1eff0eef7a.

Now that the test account is premium again it is very fast and this is
no longer needed.
2021-10-31 19:18:54 +00:00
albertony
a8059b8a90
docs/mount: add note that to execute programs one must set custom filesystem permissions (#5771) 2021-10-31 00:48:56 +02:00
Nick Craig-Wood
cf2c2792e6 s3: fix corrupted on transfer: sizes differ 0 vs xxxx with Ceph
In this commit, released in 1.56.0 we started reading the size of the
object from the Content-Length header as returned by the GET request
to read the object.

4401d180aa s3: add --s3-no-head-object

However some object storage systems, notably Ceph, don't return a
Content-Length header.

The new code correctly calls the setMetaData function with a nil
pointer to the ContentLength.

However due to this commit from 2014, released in v1.18, the
setMetaData function was not ignoring the size as it should have done.

0da6f24221  s3: use official github.com/aws/aws-sdk-go including multipart upload #101

This commit correctly ignores the content length if not set.

Fixes #5732
2021-10-30 12:01:09 +01:00
Nick Craig-Wood
e6e1c49b58 s3: fix shared_credentials_file auth after reverting incorrect fix #5762
Before this change the `shared_credentials_file` config option was
being ignored.

The correct value is passed into the SDK but it only sets the
credentials in the default provider. Unfortunately we wipe the default
provider in order to install our own chain if env_auth is true.

This patch restores the shared credentials file in the session
options, exactly the same as how we restore the profile.

Original fix:

1605f9e14d s3: Fix shared_credentials_file auth
2021-10-30 11:54:17 +01:00
Nick Craig-Wood
712f9c9760 s3: fix IAM Role for Service Account not working and other auth problems
This patch reverts this commit

1605f9e14d s3: Fix shared_credentials_file auth

It unfortunately had the side effect of making the s3 SDK ignore the
config in our custom chain and use the default provider. This means
that advanced auth was being ignored such as --s3-profile with
role_arn.

Fixes #5468
Fixes #5762
2021-10-30 11:54:17 +01:00
albertony
a238877ad8
docs: note that destination is always a directory (#5766) 2021-10-30 00:30:00 +02:00
Ivan Andreev
70297c3aed sync/test: TestConcurrentTruncate needs empty files - skip on uptobox 2021-10-28 17:04:56 +03:00
Nolan Woods
a074a2b983 lib/http: Fix handling of ssl credentials
Adds a test that makes an actual http and https request against the server
2021-10-27 14:46:10 +03:00
Nick Craig-Wood
00ceeef21c hdfs: wait longer for the server to start up in the integration tests #5734
This needs fixing properly so rclone knows when the server has started
properly.
2021-10-23 22:53:17 +01:00
Nick Craig-Wood
2e81b78486 Add Dmitry Bogatov to contributors 2021-10-23 22:53:17 +01:00
Dmitry Bogatov
bb11803f1f Create direct share link for "koofr" backend
Instead of creating link to web interface, create direct link usable by
curl(1) or wget(1).
2021-10-23 15:00:33 +03:00
Nick Craig-Wood
a542ddf60a hdfs: attempt to make integration tests more reliable #5734
This makes sure the namenode is accepting TCP connections before
starting the integration tests in an attempt to make them more
reliable.
2021-10-22 13:07:48 +01:00
Nick Craig-Wood
257f5d279a filefabric: fix directory move after API change #5734
The API has changed in the directory move call JSON response from
returning a TaskID as a string to returning it as an integer. In other
places it is still returned as a string though.

This patch allows the TaskID to be an integer or a string in the JSON
response and keeps it internally as a string like before.
2021-10-22 12:58:00 +01:00
albertony
4f05ece39e test: fix touchdir test on backends without modtime support 2021-10-22 13:37:34 +02:00
albertony
9c8c0a58b5 touch: fix recursive touch due to recently introduced error ErrorIsDir 2021-10-22 13:37:34 +02:00