rclone/backend
Thomas Kriechbaumer 584e705c0c s3: introduce list_chunk option for bucket listing
The S3 ListObject API returns paginated bucket listings, with
"MaxKeys" items for each GET call.

The default value is 1000 entries, but for buckets with millions of
objects it might make sense to request more elements per request, if
the backend supports it. This commit adds a "list_chunk" option for
the user to specify a lower or higher value.

This commit does not add safe guards around this value - if a user
decides to request a too large list, it might result in connection
timeouts (on the server or client).

In AWS S3, there is a fixed limit of 1000, some other services might
have one too.  In Ceph, this can be configured in RadosGW.
2020-01-02 12:15:01 +00:00
..
alias build: fix up package paths after repo move 2019-07-28 18:47:38 +01:00
all New backend for Citrix Sharefile - Fixes #1543 2019-09-30 12:28:33 +01:00
amazonclouddrive amazonclouddrive: use lib/encoder 2019-09-30 22:00:24 +01:00
azureblob azureblob: disable logging to the Windows event log 2019-10-21 11:50:31 +01:00
b2 b2: remove unverified: prefix on sha1 - fixes #3654 2019-10-23 08:41:56 +01:00
box box: use lib/encoder 2019-09-30 14:05:49 +01:00
cache cache: follow move of upstream library github.com/coreos/bbolt github.com/etcd-io/bbolt 2019-11-14 21:51:34 +00:00
chunker chunker: reduce length of temporary suffix 2019-12-09 16:56:32 +00:00
crypt config: give config questions default values - fixes #3672 2019-11-05 11:53:44 +00:00
drive drive: make sure invalid auth for teamdrives always reports an error 2019-11-28 10:51:17 +00:00
dropbox hash: fix hash names for DropboxHash and CRC-32 2019-10-30 12:20:10 +00:00
fichier fichier: fix accessing files > 2GB on 32 bit systems - fixes #3581 2019-10-01 16:03:49 +01:00
ftp ftp: allow disabling EPSV mode 2019-10-10 21:00:41 +01:00
googlecloudstorage googlecloudstorage: use lib/encoder 2019-09-30 22:00:24 +01:00
googlephotos rest: add context propagation to rest library #3257 2019-09-09 23:27:07 +01:00
http http: fix race introduced in 7982aaf151 2019-09-14 08:48:13 +01:00
hubic Add context to all http.NewRequest #3257 2019-09-09 23:27:07 +01:00
jottacloud jottacloud: add URL to generate Login Token to config wizard 2019-11-28 10:03:48 +00:00
koofr koofr: use rclone HTTP client. 2019-11-14 11:36:44 +00:00
local local: make source file being updated errors be NoLowLevelRetry errors #3777 2019-12-06 10:54:03 +00:00
mailru mailru: comment out some debugging statements 2019-10-02 20:10:01 +01:00
mega mega: fix after lib/encoder change 2019-10-02 12:41:52 +01:00
onedrive onedrive: no trailing slash reading metadata... 2019-10-29 13:33:11 +00:00
opendrive opendrive: use lib/encoder 2019-09-30 14:05:49 +01:00
pcloud pcloud: use lib/encoder 2019-09-30 22:00:24 +01:00
premiumizeme premiumizeme: use lib/encoder 2019-10-02 11:34:08 +01:00
putio putio: add ability to resume uploads 2019-10-14 20:01:16 +01:00
qingstor vendor: update all dependencies 2019-11-14 21:51:34 +00:00
s3 s3: introduce list_chunk option for bucket listing 2020-01-02 12:15:01 +00:00
sftp sftp: add --sftp-skip-links to skip symlinks and non regular files - fixes #3716 2019-11-24 16:10:53 +00:00
sharefile sharefile: use lib/encoder 2019-09-30 22:00:25 +01:00
swift swift: fix parsing of X-Object-Manifest 2019-11-21 13:25:02 +00:00
union docs: make the config walkthroughs consistent for each backend 2019-08-26 10:47:17 +01:00
webdav webdav: fix case of "Bearer" in Authorization: header to agree with RFC 2019-11-27 12:04:31 +00:00
yandex yandex: use lib/encoder 2019-10-05 10:22:43 +01:00