diff --git a/docs/content/drive.md b/docs/content/drive.md index c8956bcd0..c7326e906 100644 --- a/docs/content/drive.md +++ b/docs/content/drive.md @@ -312,6 +312,45 @@ d) Delete this remote y/e/d> y ``` +### --fast-list ### + +This remote supports `--fast-list` which allows you to use fewer +transactions in exchange for more memory. See the [rclone +docs](/docs/#fast-list) for more details. + +It does this by combining multiple `list` calls into a single API request. + +This works by combining many `'%s' in parents` filters into one expression. +To list the contents of directories a, b and c, the the following requests will be send by the regular `List` function: +``` +trashed=false and 'a' in parents +trashed=false and 'b' in parents +trashed=false and 'c' in parents +``` +These can now be combined into a single request: +``` +trashed=false and ('a' in parents or 'b' in parents or 'c' in parents) +``` + +The implementation of `ListR` will put up to 50 `parents` filters into one request. +It will use the `--checkers` value to specify the number of requests to run in parallel. + +In tests, these batch requests were up to 20x faster than the regular method. +Running the following command against different sized folders gives: +``` +rclone lsjson -vv -R --checkers=6 gdrive:folder +``` + +small folder (220 directories, 700 files): + +- without `--fast-list`: 38s +- with `--fast-list`: 10s + +large folder (10600 directories, 39000 files): + +- without `--fast-list`: 22:05 min +- with `--fast-list`: 58s + ### Modified time ### Google drive stores modification times accurate to 1 ms. diff --git a/docs/content/overview.md b/docs/content/overview.md index 5c710bc6f..0e975963e 100644 --- a/docs/content/overview.md +++ b/docs/content/overview.md @@ -134,7 +134,7 @@ operations more efficient. | Dropbox | Yes | Yes | Yes | Yes | No [#575](https://github.com/ncw/rclone/issues/575) | No | Yes | Yes | Yes | | FTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/ncw/rclone/issues/2178) | No | | Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/ncw/rclone/issues/2178) | No | -| Google Drive | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | +| Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | HTTP | No | No | No | No | No | No | No | No [#2178](https://github.com/ncw/rclone/issues/2178) | No | | Hubic | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/ncw/rclone/issues/2178) | Yes | | Mega | Yes | No | Yes | Yes | No | No | No | No [#2178](https://github.com/ncw/rclone/issues/2178) | Yes | diff --git a/fstest/test_all/test_all.go b/fstest/test_all/test_all.go index 3c1cf79f2..680950ddc 100644 --- a/fstest/test_all/test_all.go +++ b/fstest/test_all/test_all.go @@ -45,7 +45,7 @@ var ( { Name: "TestCryptDrive:", SubDir: false, - FastList: false, + FastList: true, }, { Name: "TestCryptSwift:", @@ -55,7 +55,7 @@ var ( { Name: "TestDrive:", SubDir: false, - FastList: false, + FastList: true, }, { Name: "TestDropbox:",