2014-04-25 01:11:08 +08:00
|
|
|
% rclone(1) User Manual
|
|
|
|
|
% Nick Craig-Wood
|
2014-07-04 04:56:54 +08:00
|
|
|
% Jul 7, 2014
|
2014-04-25 01:11:08 +08:00
|
|
|
|
2013-06-28 03:00:01 +08:00
|
|
|
Rclone
|
|
|
|
|
======
|
2012-11-19 01:32:31 +08:00
|
|
|
|
2014-04-18 05:39:07 +08:00
|
|
|
[](http://rclone.org/)
|
2014-03-16 22:01:17 +08:00
|
|
|
|
2014-04-27 00:43:41 +08:00
|
|
|
Rclone is a command line program to sync files and directories to and from
|
2013-06-28 03:00:01 +08:00
|
|
|
|
|
|
|
|
* Google Drive
|
2014-03-16 22:01:17 +08:00
|
|
|
* Amazon S3
|
|
|
|
|
* Openstack Swift / Rackspace cloud files / Memset Memstore
|
2014-07-18 03:03:11 +08:00
|
|
|
* Dropbox
|
|
|
|
|
* Google Cloud Storage
|
2013-06-28 03:00:01 +08:00
|
|
|
* The local filesystem
|
2012-11-19 01:32:31 +08:00
|
|
|
|
2014-03-16 01:39:56 +08:00
|
|
|
Features
|
2012-11-19 01:32:31 +08:00
|
|
|
|
2014-03-16 01:39:56 +08:00
|
|
|
* MD5SUMs checked at all times for file integrity
|
|
|
|
|
* Timestamps preserved on files
|
|
|
|
|
* Partial syncs supported on a whole file basis
|
|
|
|
|
* Copy mode to just copy new/changed files
|
|
|
|
|
* Sync mode to make a directory identical
|
|
|
|
|
* Check mode to check all MD5SUMs
|
2014-03-16 22:01:17 +08:00
|
|
|
* Can sync to and from network, eg two different Drive accounts
|
|
|
|
|
|
2014-04-18 05:39:07 +08:00
|
|
|
See the Home page for more documentation and configuration walkthroughs.
|
2014-03-16 22:01:17 +08:00
|
|
|
|
|
|
|
|
* http://rclone.org/
|
2012-11-19 01:32:31 +08:00
|
|
|
|
|
|
|
|
Install
|
|
|
|
|
-------
|
|
|
|
|
|
2013-06-28 03:00:01 +08:00
|
|
|
Rclone is a Go program and comes as a single binary file.
|
2012-11-19 01:32:31 +08:00
|
|
|
|
2014-04-25 01:11:08 +08:00
|
|
|
Download the binary for your OS from
|
2012-11-19 01:32:31 +08:00
|
|
|
|
2014-04-25 01:11:08 +08:00
|
|
|
* http://rclone.org/downloads/
|
2012-11-19 01:32:31 +08:00
|
|
|
|
|
|
|
|
Or alternatively if you have Go installed use
|
|
|
|
|
|
2014-04-25 01:11:08 +08:00
|
|
|
go install github.com/ncw/rclone
|
2012-11-19 01:32:31 +08:00
|
|
|
|
2014-03-16 22:01:17 +08:00
|
|
|
and this will build the binary in `$GOPATH/bin`.
|
|
|
|
|
|
2014-03-16 01:39:56 +08:00
|
|
|
Configure
|
|
|
|
|
---------
|
|
|
|
|
|
|
|
|
|
First you'll need to configure rclone. As the object storage systems
|
|
|
|
|
have quite complicated authentication these are kept in a config file
|
|
|
|
|
`.rclone.conf` in your home directory by default. (You can use the
|
2014-04-27 00:43:41 +08:00
|
|
|
`--config` option to choose a different config file.)
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
The easiest way to make the config is to run rclone with the config
|
|
|
|
|
option, Eg
|
|
|
|
|
|
2014-03-16 08:28:32 +08:00
|
|
|
rclone config
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2012-11-19 01:32:31 +08:00
|
|
|
Usage
|
|
|
|
|
-----
|
|
|
|
|
|
2014-03-16 01:39:56 +08:00
|
|
|
Rclone syncs a directory tree from local to remote.
|
|
|
|
|
|
2014-04-27 00:43:41 +08:00
|
|
|
Its basic syntax is
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Syntax: [options] subcommand <parameters> <parameters...>
|
|
|
|
|
|
2014-03-16 08:28:32 +08:00
|
|
|
See below for how to specify the source and destination paths.
|
|
|
|
|
|
|
|
|
|
Subcommands
|
|
|
|
|
-----------
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone copy source:path dest:path
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Copy the source to the destination. Doesn't transfer
|
|
|
|
|
unchanged files, testing first by modification time then by
|
|
|
|
|
MD5SUM. Doesn't delete files from the destination.
|
|
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone sync source:path dest:path
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Sync the source to the destination. Doesn't transfer
|
|
|
|
|
unchanged files, testing first by modification time then by
|
|
|
|
|
MD5SUM. Deletes any files that exist in source that don't
|
|
|
|
|
exist in destination. Since this can cause data loss, test
|
2014-04-27 00:43:41 +08:00
|
|
|
first with the `--dry-run` flag.
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone ls [remote:path]
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-07-18 03:03:11 +08:00
|
|
|
List all the objects in the the path with sizes.
|
|
|
|
|
|
|
|
|
|
rclone lsl [remote:path]
|
|
|
|
|
|
|
|
|
|
List all the objects in the the path with sizes and timestamps.
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone lsd [remote:path]
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-04-27 00:43:41 +08:00
|
|
|
List all directories/objects/buckets in the the path.
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone mkdir remote:path
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Make the path if it doesn't already exist
|
|
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone rmdir remote:path
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Remove the path. Note that you can't remove a path with
|
|
|
|
|
objects in it, use purge for that.
|
|
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone purge remote:path
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Remove the path and all of its contents.
|
|
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone check source:path dest:path
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Checks the files in the source and destination match. It
|
|
|
|
|
compares sizes and MD5SUMs and prints a report of files which
|
|
|
|
|
don't match. It doesn't alter the source or destination.
|
|
|
|
|
|
2014-07-18 03:03:11 +08:00
|
|
|
rclone md5sum remote:path
|
|
|
|
|
|
|
|
|
|
Produces an md5sum file for all the objects in the path. This is in
|
|
|
|
|
the same format as the standard md5sum tool produces.
|
|
|
|
|
|
2014-03-16 01:39:56 +08:00
|
|
|
General options:
|
2014-04-27 00:43:41 +08:00
|
|
|
|
|
|
|
|
```
|
|
|
|
|
--checkers=8: Number of checkers to run in parallel.
|
|
|
|
|
--config="~/.rclone.conf": Config file.
|
|
|
|
|
-n, --dry-run=false: Do a trial run with no permanent changes
|
|
|
|
|
--modify-window=1ns: Max time diff to be considered the same
|
|
|
|
|
-q, --quiet=false: Print as little stuff as possible
|
|
|
|
|
--stats=1m0s: Interval to print stats
|
|
|
|
|
--transfers=4: Number of file transfers to run in parallel.
|
|
|
|
|
-v, --verbose=false: Print lots more stuff
|
|
|
|
|
```
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Developer options:
|
2014-04-27 00:43:41 +08:00
|
|
|
|
|
|
|
|
```
|
|
|
|
|
--cpuprofile="": Write cpu profile to file
|
|
|
|
|
```
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Local Filesystem
|
|
|
|
|
----------------
|
|
|
|
|
|
|
|
|
|
Paths are specified as normal filesystem paths, so
|
|
|
|
|
|
|
|
|
|
rclone sync /home/source /tmp/destination
|
|
|
|
|
|
2014-04-27 00:43:41 +08:00
|
|
|
Will sync `/home/source` to `/tmp/destination`
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Swift / Rackspace cloudfiles / Memset Memstore
|
|
|
|
|
----------------------------------------------
|
|
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
Paths are specified as remote:container (or remote: for the `lsd`
|
2014-05-06 02:52:52 +08:00
|
|
|
command.) You may put subdirectories in too, eg
|
|
|
|
|
`remote:container/path/to/dir`.
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-03-16 08:28:32 +08:00
|
|
|
So to copy a local directory to a swift container called backup:
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone sync /home/source swift:backup
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
The modified time is stored as metadata on the object as
|
2014-03-16 08:28:32 +08:00
|
|
|
`X-Object-Meta-Mtime` as floating point since the epoch.
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
This is a defacto standard (used in the official python-swiftclient
|
|
|
|
|
amongst others) for storing the modification time (as read using
|
|
|
|
|
os.Stat) for an object.
|
|
|
|
|
|
|
|
|
|
Amazon S3
|
|
|
|
|
---------
|
|
|
|
|
|
2014-05-06 02:52:52 +08:00
|
|
|
Paths are specified as remote:bucket. You may put subdirectories in
|
|
|
|
|
too, eg `remote:bucket/path/to/dir`.
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
So to copy a local directory to a s3 container called backup
|
|
|
|
|
|
2014-03-28 01:31:57 +08:00
|
|
|
rclone sync /home/source s3:backup
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
The modified time is stored as metadata on the object as
|
2014-03-16 08:28:32 +08:00
|
|
|
`X-Amz-Meta-Mtime` as floating point since the epoch.
|
2014-03-16 01:39:56 +08:00
|
|
|
|
|
|
|
|
Google drive
|
|
|
|
|
------------
|
|
|
|
|
|
2014-07-18 03:03:11 +08:00
|
|
|
Paths are specified as remote:path Drive paths may be as deep as required.
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-03-16 22:01:17 +08:00
|
|
|
The initial setup for drive involves getting a token from Google drive
|
2014-04-27 00:43:41 +08:00
|
|
|
which you need to do in your browser. `rclone config` walks you
|
2014-03-16 22:01:17 +08:00
|
|
|
through it.
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-03-16 22:01:17 +08:00
|
|
|
To copy a local directory to a drive directory called backup
|
|
|
|
|
|
2014-07-18 03:03:11 +08:00
|
|
|
rclone copy /home/source remote:backup
|
|
|
|
|
|
|
|
|
|
Google drive stores modification times accurate to 1 ms natively.
|
|
|
|
|
|
|
|
|
|
Dropbox
|
|
|
|
|
-------
|
|
|
|
|
|
|
|
|
|
Paths are specified as remote:path Dropbox paths may be as deep as required.
|
|
|
|
|
|
|
|
|
|
The initial setup for dropbox involves getting a token from Dropbox
|
|
|
|
|
which you need to do in your browser. `rclone config` walks you
|
|
|
|
|
through it.
|
|
|
|
|
|
|
|
|
|
To copy a local directory to a drive directory called backup
|
|
|
|
|
|
|
|
|
|
rclone copy /home/source dropbox:backup
|
|
|
|
|
|
|
|
|
|
Md5sums and timestamps in RFC3339 format accurate to 1ns are stored in
|
|
|
|
|
a Dropbox datastore called "rclone". Dropbox datastores are limited
|
|
|
|
|
to 100,000 rows so this is the maximum number of files rclone can
|
|
|
|
|
manage on Dropbox.
|
|
|
|
|
|
|
|
|
|
Google Cloud Storage
|
|
|
|
|
--------------------
|
|
|
|
|
|
|
|
|
|
Paths are specified as remote:path Google Cloud Storage paths may be
|
|
|
|
|
as deep as required.
|
|
|
|
|
|
|
|
|
|
The initial setup for Google Cloud Storage involves getting a token
|
|
|
|
|
from Google which you need to do in your browser. `rclone config`
|
|
|
|
|
walks you through it.
|
|
|
|
|
|
|
|
|
|
To copy a local directory to a google cloud storage directory called backup
|
|
|
|
|
|
|
|
|
|
rclone copy /home/source remote:backup
|
2014-03-16 22:01:17 +08:00
|
|
|
|
2014-07-18 03:03:11 +08:00
|
|
|
Google google cloud storage stores md5sums natively and rclone stores
|
|
|
|
|
modification times as metadata on the object, under the "mtime" key in
|
|
|
|
|
RFC3339 format accurate to 1ns.
|
2012-11-19 01:32:31 +08:00
|
|
|
|
2014-05-06 02:52:52 +08:00
|
|
|
Single file copies
|
|
|
|
|
------------------
|
|
|
|
|
|
|
|
|
|
Rclone can copy single files
|
|
|
|
|
|
|
|
|
|
rclone src:path/to/file dst:path/dir
|
|
|
|
|
|
|
|
|
|
Or
|
|
|
|
|
|
|
|
|
|
rclone src:path/to/file dst:path/to/file
|
|
|
|
|
|
|
|
|
|
Note that you can't rename the file if you are copying from one file to another.
|
|
|
|
|
|
2012-11-19 01:32:31 +08:00
|
|
|
License
|
|
|
|
|
-------
|
|
|
|
|
|
|
|
|
|
This is free software under the terms of MIT the license (check the
|
|
|
|
|
COPYING file included in this package).
|
|
|
|
|
|
2014-03-16 01:39:56 +08:00
|
|
|
Bugs
|
|
|
|
|
----
|
|
|
|
|
|
2014-03-16 22:01:17 +08:00
|
|
|
* Drive: Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
|
|
|
|
|
* quota is 100.0 requests/second/user
|
|
|
|
|
* Empty directories left behind with Local and Drive
|
|
|
|
|
* eg purging a local directory with subdirectories doesn't work
|
2014-03-16 01:39:56 +08:00
|
|
|
|
2014-04-25 01:11:08 +08:00
|
|
|
Changelog
|
|
|
|
|
---------
|
2014-07-20 18:28:50 +08:00
|
|
|
* v1.03 - 2014-07-20
|
|
|
|
|
* swift, s3, dropbox: fix updated files being marked as corrupted
|
|
|
|
|
* Make compile with go 1.1 again
|
2014-07-19 20:12:20 +08:00
|
|
|
* v1.02 - 2014-07-19
|
|
|
|
|
* Implement Dropbox remote
|
|
|
|
|
* Implement Google Cloud Storage remote
|
|
|
|
|
* Verify Md5sums and Sizes after copies
|
|
|
|
|
* Remove times from "ls" command - lists sizes only
|
|
|
|
|
* Add add "lsl" - lists times and sizes
|
|
|
|
|
* Add "md5sum" command
|
2014-07-05 04:15:27 +08:00
|
|
|
* v1.01 - 2014-07-04
|
|
|
|
|
* drive: fix transfer of big files using up lots of memory
|
2014-07-04 04:56:54 +08:00
|
|
|
* v1.00 - 2014-07-03
|
2014-07-05 04:15:27 +08:00
|
|
|
* drive: fix whole second dates
|
2014-06-27 00:49:26 +08:00
|
|
|
* v0.99 - 2014-06-26
|
|
|
|
|
* Fix --dry-run not working
|
|
|
|
|
* Make compatible with go 1.1
|
2014-05-31 00:29:20 +08:00
|
|
|
* v0.98 - 2014-05-30
|
|
|
|
|
* s3: Treat missing Content-Length as 0 for some ceph installations
|
|
|
|
|
* rclonetest: add file with a space in
|
2014-05-06 05:25:29 +08:00
|
|
|
* v0.97 - 2014-05-05
|
|
|
|
|
* Implement copying of single files
|
|
|
|
|
* s3 & swift: support paths inside containers/buckets
|
2014-04-25 01:11:08 +08:00
|
|
|
* v0.96 - 2014-04-24
|
|
|
|
|
* drive: Fix multiple files of same name being created
|
|
|
|
|
* drive: Use o.Update and fs.Put to optimise transfers
|
|
|
|
|
* Add version number, -V and --version
|
|
|
|
|
* v0.95 - 2014-03-28
|
|
|
|
|
* rclone.org: website, docs and graphics
|
|
|
|
|
* drive: fix path parsing
|
|
|
|
|
* v0.94 - 2014-03-27
|
|
|
|
|
* Change remote format one last time
|
|
|
|
|
* GNU style flags
|
|
|
|
|
* v0.93 - 2014-03-16
|
|
|
|
|
* drive: store token in config file
|
|
|
|
|
* cross compile other versions
|
|
|
|
|
* set strict permissions on config file
|
|
|
|
|
* v0.92 - 2014-03-15
|
|
|
|
|
* Config fixes and --config option
|
|
|
|
|
* v0.91 - 2014-03-15
|
|
|
|
|
* Make config file
|
|
|
|
|
* v0.90 - 2013-06-27
|
|
|
|
|
* Project named rclone
|
|
|
|
|
* v0.00 - 2012-11-18
|
|
|
|
|
* Project started
|
|
|
|
|
|
|
|
|
|
|
2012-11-19 01:32:31 +08:00
|
|
|
Contact and support
|
|
|
|
|
-------------------
|
|
|
|
|
|
|
|
|
|
The project website is at:
|
|
|
|
|
|
2014-03-16 22:01:17 +08:00
|
|
|
* https://github.com/ncw/rclone
|
2012-11-19 01:32:31 +08:00
|
|
|
|
2014-04-27 00:43:41 +08:00
|
|
|
There you can file bug reports, ask for help or send pull requests.
|
2012-11-19 01:32:31 +08:00
|
|
|
|
|
|
|
|
Authors
|
|
|
|
|
-------
|
|
|
|
|
|
2014-03-16 22:01:17 +08:00
|
|
|
* Nick Craig-Wood <nick@craig-wood.com>
|
2012-11-19 01:32:31 +08:00
|
|
|
|
|
|
|
|
Contributors
|
|
|
|
|
------------
|
|
|
|
|
|
2014-03-16 22:01:17 +08:00
|
|
|
* Your name goes here!
|
