8.7 KiB
| title | description | date |
|---|---|---|
| Remote Control | Remote controlling rclone | 2018-03-05 |
Remote controlling rclone
If rclone is run with the --rc flag then it starts an http server
which can be used to remote control rclone.
NB this is experimental and everything here is subject to change!
Supported parameters
--rc
Flag to start the http server listen on remote requests
--rc-addr=IP
IPaddress:Port or :Port to bind server to. (default "localhost:5572")
--rc-cert=KEY
SSL PEM key (concatenation of certificate and CA certificate)
--rc-client-ca=PATH
Client certificate authority to verify clients with
--rc-htpasswd=PATH
htpasswd file - if not provided no authentication is done
--rc-key=PATH
SSL PEM Private key
--rc-max-header-bytes=VALUE
Maximum size of request header (default 4096)
--rc-user=VALUE
User name for authentication.
--rc-pass=VALUE
Password for authentication.
--rc-realm=VALUE
Realm for authentication (default "rclone")
--rc-server-read-timeout=DURATION
Timeout for server reading data (default 1h0m0s)
--rc-server-write-timeout=DURATION
Timeout for server writing data (default 1h0m0s)
Accessing the remote control via the rclone rc command
Rclone itself implements the remote control protocol in its rclone rc command.
You can use it like this
$ rclone rc rc/noop param1=one param2=two
{
"param1": "one",
"param2": "two"
}
Run rclone rc on its own to see the help for the installed remote
control commands.
Supported commands
cache/expire: Purge a remote from cache
Purge a remote from the cache backend. Supports either a directory or a file.
Params:
- remote = path to remote (required)
- withData = true/false to delete cached data (chunks) as well (optional)
Eg
rclone rc cache/expire remote=path/to/sub/folder/
rclone rc cache/expire remote=/ withData=true
cache/stats: Get cache stats
Show statistics for the cache remote.
core/bwlimit: Set the bandwidth limit.
This sets the bandwidth limit to that passed in.
Eg
rclone rc core/bwlimit rate=1M
rclone rc core/bwlimit rate=off
The format of the parameter is exactly the same as passed to --bwlimit
except only one bandwidth may be specified.
core/gc: Runs a garbage collection.
This tells the go runtime to do a garbage collection run. It isn't
necessary to call this normally, but it can be useful for debugging
memory problems.
core/memstats: Returns the memory statistics
This returns the memory statistics of the running program. What the values mean
are explained in the go docs: https://golang.org/pkg/runtime/#MemStats
The most interesting values for most people are:
- HeapAlloc: This is the amount of memory rclone is actually using
- HeapSys: This is the amount of memory rclone has obtained from the OS
- Sys: this is the total amount of memory requested from the OS
- It is virtual memory so may include unused memory
core/pid: Return PID of current process
This returns PID of current process.
Useful for stopping rclone process.
rc/error: This returns an error
This returns an error with the input as part of its error string.
Useful for testing error handling.
rc/list: List all the registered remote control commands
This lists all the registered remote control commands as a JSON map in
the commands response.
rc/noop: Echo the input to the output parameters
This echoes the input parameters to the output parameters for testing
purposes. It can be used to check that rclone is still alive and to
check that parameter passing is working properly.
vfs/forget: Forget files or directories in the directory cache.
This forgets the paths in the directory cache causing them to be
re-read from the remote when needed.
If no paths are passed in then it will forget all the paths in the
directory cache.
rclone rc vfs/forget
Otherwise pass files or dirs in as file=path or dir=path. Any
parameter key starting with file will forget that file and any
starting with dir will forget that dir, eg
rclone rc vfs/forget file=hello file2=goodbye dir=home/junk
Accessing the remote control via HTTP
Rclone implements a simple HTTP based protocol.
Each endpoint takes an JSON object and returns a JSON object or an
error. The JSON objects are essentially a map of string names to
values.
All calls must made using POST.
The input objects can be supplied using URL parameters, POST
parameters or by supplying "Content-Type: application/json" and a JSON
blob in the body. There are examples of these below using curl.
The response will be a JSON blob in the body of the response. This is
formatted to be reasonably human readable.
If an error occurs then there will be an HTTP error status (usually
400) and the body of the response will contain a JSON encoded error
object.
Using POST with URL parameters only
curl -X POST 'http://localhost:5572/rc/noop/?potato=1&sausage=2'
Response
{
"potato": "1",
"sausage": "2"
}
Here is what an error response looks like:
curl -X POST 'http://localhost:5572/rc/error/?potato=1&sausage=2'
{
"error": "arbitrary error on input map[potato:1 sausage:2]",
"input": {
"potato": "1",
"sausage": "2"
}
}
Note that curl doesn't return errors to the shell unless you use the -f option
$ curl -f -X POST 'http://localhost:5572/rc/error/?potato=1&sausage=2'
curl: (22) The requested URL returned error: 400 Bad Request
$ echo $?
22
Using POST with a form
curl --data "potato=1" --data "sausage=2" http://localhost:5572/rc/noop/
Response
{
"potato": "1",
"sausage": "2"
}
Note that you can combine these with URL parameters too with the POST
parameters taking precedence.
curl --data "potato=1" --data "sausage=2" "http://localhost:5572/rc/noop/?rutabaga=3&sausage=4"
Response
{
"potato": "1",
"rutabaga": "3",
"sausage": "4"
}
Using POST with a JSON blob
curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' http://localhost:5572/rc/noop/
response
{
"password": "xyz",
"username": "xyz"
}
This can be combined with URL parameters too if required. The JSON
blob takes precedence.
curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' 'http://localhost:5572/rc/noop/?rutabaga=3&potato=4'
{
"potato": 2,
"rutabaga": "3",
"sausage": 1
}
Debugging rclone with pprof
If you use the --rc flag this will also enable the use of the go
profiling tools on the same port.
To use these, first install go.
Then (for example) to profile rclone's memory use you can run:
go tool pprof -web http://localhost:5572/debug/pprof/heap
This should open a page in your browser showing what is using what
memory.
You can also use the -text flag to produce a textual summary
$ go tool pprof -text http://localhost:5572/debug/pprof/heap
Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
flat flat% sum% cum cum%
1024.03kB 66.62% 66.62% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode
513kB 33.38% 100% 513kB 33.38% net/http.newBufioWriterSize
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/cmd/all.init
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/cmd/serve.init
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/cmd/serve/restic.init
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2.init
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init
0 0% 100% 1024.03kB 66.62% github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init.0
0 0% 100% 1024.03kB 66.62% main.init
0 0% 100% 513kB 33.38% net/http.(*conn).readRequest
0 0% 100% 513kB 33.38% net/http.(*conn).serve
0 0% 100% 1024.03kB 66.62% runtime.main
Possible profiles to look at:
- Memory:
go tool pprof http://localhost:5572/debug/pprof/heap - 30-second CPU profile:
go tool pprof http://localhost:5572/debug/pprof/profile - 5-second execution trace:
wget http://localhost:5572/debug/pprof/trace?seconds=5
See the net/http/pprof docs
for more info on how to use the profiling and for a general overview
see the Go team's blog post on profiling go programs.
The profiling hook is zero overhead unless it is used.