Commit Graph

92 Commits

Author SHA1 Message Date
Nick Craig-Wood
28f9b9b611 drive: detect files using file size as well as md5 - fixes 980 2017-02-03 08:00:03 +00:00
Nick Craig-Wood
7679620f4b drive: Experimentally add --drive-list-chunk 2017-02-02 21:49:02 +00:00
okaresz
cbc113492a Add Drive specific option: --drive-skip-gdocs - fixes #1035 2017-01-29 20:53:51 +00:00
Nick Craig-Wood
1fa258c2b4 Define a new Features() method for Fs
Optional interfaces are becoming more important in rclone,
--track-renames and --backup-dir both rely on them.

Up to this point rclone has used interface upgrades to define optional
behaviour on Fs objects.  However when one Fs object wraps another it
is very difficult for this scheme to work accurately.  rclone has
relied on specific error messages being returned when the interface
isn't supported - this is unsatisfactory because it means you have to
call the interface to see whether it is supported.

This change enables accurate detection of optional interfaces by use
of a Features struct as returned by an obligatory Fs.Features()
method.  The Features struct contains flags and function pointers
which can be tested against nil to see whether they can be used.

As a result crypt and hubic can accurately reflect the capabilities of
the underlying Fs they are wrapping.
2017-01-16 17:33:25 +00:00
Nick Craig-Wood
0d75d2585f Allow all options to be set from environment variables
The option names are munged changing - to _ making upper case and
prepending RCLONE_.  The values are as parsed by pflag.
2017-01-03 22:42:47 +00:00
Nick Craig-Wood
c55402caa2 drive: create destination directory on Move() 2016-12-09 16:57:07 +00:00
Nick Craig-Wood
d132dc7640 drive: make DirMove more efficient and complain about moving the root 2016-12-09 16:57:07 +00:00
Nick Craig-Wood
48a2e3844d Add optional interface DirCacheFlush for making the tests more reliable
This is defined for the users of dircache drive, onedrive, and acd.

This helps fix the DirMove tests on acd.
2016-12-09 16:57:07 +00:00
Nick Craig-Wood
aaa1370a36 Add directory parameter to Rmdir and Mkdir #100 #831
This will enable rclone to manage directories properly in the future.
2016-11-26 12:02:53 +00:00
Nick Craig-Wood
4803ce010e Make exponential backoff work exactly as per google specification - fixes #583 2016-10-17 17:57:09 +01:00
Nick Craig-Wood
aef2ac5c04 Add options for Open and implement Range for all remotes 2016-10-05 21:03:56 +01:00
Nick Craig-Wood
945f49ab5e Make ContentType be preserved for cloud -> cloud copies - fixes #733 2016-10-03 20:02:04 +01:00
Nick Craig-Wood
c117eaf5a2 drive: add .epub, .odp and .tsv as export formats. 2016-09-19 18:08:10 +01:00
Nick Craig-Wood
0cb9bb3b54 Redo http Transport code
* Insert User-Agent in Transport - fixes #199
  * Update timeouts to use Context
  * Modernise transport
2016-09-12 17:50:19 +01:00
Nick Craig-Wood
6a4e424630 Re-implement Obscure/Reveal so they use AES-CTR encryption 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
ab43005422 Make NewObject return an error
* make it return an error
  * make a canonical error fs.ErrorNotFound
  * make a test for it
  * remove logs/debugs of error
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
b1f131964e Rename NewFsObject to NewObject 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
1a87b69376 Get rid of LimitedFs - FIXME needs docs on copying single files
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.

Use this to remove the LimitedFs and just add this file to the
--files-from list.

This means that server side operations can be used also.

Fixes #518
Fixes #545
2016-06-28 08:51:43 +01:00
Nick Craig-Wood
cc6a776034 drive, acd: Tweak logging after changing Fs.Put so that it must cope with existing files 2016-06-18 10:54:42 +01:00
Nick Craig-Wood
2cfb3834f2 Log errors with %v 2016-06-18 09:36:47 +01:00
Nick Craig-Wood
df1092ef33 Change Fs.Put so that it must cope with existing files
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.

None of the other Fses needed adjusting.

Fixes #483
2016-06-13 19:29:10 +01:00
Nick Craig-Wood
4c5b2833b3 Convert to using github.com/pkg/errors everywhere 2016-06-13 17:43:03 +01:00
Nick Craig-Wood
661715733a Make sure we don't use conflicting content types on upload - fixes #513 2016-06-09 17:52:58 +01:00
Nick Craig-Wood
0a922ad1dc acd: Reauth on 401 errors
Fixes #493
Fixes #501
2016-05-28 16:49:26 +01:00
Nick Craig-Wood
c2d0e86431 Add more tests for List() and fix resulting problems 2016-05-07 14:50:35 +01:00
Nick Craig-Wood
68ec6a9f5b Add a directory parameter to Fs.List() 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
2016-05-06 16:52:34 +01:00
Nick Craig-Wood
ddb47758f3 drive: increase default chunk size to 8 MB and document - fixes #397 2016-04-12 21:33:55 +01:00
Nick Craig-Wood
6cc9c09610 drive: preserve mime type on file update - fixes #417 2016-04-04 16:58:42 +01:00
Nick Craig-Wood
414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
fdd4b4ee22 drive: Add missing retries for Move and DirMove 2016-03-06 18:15:01 +00:00
Nick Craig-Wood
2119fb4314 drive: tweak pacer to speed up directory listings and make more reliable 2016-03-06 18:15:01 +00:00
Nick Craig-Wood
7d4e143dee Make it obvious that the client secrets are encrypted 2016-02-28 19:57:19 +00:00
Nick Craig-Wood
b279df2e67 Drive: disable copy and move for google docs - fixes #332 2016-02-28 09:35:28 +00:00
Nick Craig-Wood
e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
klauspost
ef06371c93 Create separate interface for object information.
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
2016-02-19 13:31:09 +00:00
Nick Craig-Wood
1cd0d9a1f2 Fix listing drive docs at root - fixes #336
* Remove full drive list code
    * it is slower and uses more data
    * having two directory listing routines is causing problems (including this one)
    * less code is more
  * Make sure we don't recurse into directories we don't own
  * Fix export extension handling and add tests
2016-02-15 16:46:43 +00:00
Nick Craig-Wood
558bc2e132 drive: Export Google documents - fixes #49
Rclone will download one format of a google doc. The choice of which
export format is controlled by the `--drive-formats` flag.
2016-01-31 16:10:43 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types.
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
Nick Craig-Wood
5189231a34 Tweaks to rclone authorize
* Document the headless / remote setup procedure
  * Move Config constants into fs
  * Parse arguments in main for Authorize
2016-01-07 20:31:23 +00:00
klauspost
bcbd30bb8a Add easier headless configuration.
This will allow setting up a remote with copy&paste of values to a headless machine. It will allow copy+pasting a token into the configuration.

This requires rclone to be on a machine with a proper browser. Custom client id and secrets are supported.

To test token generation, use `rclone auth "fs type"`.
2016-01-07 20:31:23 +00:00
Björn Harrtell
78edafcaac drive: add --drive-auth-owner-only to only consider files owned by the user. 2016-01-05 17:02:04 +00:00
Nick Craig-Wood
b257de4aba Be more constistent with naming in remotes
* External objects are called Fs and Object
  * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
Nick Craig-Wood
ff16e0f6df Factor common error handling into fs module 2015-11-06 15:26:58 +00:00
Nick Craig-Wood
bf562d7373 Tweak wording on client/secret ids in acd, gcs and drive - fixes #155 2015-10-03 16:04:45 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
klauspost
6fbd9cf24b Make Google Drive Directory reads concurrent for increased speed.
If you use the -v flag then this will show the progress of directory
listing.  Fixes #143.
2015-09-17 18:42:10 +01:00
Nick Craig-Wood
0872ec3204 Allow user to override credentials again in drive, gcs and acd - fixes #139 2015-09-16 20:08:40 +01:00
Nick Craig-Wood
073d112204 Factor pacer module from Drive and use it in Amazon Cloud Drive for
smooth API pacing and retry logic.
2015-09-11 19:18:41 +01:00
Nick Craig-Wood
ea12e446ca Factor DirCache from drive into its own module 2015-09-05 11:58:54 +01:00