Commit Graph

1643 Commits

Author SHA1 Message Date
Nick Craig-Wood
d178233e74 sync,march: check the cancel context on every channel send and receive
This fixes a deadlock on sync when all the copying channels receive a
Fatal Error.
2018-05-05 12:58:28 +01:00
Fabian Möller
9fdf273614 fs: improve ChunkedReader
- make Close permanent and return errors afterwards
- use RangeSeek from the wrapped reader if present
- add a limit to chunk growth
- correct RangeSeek interface behavior
- add tests
2018-05-05 12:49:42 +01:00
Nick Craig-Wood
55674c0bfc Start v1.41-DEV development 2018-04-28 12:37:55 +01:00
Nick Craig-Wood
e4c380b2a8 Version v1.41 2018-04-28 11:46:27 +01:00
Nick Craig-Wood
74cbdea0ef Revert "copy: create (pseudo copy) empty source directories to destination"
Unfortunately this commit attempts to create every directory rather
than just the empty ones, so will need re-working.

Removing this feature for the 1.41 release

This reverts commit 0daced29db.
2018-04-28 10:02:32 +01:00
ishuah
0daced29db copy: create (pseudo copy) empty source directories to destination - fixes #1837 2018-04-27 16:15:32 +01:00
Richard Yang
a81ec00a8c dedupe: Add dedupe largest functionality - fixes #2269 2018-04-26 16:21:07 +01:00
Nick Craig-Wood
9f04ce282e rc: fix setting bwlimit to unlimited 2018-04-26 12:21:29 +01:00
Nick Craig-Wood
764440068e filter: fix --min-age and --max-age together check
Somehow in the code reorganisation of
11da2a6c9b the check for --min-age and
--max-age got switched around.  This commit fixes that and means you
can use --min-age and --max-age together.
2018-04-26 09:17:22 +01:00
Nick Craig-Wood
a703216286 filter: take double negatives out of filter flag help 2018-04-26 09:17:13 +01:00
Nick Craig-Wood
9ab2521ef2 rc: autogenerate and tidy the docs and commands
* Rename rc/pid -> core/pid
  * Sort the output of `rc list`
  * Make a script to autogenerate the docs
  * Tidy docs
2018-04-23 20:57:17 +01:00
Nick Craig-Wood
21a10e58c9 rc: implement core/memstats to print internal memory usage info 2018-04-23 20:49:36 +01:00
Nick Craig-Wood
24980d7123 config: fix typo in error message #2268 2018-04-21 22:49:30 +01:00
Nick Craig-Wood
bcdb7719c6 fs/hash: install QuickXorHash as a supported rclone hash type #2262 2018-04-20 21:02:57 +01:00
Nick Craig-Wood
c51d97c752 hashsum: make generic tool for any hash to produce md5sum like output 2018-04-20 21:02:37 +01:00
Nick Craig-Wood
ef3bcec76c fs: Extend SizeSuffix to include TB and PB for rclone about 2018-04-17 21:53:42 +01:00
Nick Craig-Wood
1ac6dacf0f about: complete other providers and re-work internals
* Implement about for:
    * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
  * Implement `--json` and `---full` flag for `rclone about`
  * change About interface to return a Usage structure
  * Remove operations.About as it is too thin an interface
  * Implement Integration test

Relates to #1138 and #1564
2018-04-17 21:53:27 +01:00
a-roussos
94e277d759 about: add new command 'about' to get quota info from a remote
Implemented for drive only.

Relates to #1138 and #1564.
2018-04-17 21:50:14 +01:00
Nick Craig-Wood
8c3740c2c5 config: Improve the Provider matching to have a negated match #2140
This makes it easier to make classes of provider in the config.
2018-04-13 16:06:37 +01:00
Giri Badanahatti
acd5d4377e config,s3: hierarchical configuration support #2140
This introduces a method of making provider specific configuration
within a remote.  This is useful particularly in s3.

This commit does the basic configuration in S3 for IBM COS.
2018-04-13 16:05:35 +01:00
Peter Baumgartner
1db68571fd s3,swift: Add --use-server-modtime
`--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
2018-04-13 13:32:17 +01:00
Nick Craig-Wood
d97fe3b824 fs/operations: make dedupe work with mega
* factor into its own files
  * remove assumptions about having a given hash type
  * make tests work if the remote has no hash
2018-04-13 13:23:55 +01:00
Nick Craig-Wood
dcf8334673 fs: add --dump goroutines and --dump openfiles
These are developer flags useful for tracking down resource leaks.
2018-04-11 20:55:58 +01:00
Nick Craig-Wood
37be78705d fs/fshttp: limit MaxIdleConns and MaxIdleConnsPerHost
Before this change mega (which uses a different host per download)
would open too many sockets.
2018-04-11 20:51:28 +01:00
Michael P. Dubner
92b266d361 rc: new call rc/pid - closes #2211 2018-04-09 13:33:04 +01:00
Nick Craig-Wood
cbec59146a lsf: make sure we use localtime in tests - fixes Box integration tests
This problem was introduced with eca99b33c0.  It seems Box is the only
remote which converts time zones, so if you give it a GMT time zone,
it returns a PST time zone which represents the same instant.
2018-04-09 11:46:49 +01:00
Nick Craig-Wood
be54fd8f70 Remove builds conditional on go1.7 since that is now guaranteed #2154
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
2018-04-07 11:42:55 +01:00
Nick Craig-Wood
e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood
80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Nick Craig-Wood
67023f0040 Require go1.7 for compilation #2154
* Update the travis tests to exclude go1.6
  * Update the compile check to require go1.7+
  * Update misc go1.6 workarounds marked in the source
2018-04-06 20:18:14 +01:00
Matt Holt
e13f65b953 serve restic: Print actual listener address 2018-04-04 14:56:26 +01:00
Nick Craig-Wood
eca99b33c0 lsd,lsf: make sure all times we output are in local time - fixes #2183
Previous to this change times from lsd/lsf were output in whatever
timezone they were in whereas times from lsl were converted to
localtime.
2018-04-01 15:40:04 +01:00
Stefan
a8267d1628
link: allow creating public link to files and folders - closes #1562 2018-03-29 09:10:19 +02:00
Mateusz Pabian
91b068ad3a sync: implement --ignore-errors - fixes #642 2018-03-23 22:01:10 +00:00
Nick Craig-Wood
940df88eb2 Start v1.40-DEV development 2018-03-19 14:20:48 +00:00
Nick Craig-Wood
26f1c55987 Version v1.40 2018-03-19 10:06:13 +00:00
Stefan
86892467d9
config: load config file only on first access (closes #1659, closes #2096) (#2147) 2018-03-17 12:36:30 +01:00
Nick Craig-Wood
4295428a0f fs/accounting: add remote control of bwlimit 2018-03-17 10:34:02 +00:00
Nick Craig-Wood
5bf639048f sync: log an error that --track-renames doesn't work with sync or move
Fixes #2008
2018-03-17 10:34:02 +00:00
remusb
97b48cf988 rc: add support for Go 1.6 2018-03-14 22:58:20 +02:00
Nick Craig-Wood
86e5a35491 Implement Remote Control for rclone #2111
This implements a remote control protocol activated with the --rc flag
and a new command `rclone rc` to use that interface.

Still to do
  * docs - need finishing
  * tests
2018-03-14 22:58:20 +02:00
Nick Craig-Wood
dfd0f4c5a4 sync: when using --backup-dir don't delete files if we can't set their modtime
This is a problem when syncing a file which just needed its modtime
set with dropbox which can't set the mod time of a file without
re-uploading it.

Before this change we would delete the file, then the server side move
would fail moving the file to the backup-dir because it no longer
existed.

After this change the destination file is moved to the backup-dir
instead of being deleted and the new file is uploaded.

Fixes #2134
2018-03-13 16:05:06 +00:00
Nick Craig-Wood
0c9dc006c5 fs: make display of default values of --min-age/--max-age be off - Fixes #2121 2018-03-13 09:06:07 +00:00
Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it (#2094)
* fs: rename DirChangeNotify to ChangeNotify

* cache: switch to ChangeNotify

* ChangeNotify: keep order of notifications
2018-03-08 22:03:34 +02:00
Nick Craig-Wood
e0eb666dbf fs/walk: fix new golint warning about unused variables in range 2018-03-02 17:01:58 +00:00
Nick Craig-Wood
abc736df1d cat: Use RangeOption for limited fetches to make more efficient #1825 2018-02-19 18:00:39 +00:00
Nick Craig-Wood
ab0d06eb16 fs: Make RangeOption mandatory #1825 2018-02-19 18:00:39 +00:00
Mateusz
afc963ed92 config: retry saving the config after failure - fixes #2060 2018-02-19 17:59:27 +00:00
Fabian Möller
451cd6d971 fs: add ChunkedReader 2018-02-19 15:45:24 +00:00
Fabian Möller
a647c54888 fs: add RangeSeeker interface 2018-02-19 15:45:24 +00:00
Nick Craig-Wood
a5c3bcc9c7 fshttp: fix idle timeouts for HTTP connections #2057
Now we only nudge on the idle timeout after a successful Read or Write
which returns some bytes and no errors.
2018-02-16 10:35:41 +00:00
Bjørn Erik Pedersen
ab8c0a81fa Add a delete threshold to sync (--max-delete)
Fixes #959
2018-02-12 11:29:58 +00:00
Nick Craig-Wood
d0d6b83a7a fs/accounting: rework to enable accounting to work with crypt and b2
This removes the old system of part accounting and replaces it with a
system of popping off the accounting reader and wrapping up new ones
as necessary.

This makes it much easier to carry the context down the chain of
wrapped readers and get the limiting as near as possible to the
output.  This makes the accounting more accurate and the bandwidth
limiting smoother.

Fixes #2029 and Fixes #1443
2018-02-02 15:14:41 +00:00
Nick Craig-Wood
bea02fcf52 fs/accounting: factor into separate files without changing functionality 2018-02-02 15:14:40 +00:00
Nick Craig-Wood
6fb868e00c config: fix --log-level flag after code reorganization - fixes #2043 2018-02-02 14:07:44 +00:00
Nick Craig-Wood
8df78f2b6d operations: ignore size of objects when they are < 0 #320
This allows google docs to be transferred and checked correctly.
2018-01-31 16:22:05 +00:00
Nick Craig-Wood
2eb5cfb7ad fs: Formalize the ObjectUnWrapper interface 2018-01-31 16:21:41 +00:00
Nick Craig-Wood
fe52502f19 fs: Adjust RangeOption.Decode to return -1 for read to end
A Range request can never request 0 bytes however this change was made
to make a clearer signal that the limit means read to the end.

Add test and more documentation and fixup uses
2018-01-27 14:31:29 +00:00
Nick Craig-Wood
9a73688e3a fs: Add ParseRangeOption to parse incoming Range: requests 2018-01-27 13:16:37 +00:00
Nick Craig-Wood
bc3ee977f4 fs/hash: move interface assertion to tests so it doesn't pull in spf13/flag 2018-01-26 14:35:18 +00:00
Nick Craig-Wood
a6833b68ca local: factor RangeOption code to Decode() method and readers.LimitedReadCloser #1825 2018-01-22 19:44:00 +00:00
Nick Craig-Wood
d876392d15 onedrive: Factor code into fs.FixRangeOption 2018-01-22 17:05:00 +00:00
Nick Craig-Wood
84bc4dc142 Clarify RangeOption semantics 2018-01-21 09:51:28 +00:00
Nick Craig-Wood
ee66419a27 fs/fserrors: Add test for error from #1964 2018-01-19 17:07:40 +00:00
Nick Craig-Wood
5ad226ab54 fs: Add dir option to fs.Purge #1891
Purge optional interface needs fixing too.
2018-01-19 11:45:50 +00:00
Nick Craig-Wood
97c414f025 config/hash: rename more symbols after factoring into own package 2018-01-18 20:27:52 +00:00
Nick Craig-Wood
71722b5b95 config: factor Obscure and Reveal into its own package 2018-01-18 20:19:55 +00:00
Ernest Borowski
5cce74d630 flags: remove --no-traverse flag because it is obsolete - fixes #1813
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2018-01-18 11:00:25 +00:00
Iakov Davydov
acd55a8f65 local, fs: --exclude-if-present ignores directories which it doesn't have permission for - fixes #1959 2018-01-16 20:00:16 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts.
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
2018-01-15 17:51:14 +00:00
Nick Craig-Wood
b8b620f5c2 Move all backends into backend directory 2018-01-12 17:07:38 +00:00
Nick Craig-Wood
712e6a8085 lsf: fix integration tests 2018-01-11 13:52:15 +00:00
Will Gunn
5c7e8d5a2b fs: Add --stats-file-name-length to specify the printed file name length for stats
Fixes #1206
2018-01-10 20:32:36 +00:00
Nick Craig-Wood
c74c3b37da lsf: add option to print hashes 2018-01-06 17:53:37 +00:00
Nick Craig-Wood
7c71ee1a5b fs: fix TestListFormat on remotes which return 0 as dir size not -1 2018-01-06 17:47:42 +00:00
Jakub Tasiemski
0d041602cf cmd: new command lsf 2018-01-06 14:39:31 +00:00
Nick Craig-Wood
b412c745a1 Start v1.39-DEV development 2017-12-23 13:40:28 +00:00
Nick Craig-Wood
f34a9116d4 Version v1.39 2017-12-23 13:07:45 +00:00
Nick Craig-Wood
7e585cda96 fs: fix TestRmdirsLeaveRoot test 2017-12-14 08:57:28 +00:00
lewapm
9c242edc10 rmdirs: add --leave-root flag - fixes #1874 2017-12-13 10:23:54 +00:00
ishuah
b05e472d2e stats: condensed transfer output to fit 80x25 terminals 2017-12-09 10:48:36 +03:00
remusb
25b073c767
fs: add Wrap feature for FS to identify their parent FS (#1884) 2017-12-06 17:14:34 +02:00
Nick Craig-Wood
fa4a25a73b fs: only test one level of cache
Can't test multiple caches at once as can only have 1 DB open at once
2017-12-04 15:50:59 +00:00
Ernest Borowski
934df67aef filter: warn the user if he use --include and --exclude together fixes #1764
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2017-12-04 14:20:01 +00:00
Nick Craig-Wood
4b185355df fs: rcat - use in memory object and Copy for more reliable transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
7d15c33e42 fs: make Copy and Move return the destination object if possible 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
11332a19a0 fs: make an in memory object for short transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
639e812789 fs: integration tests: add SUMMARY heading for log scraping 2017-11-29 15:55:37 +00:00
Nick Craig-Wood
1c6cad2252 fs: integration tests: add 30 minute timeout per test 2017-11-29 13:51:17 +00:00
ishuah
aab8051f50 move: add --delete-empty-src-dirs flag - fixes #1854 2017-11-28 11:38:19 +03:00
Nick Craig-Wood
fdb01437d8 fs: Allow the http Transport to have an optional filter request function 2017-11-24 09:07:56 +00:00
Nick Craig-Wood
02ffd43572 fs: Save the config before asking for a token - fixes #1220
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.

This means the post config is done after the user has confirmed the
config is OK which can't be helped.
2017-11-23 14:01:32 +00:00
Nick Craig-Wood
e53892f53b fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer.  Since we know the size
of the chunks, pre-allocating it should be much more efficient.

RepeatableReaderBuffer uses the buffer passed in.

RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
2017-11-23 13:53:46 +00:00
ishuah
6c62fced60 move: fixed root source directories getting deleted after move - fixes #1849 2017-11-23 12:01:35 +03:00
Nick Craig-Wood
159fce0106 fs: fix --cache-dir to have some effect 2017-11-22 17:05:02 +00:00
Nick Craig-Wood
cbe5d7ce64 fs: Remove X-Auth-Token: from headers when dumping for swift 2017-11-21 17:32:07 +00:00
Nick Craig-Wood
1a65a4e769 fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain

  * headers - HTTP headers as before
  * bodies  - HTTP bodies as before
  * requests - HTTP request bodies
  * responses - HTTP response bodies
  * auth - HTTP auth
  * filters - Filter rexeps

Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
2017-11-21 17:32:07 +00:00
ishuah
b4aa920a3d stats: show the amount of data transferred in kb/mb - fixes #1167 2017-11-21 12:40:02 +03:00
Nick Craig-Wood
8951875c21 vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
e946a8eab0 fs: Add CacheDir config variable 2017-11-20 12:00:32 +00:00
Nick Craig-Wood
46947b3b9b rcat: fix goroutine leak
This was leaking goroutines in the short file case beause it wasn't
calling Close() on the Account object.  This became apparent when
testing with mount.
2017-11-20 12:00:32 +00:00
Iakov Davydov
7d3abdc463 tests for --exclude-if-present 2017-11-17 21:51:11 +00:00
Iakov Davydov
538246f6c3 support exclude file in --fast-list mode 2017-11-17 21:51:11 +00:00
Iakov Davydov
557dd8f031 ListDirSorted check for excludefile 2017-11-17 21:51:11 +00:00
Iakov Davydov
37aaa19f3a new option: --exclude-if-present 2017-11-17 21:51:11 +00:00
Iakov Davydov
cef2e3bf83 path -> startPath in walkRDirTree (we need the path package) 2017-11-17 21:51:11 +00:00
Iakov Davydov
a3a436ce16 WalkRDirTree: return error if unknown item type 2017-11-17 21:51:11 +00:00
Iakov Davydov
5d05df3124 ListContainsExcludeFile: checks for exclude file in the list 2017-11-17 21:51:11 +00:00
Iakov Davydov
421ba84e12 DirTree.Prune: deletes several directories 2017-11-17 21:51:11 +00:00
Iakov Davydov
7ae7080824 FileExists check if a file exists 2017-11-17 21:51:11 +00:00
Remus Bunduc
016abf825e cache: first version 2017-11-15 15:23:21 +00:00
ishuah
aec2265be0 rclone: implement exit codes - #1136 2017-11-15 17:48:37 +03:00
Substantiel
2423fa40e2 config: add password sub command for setting obscured passwords 2017-11-15 14:44:45 +00:00
ishuah
c65734ee69 move: delete source directory after successful move - fixes #1642 2017-11-07 22:21:38 +00:00
Substantiel
7749157596 Add --auto-confirm flag 2017-11-05 21:56:50 +00:00
Nick Craig-Wood
cc7ed13b9b fs: factor test running code into fstest/run.go 2017-11-03 12:59:58 +00:00
ishuah
10f27e2ff2 allow trailing+leading whitespace for passwords - #1717
warn users when they enter passwords with leading/trailing whitespaces

Updated config_test.go, removing deprecated test case and updated TestReveal
2017-10-20 15:46:17 +01:00
Nick Craig-Wood
115053930e Make error messages less crypting when revealing an unobscured password - fixes #1743 2017-10-16 22:03:06 +01:00
Nick Craig-Wood
edfab09eb9 config: add sub commands for full config file management
Previously config sub commands were manually parsed rather than using
cobra.

Make config command have the following sub commands:

 * create    Create a new remote with name, type and options.
 * delete    Delete an existing remote <name>.
 * dump      Dump the config file as JSON.
 * edit      Enter an interactive configuration session.
 * file      Show path of configuration file in use.
 * providers List in JSON format all the providers and options.
 * show      Print (decrypted) config file, or the config for a single remote.
 * update    Update options in an existing remote.

The following changes were made to existing commands

 * listproviders was renamed to providers
 * listoptions was removed in favour of providing the output in providers
 * jsonconfig was renamed to create
 * an optional parameter was added to the show command
2017-10-14 11:50:41 +01:00
thierry
0575623dff Add config listproviders, listoptions, jsonconfig for automated config
Addition of a method listing the providers, a method listing the
options of a provider and method of manual configuration.
2017-10-13 17:17:36 +01:00
Nick Craig-Wood
fc8b13c993 moveto/copyto: Fix to allow copying to the same name - fixes #1736 2017-10-12 20:45:36 +01:00
Nick Craig-Wood
43ced30f11 fs: Add more errors to retry - fixes #1733 2017-10-10 19:51:02 +01:00
Nick Craig-Wood
3eb7f52e39 fs: Add "unexpected EOF reading trailer" as a retriable error - fixes #1730 2017-10-09 17:29:16 +01:00
Nick Craig-Wood
7f3dc9b5c4 Implement WebDAV remote #580
This has special knowledge of Owncloud and Nextcloud to enable more
functionality such as mod times.
2017-10-09 16:19:37 +01:00
Nick Craig-Wood
00fe6d95da fs: fix duplicate files causing spurious copies
Before this fix duplicate files (on Google Drive) caused the next file
to be spuriously copied.  `rclone dedupe` worked around the problem.
2017-10-02 16:52:53 +01:00
Nick Craig-Wood
9e9297838f Implement pcloud remote - #418 2017-10-01 11:37:35 +01:00
Nick Craig-Wood
1c10497b68 Start v1.38-DEV development 2017-09-30 15:16:09 +01:00
Nick Craig-Wood
d96e45ba5b Version v1.38 2017-09-30 14:20:43 +01:00
Nick Craig-Wood
657b3a674d fs: fix test_all -clean to run just one cleaning thread per remote 2017-09-30 11:07:09 +01:00
Nick Craig-Wood
2857ed5c35 fs: fix --immutable tests on remotes which don't have modtime 2017-09-28 08:56:30 +01:00
Jacob McNamee
2d8e75cab4 Implement --immutable option 2017-09-24 20:00:00 +01:00
Stefan
85877f3adc config: add show/file subcommands which print the config/its path (fixes #1086) 2017-09-19 17:59:19 +02:00
Nick Craig-Wood
87335de8a8 fs: fix filename normalization issues in the tests when running on OS X 2017-09-17 15:31:22 +01:00
Stefan Breunig
12405f9f41 fuse: re-use rcat to support uploads for all remotes (fixes #1672) 2017-09-16 22:49:08 +02:00
Stefan
234bfae0d5 b2: implement streaming upload of files with unknown length (see #1614) (closes #1686) 2017-09-16 22:43:48 +02:00
Nick Craig-Wood
4ac9a65049 fs: stop normalizing file names but do a normalized compare in the sync
This works by using a transform function to transform file names when
doing a compare when matching file names in a directory.  rclone now
UTF-8 normalizes the file names and does a case insensitive compare if
the destination remote is case insensitive.

This deprecates the --local-no-unicode-normalization flag.

Fixes #1477
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
a8e41f081c fs: re-implement check and cryptcheck using the same traversal as sync
This makes them 100% consistent with sync and also make them use less
memory as they no longer build the whole tree in memory first.

Fixes #1657
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
261c7ad9e4 fs: make syncCopyMove use context for go routine cancellation 2017-09-16 19:49:31 +01:00
Nick Craig-Wood
fe96d5cf0a fs: factor multiple directory traverse out of sync 2017-09-16 19:49:31 +01:00
Nick Craig-Wood
6df12b3f00 fs: improve retriable error detection 2017-09-16 19:48:49 +01:00
Stefan Breunig
9d3f11b493 amazonclouddrive, rcat: ensure rcat integration test passes even with AmazonCloudDrive (fixes: #1680) 2017-09-15 18:09:04 +02:00
Nick Craig-Wood
798502b204 fs: add more errors to be considered temporary errors
This makes a framework for adding temporary errors identified by
syscall number or by error string.

Fixes #1660
2017-09-14 18:01:43 +01:00
Stefan Breunig
56dedc49e3 rcat: properly report if the upload fails 2017-09-13 20:21:52 +02:00
Girish Ramakrishnan
4c45cbea18 copy: error out if dst could not be listed 2017-09-12 09:29:44 +01:00
Stefan Breunig
a91448c83a rcat: honor --dry-run even for small files 2017-09-11 22:28:16 +02:00
Stefan Breunig
80b1f2a494 rcat: configurable small files cutoff and implement proper upload verification 2017-09-11 08:26:53 +02:00
Stefan Breunig
57817397a0 rcat: directly upload small files without streaming them 2017-09-11 08:25:34 +02:00
Nick Craig-Wood
cbea06026a Make check obey --ignore-size - fixes #1643 2017-09-01 11:20:41 +01:00
Nick Craig-Wood
2c8d6e86cc fs: fix gofmt 2017-08-31 16:01:19 +01:00
cbruegg
bb6300b032 Fix bwlimit toggle in conjunction with schedules (Fixes #1607) 2017-08-31 15:33:29 +01:00
Stefan Breunig
a122b9fa7a yandex: implement streaming uploads (see #1614) 2017-08-19 14:07:23 +02:00
Alex McGrath Kraak
2cc9071791 http: add --user-agent option. close #1557 2017-08-17 06:49:27 +02:00
Nick Craig-Wood
265fb8a5e2 fs: Manage empty directories - fixes #100
During the sync we collect a list of directories which should be empty
and attempt to rmdir them at the end of the sync.  If the directories
are not empty then the rmdir will fail, logging a message but not
erroring the sync.
2017-08-09 21:07:00 +01:00
Nick Craig-Wood
8a1a900733 fstest: use Feature.CanHaveEmptyDirectories to sharpen tests
Now we actually test whether the directories are present or not,
filtering out empty directories in the test using the
CanHaveEmptyDirectories flag.
2017-08-09 20:55:08 +01:00
Nick Craig-Wood
20ae7d562b fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes 2017-08-09 20:55:08 +01:00
Nick Craig-Wood
ec2ea37ad2 fs: Add --disable flag to disable optional features - fixes #1551
Eg to disable server side copy use `--disable copy`, to see a list of
what you can disable, `--disable help`.
2017-08-07 21:34:45 +01:00
Nick Craig-Wood
bfe812ea6b dedupe: implement merging of duplicate directories - fixes #1243 2017-08-07 15:36:41 +01:00
Nick Craig-Wood
db1995e63a Add MergeDirs optional interface and implement it for drive 2017-08-07 15:32:47 +01:00
Nick Craig-Wood
81a2ab599f fs: add optional ID to fs.Directory and set it in the remotes which care 2017-08-07 15:31:22 +01:00
Stefan Breunig
40f24e0ea3 config: use absolute ConfigPath to ensure newly written config is on the same mount - fixes #1569 2017-08-05 12:13:25 +02:00
Nick Craig-Wood
92d2e1f8d7 azureblob: rework and complete #801
* Fixup bitrot (rclone and Azure library)
  * Implement Copy
  * Add modtime to metadata under mtime key as RFC3339Nano
  * Make multipart upload work
  * Make it pass the integration tests
  * Fix uploading of zero length blobs
  * Rename to azureblob as it seems likely we will do azurefile
  * Add docs
2017-08-04 22:56:16 +01:00
Andrei Dragomir
036fd61a50 Added Azure Blob storage support #801 2017-08-04 22:54:27 +01:00
wuyu
ec5b72f8d5 Add new QingStor remote
Add new package qingstor to support QingStor API.

Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.

Add new docs for it.
2017-08-04 17:25:47 +01:00
Nick Craig-Wood
f682002b84 fs: Make tests create a new bucket rather than purging the old one
This enables QingStor to pass the tests as it has a 2 minute lockout
on deleting the old bucket then creating it again.
2017-08-04 17:09:28 +01:00
Stefan Breunig
28a18303f3 implement rcat – fixes #230, fixes #1001 2017-08-03 21:42:35 +02:00
Nick Craig-Wood
3e3a59768e fs/test_all: fix after fstest factorisation 2017-08-03 20:01:05 +01:00
Nick Craig-Wood
e01741b557 fs: Cleaning up directories in test is no longer needed
..as it is done in the finalise method.
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
71fe046937 fs: Add Find method to DirTree 2017-07-26 16:38:53 +01:00
Nick Craig-Wood
d5ff7104e5 fs: Implement NewDirTree for non --fast-list 2017-07-26 16:38:44 +01:00
Nick Craig-Wood
cd4895690a fstest: Factor test initialisation into Initialise() 2017-07-26 16:38:33 +01:00
Nick Craig-Wood
1ecf2bcbd5 fs: fix typo in --bind description 2017-07-23 23:08:33 +01:00
Nick Craig-Wood
c3d6cc91ec Fix --bind flag changes under go1.6
Correcting 9f24639568
2017-07-23 22:36:32 +01:00
Nick Craig-Wood
9f24639568 Add --bind flag for choosing the local addr on outgoing connections - fixes #1087
Supported by all remotes except FTP.
2017-07-23 16:27:39 +01:00
Nick Craig-Wood
7b81f12dad box: add docs
* reorder remotes so they are in alphabetical order by full name everywhere
  * update CONTRIBUTING doc
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
d279161cee Implement box storage remote - #97 2017-07-23 11:32:34 +01:00
Nick Craig-Wood
db6009126d Fix test failure with new stretchr/testify - fixes #1550 2017-07-23 08:59:07 +01:00
Nick Craig-Wood
c8b5ee1e54 Start v1.37-DEV development 2017-07-22 20:43:06 +01:00
Nick Craig-Wood
a73ecec11f Version v1.37 2017-07-22 20:04:29 +01:00
Nick Craig-Wood
0cf19ef66a Make ListDirSorted check for subdirectories and write test 2017-07-19 09:36:27 +01:00
Nick Craig-Wood
655891170f Check in ListDirSorted that the directory entries all belong 2017-07-18 23:39:42 +01:00
Nick Craig-Wood
6f71260acf Add --tpslimit and --tpslimit-burst to limit transactions per second for HTTP
This is useful if you are being rate limited or banned by your cloud
storage provider.
2017-07-16 17:25:39 +01:00
Nick Craig-Wood
62e28d0a72 Replace token bucket limiter github.com/tsenart/tb with golang.org/x/time/rate
In tests tsenart/tb has proved inaccurate at low rates.
2017-07-16 16:14:44 +01:00
Nick Craig-Wood
72add5ab27 sync: state whether duplicates are objects are directories 2017-07-08 15:42:18 +01:00
Nick Craig-Wood
69ff009264 Use a stable sort for sorting directory entries
This is useful if there are duplicates. Assuming the remote delivers
the entries in a consistent order, this will give the best user
experience in syncing as it will consistently use the first entry for
the sync comparison.
2017-07-06 14:07:26 +01:00
Nick Craig-Wood
27b157580e Move make_test_files.go into bin 2017-07-06 11:54:57 +01:00
Nick Craig-Wood
f0512d1a52 Fix missing fs.Dir -> fs.Directory 2017-07-06 11:31:36 +01:00
Nick Craig-Wood
575e779b55 Warn about duplicate files when syncing - fixes #1506
Error about unsorted directories and test thoroughly
2017-06-30 21:24:13 +01:00
Nick Craig-Wood
e7d04fc103 Create fs.Directory interface and use it everywhere 2017-06-30 14:26:59 +01:00
Nick Craig-Wood
e2d7d413ef fs: rename BasicInfo to DirEntry 2017-06-30 14:26:58 +01:00
Nick Craig-Wood
e7e9aa0dfa fs: Remove unused ListFser interface 2017-06-30 14:26:58 +01:00
Nick Craig-Wood
f88300a153 Don't Mkdir at the start of sync - fixes #1131
This is possible now that the bucket based remotes will create the
buckets on demand (9c1e703777).
2017-06-29 12:31:53 +01:00
Nick Craig-Wood
e54087ece1 Fix config tests to save configData which fixes subsequent tests 2017-06-29 12:31:53 +01:00
Nick Craig-Wood
54561fd2bc s3: work around eventual consistency in bucket creation
Deleting a bucket then testing its existence can give the wrong
result.  Work around by keeping a flag as to whether we have deleted
the bucket.
2017-06-29 12:31:52 +01:00
Nick Craig-Wood
f3c7e1a9dd Debug directory creation and removal - fixes #1192 2017-06-27 22:19:35 +01:00
Nick Craig-Wood
aa20486485 Add --stats-log-level so can see --stats without -v - fixes #1180
The most common use for this flag is likely to be showing the stats
without using -v by using `--stats-log-level NOTICE`.
2017-06-26 22:50:37 +01:00
Stefan Breunig
52b042971a keep file permissions and try to keep user/group on supported systems (fixes #1467) 2017-06-25 09:05:24 +02:00
Stefan Breunig
2d2778eabf don't delete remote if name does not change while renaming (fixes #1495) 2017-06-25 08:55:54 +02:00
Vasiliy Tolstov
6b89e6c381 add new http remote filesystem
Signed-off-by: Vasiliy Tolstov <v.tolstov@selfip.ru>
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
b76cd4abd2 Fix Range header option 2017-06-20 21:27:14 +01:00
Nick Craig-Wood
4fd9570332 fs: Use an in place filter in ListDirSorted 2017-06-20 21:27:14 +01:00
Nick Craig-Wood
5250fcdf08 core: fix data race in walk
This was detected by the race detector when the client of Walk() sorted entries.
2017-06-15 16:46:32 +01:00
Nick Craig-Wood
64662bef8d Deprecate --old-sync-method it is replaced with --fast-list
Remove old sync method code.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
0b8d9084fc test_all: print command line so it can be cut and pasted into bash 2017-06-14 16:49:40 +01:00
Nick Craig-Wood
8a6a8b9623 Change List interface and add ListR optional interface
This simplifies the implementation of remotes.  The only required
interface is now `List` which is a simple one level directory list.

Optionally remotes may implement `ListR` if they have an efficient way
of doing a recursive list.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
6fc88ff32e Use --fast-list flag for sync/copy/move - fixes #1277
Redo test framework to take a -fast-list flag and test remotes with that flag.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
50928a5027 Implement --fast-list flag.
This is supported remotes which can do a recursive listing.  It will
use more memory.

This is related to #1277 but doesn't fix that issue yet.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
53c3e5f0ab Add placeholder support for ListR interface.
The ListR interface will be implemented by remotes that can do a
recursive directory listing more efficiently than just recursing
through the directories.  These include the bucket based remotes.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
0edb025257 Fixup tests with dirs vs bucket based fs 2017-06-14 16:49:40 +01:00
Nick Craig-Wood
7e20e16cff core: Implement Walk directory listing and use in place of Lister
This is in preparation for removing the Lister code and replacing the
fundamental operation in the Fs with listing a single directory.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
68333d34a1 dropbox: make setting mod time on existing files work properly
This is a fix left over from the v2 conversion.  Dropbox ignores the
client modification on an incoming file if it was identical to the
existing file.  This change deletes the existing file first before
re-uploading the new one.
2017-06-13 13:58:39 +01:00
Nick Craig-Wood
740b3f6ae2 Fix problems found with ineffassign 2017-06-13 11:52:36 +01:00
Nick Craig-Wood
9a11d3efd9 Revert "Start Cat tests from 2 as onedrive doesn't support ranging from 1"
Now that https://github.com/OneDrive/onedrive-api-docs/issues/543 is
fixed, this can be reverted.

This reverts commit 320c53eab0.
2017-06-10 13:48:00 +01:00
Nick Craig-Wood
ee14efd3c2 config: fix menu selection when no remotes 2017-06-10 11:39:40 +01:00
Nick Craig-Wood
52e1bfae2a oauth: Allow auth_url and token_url to be set in the config file
If set in the config file, these override the ones configured into the
remote.  This enables alternative oauth servers to be used for all
oauth remotes.  This can only be altered by editing the config file
for the moment.
2017-06-08 20:35:32 +01:00
Nick Craig-Wood
b49821956a Fix copyto/moveto test error (see #1261) 2017-06-07 14:08:46 +01:00
Nick Craig-Wood
a61ba1e7c4 moveto, copyto: report transfers and checks as per move and copy 2017-06-07 13:02:21 +01:00
Nick Craig-Wood
d30cc1e119 Factor RemoteSplit into fs 2017-06-07 12:27:33 +01:00
Nick Craig-Wood
74a3dfc4e1 Fix TestHashSums 2017-06-06 23:21:47 +01:00
Nick Craig-Wood
b047402294 config: Fix save of temp file under Windows - fixes #1458 2017-06-01 16:38:19 +01:00
Stefan
9cede6b372 fully write new config file before moving to target location (fixes #1287)
* fully write new config file before moving to target location (fixes #1287)
* do not fail if there is no previous config; print temporary config path on failure
2017-06-01 08:57:10 +01:00
Stefan Breunig
decd960867 make moveto/copyto no-ops when source and destination are the same (fixes #1261) 2017-05-30 23:01:19 +01:00
Nick Craig-Wood
20da3e6352 Add options to Put, PutUnchecked and Update, add HashOption and speed up local
* Add options to Put, PutUnchecked and Update for all Fses
  * Use these to create HashOption
  * Implement this in local
  * Pass the option in fs.Copy

This has the effect that we only calculate hashes we need to in the
local Fs which speeds up transfers significantly.
2017-05-29 12:04:52 +01:00
Nick Craig-Wood
6381959850 dropbox: support Dropbox content hashing scheme - fixes #1302
* add support to hashing module
  * add dbhashsum to list the hashes
  * add support to dropbox module

This means objects up and downloaded to/from Dropbox will have their
hashes checked.

Note after this change local objects are calculating MD5, SHA1 and
DBHASH which is excessive and needs to be fixed.
2017-05-29 12:04:44 +01:00
Stefan Breunig
a2e3af0523 poll for Google Drive changes when mounted 2017-05-28 17:54:52 +01:00
Nick Craig-Wood
88ef475629 config: allow keys to be deleted from the config file 2017-05-25 10:15:22 +01:00
Nick Craig-Wood
ae9f8304fa Attempt to make async buffer test more reliable 2017-05-24 16:24:06 +01:00
Nick Craig-Wood
e0ba1a2cd2 ftp: fix bitrot 2017-05-18 20:49:36 +01:00
Antonio Messina
c72fca2711 Add ftp backend - fixes #540 2017-05-18 20:49:36 +01:00
Nick Craig-Wood
50e79bc087 fs: Implement fs.Trace for tracing entry and exit of functions 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
17633f5460 Require go1.6 for building rclone
This is required because google.golang.org/grpc needs it.
2017-05-11 17:07:49 +01:00
Yoni Jah
cdeeff988e Added RepetableReader to fs. used in OneDrive with io.LimitedReader to display accurate speed 2017-05-02 22:31:05 +01:00
Hraban Luyat
66e8c1600e Print password prompts to stderr
This makes rclone with encrypted config better suited for use in
pipelines. E.g.:

$ rclone lsl mydrive:Some/Dir | sort -k 4

If the password prompt ("Enter configuration password") is printed to
stdout, it will be swallowed by sort. By printing it to stderr, you
still see the prompt, without sacrificing compatibility with the unix
pipeline.
2017-04-22 09:38:39 +01:00
Nick Craig-Wood
bc25190fc7 Fix misleading log message with --dry-run - fixes #1309 2017-04-10 16:07:22 +01:00
marvwatson
4810aa65a4 Update references from HTTP to HTTPS where possible 2017-03-29 05:38:34 -07:00
Nick Craig-Wood
503cd84919 Start v1.36-DEV development 2017-03-18 11:30:59 +00:00
Nick Craig-Wood
118e26f8e2 Version v1.36 2017-03-18 11:16:43 +00:00
Nick Craig-Wood
9b07d32c02 onedrive, drive, amazonclouddrive: make sure we find the root
This fixes copyto copying things to the wrong place - fixes #1231
2017-03-16 09:42:49 +00:00
Nick Craig-Wood
99f7fe736a onedrive: implement Move #197 2017-03-15 14:01:08 +00:00
Nick Craig-Wood
320c53eab0 Start Cat tests from 2 as onedrive doesn't support ranging from 1
This has been reported here: https://github.com/OneDrive/onedrive-api-docs/issues/543
2017-03-12 14:24:33 +00:00
Nick Craig-Wood
4d5b73df85 Fix TestListDirSorted eventual consistency listing problems 2017-03-12 14:00:22 +00:00
Nick Craig-Wood
b2a4ea9304 fs/buffer: Fix panic on concurrent Read/Close - fixes #1213 2017-03-06 19:22:17 +00:00
Nick Craig-Wood
10e532bce9 Fix --files-from with an empty file copying everything - fixes #1196 2017-03-04 10:12:54 +00:00
Nick Craig-Wood
4ab7e05e02 Fix MimeType propagation
In fs.Copy, don't wrap objects if possible, and if not, then add a
MimeType method into the wrapped object.
2017-03-04 10:10:55 +00:00
Nick Craig-Wood
79e3c67bbd local, yandex, dropbox: fix NewObject suceeding on a directory #1079
Add tests to make it consistent across all remotes
2017-02-25 11:09:57 +00:00
Nick Craig-Wood
e2f0feef3c Add debugging to print hash values on failed hash comparison 2017-02-23 11:23:19 +00:00
Nick Craig-Wood
e59dc81658 Stop --track-renames deleting case folded source files - fixes #1094
What was happening is that when Move was implemented as Copy + Delete,
MoveFile was seeing the files didn't need transferring (because they
were identical) then deleted the source.

The fix uses Move instead and patches onedrive to disallow a case
folded identical copy (which errors with 500 error)
2017-02-22 19:28:22 +00:00
Nick Craig-Wood
6b0f2ef4bd Fix --delete-before deleting files on copy - fixes #1166 2017-02-22 13:17:38 +00:00
Hisham Zarka
b3a526814e fix --ignore-checksum 2017-02-18 13:13:53 +04:00
Nick Craig-Wood
033d1eb7af Refactor Account interface 2017-02-17 09:15:24 +00:00
Nick Craig-Wood
ac62ef430d Prevent double closes on async buffer 2017-02-17 08:55:24 +00:00
Nick Craig-Wood
928be0f1fd mount: fix seek with buffering to use correct interface
Stop pre-cache before seeking which stops lots of excess data transfer
2017-02-17 08:55:24 +00:00
Nick Craig-Wood
6f75290678 Make async buffering start slowly to improve seek performance 2017-02-17 08:26:14 +00:00
Nick Craig-Wood
f3c5745468 Add srcRemote and dstRemote parameters to DirMove #954 2017-02-16 17:42:37 +00:00
Nick Craig-Wood
dac4bb22d3 mount: Make include and exclude filters apply to mount - fixes #1060 2017-02-15 23:28:53 +00:00
Nick Craig-Wood
b52c80e85c sync: don't update mod times if --dry-run set - fixes #1100 2017-02-15 23:09:44 +00:00
Nick Craig-Wood
f15c6b68b6 Re-add the async buffer on seek - fixes #1137 2017-02-15 22:54:21 +00:00
Nick Craig-Wood
3f778d70f7 Add sync.Pool to async reader 2017-02-15 22:37:58 +00:00
Nick Craig-Wood
7fa687b3e1 fs: Async buffer: use ReadFill to fill the chunks and increase to 1MB 2017-02-14 22:36:37 +00:00
Nick Craig-Wood
493da54113 Add --buffer-size parameter to control buffer size for copy 2017-02-14 22:36:37 +00:00
Nick Craig-Wood
541929258b check: Add --download flag to check all the data, not just hashes 2017-02-13 10:48:26 +00:00
Nick Craig-Wood
7047c67a5e sync: Fix log message containing <nil> 2017-02-13 09:23:21 +00:00
Nick Craig-Wood
01c747e7db Add cryptcheck command to check integrity of crypt remotes #1102 2017-02-12 16:30:18 +00:00
Nick Craig-Wood
666dae4229 Add --syslog flag to optionally log to syslog on capable platforms 2017-02-11 20:27:46 +00:00
Nick Craig-Wood
ac1c041377 Redo log level flags
* -vv or --log-level DEBUG
  * -v or --log-level INFO
  * --log-level NOTICE (default)
  * -q --log-level ERROR

Replace Config.Verbose and Config.Quiet with Config.LogLevel

Fixes #739 Fixes #1108 Fixes #1000
2017-02-11 20:22:42 +00:00
Nick Craig-Wood
0366ea39c5 Reassign some logging levels 2017-02-11 17:56:05 +00:00
Nick Craig-Wood
80f53176d9 Rename log functions and factor into own file 2017-02-11 17:54:50 +00:00
Nick Craig-Wood
50e190ff54 cat: don't allocate buffers if not needed to reduce memory usage 2017-02-09 11:46:53 +00:00
Nick Craig-Wood
dd20a297d6 cat: Fix go routine leak 2017-02-09 11:25:36 +00:00
Nick Craig-Wood
c0ad29c06c Clarify logging and docs for --no-traverse incompatibilities - fixes #1059 2017-02-08 22:35:12 +00:00
Nick Craig-Wood
d091d4a8bb rclone cat: add --head, --tail, --offset, --count and --discard
Fixes #819
2017-02-08 08:09:41 +00:00
Nick Craig-Wood
48cdedc97b Re-implement sync routine to work a directory at a time
Multiple directories (up to --checkers worth) are scanned at once.

This uses much less memory than the previous scheme - only the amount
of memory needed to hold an entire directory listing of objects.

For directory based remotes the speed is unchanged.

For bucket based remotes, instead of doing one API call to list the
whole bucket, it does multiple calls, one for each pseudo directory.
However these are done in parallel so in practice this seems to speed
up directory listings.

This replaces the existing sync method as it performs faster and uses
less memory.

The old sync method is available with the temporary --old-sync-method
flag.

Fixes #517
Fixes #439
Fixes #236
Fixes #1067
2017-02-04 10:30:25 +00:00
Nick Craig-Wood
7c6cd3a9e1 Make --delete-after the default and refactor --delete-{before,during,after} parsing 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
bcdd73369f Ignore --delete-before with --track-renames 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
86bec20b56 sync: factor accumulating the rename checks 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
c3b2b89473 Add ListDirSorted function to list a directory
* fix error return of readFilesFn also
2017-02-04 10:30:25 +00:00
Nick Craig-Wood
85f05c57d1 Clean empty directories between test runs 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
726cb43be9 Complete SFTP remote #521
* Add unit tests
  * Fix up remote so it passes tests
  * Add docs
2017-02-04 10:29:46 +00:00
Nick Craig-Wood
288302c2cf Make fallback purge delete empty directories too.
This was implemented to make the SFTP unit tests pass.
2017-02-04 10:29:46 +00:00
Jack Schmidt
b9a8315696 Basic SFTP support, Issue #521 2017-02-04 10:29:18 +00:00
Nick Craig-Wood
9d331ce04b Implement --ignore-checksum flag
Fixes #793 Fixes #863 Fixes #981
2017-02-03 08:11:10 +00:00
Nick Craig-Wood
2192805360 rclone config: when choosing from a list, allow the value to be entered 2017-01-29 15:51:26 +00:00
Nick Craig-Wood
db0b93c0ad rclone config: allow rename and copy of remotes - fixes #641 2017-01-29 15:37:44 +00:00
Dario Giovannetti
9d36258923 Comply with XDG Base Directory specification
Fixes #868
2017-01-26 10:22:08 +00:00
Nick Craig-Wood
b6848a3edb Fix race in Lister.Finished which was causing the tests to be unreliable 2017-01-19 20:11:17 +00:00
Nick Craig-Wood
e2bf9eb8e9 Implement --suffix for use with --backup-dir only #98
This also makes sure we remove files we are about to override in the
--backup-dir properly.
2017-01-19 20:11:17 +00:00
Nick Craig-Wood
1fa258c2b4 Define a new Features() method for Fs
Optional interfaces are becoming more important in rclone,
--track-renames and --backup-dir both rely on them.

Up to this point rclone has used interface upgrades to define optional
behaviour on Fs objects.  However when one Fs object wraps another it
is very difficult for this scheme to work accurately.  rclone has
relied on specific error messages being returned when the interface
isn't supported - this is unsatisfactory because it means you have to
call the interface to see whether it is supported.

This change enables accurate detection of optional interfaces by use
of a Features struct as returned by an obligatory Fs.Features()
method.  The Features struct contains flags and function pointers
which can be tested against nil to see whether they can be used.

As a result crypt and hubic can accurately reflect the capabilities of
the underlying Fs they are wrapping.
2017-01-16 17:33:25 +00:00
ncw
3745c526f1 Implement --backup-dir - fixes #98
The parameter of backup-dir specifies a remote that all deleted or
overwritten files will be copied to.
2017-01-16 17:33:25 +00:00
Nick Craig-Wood
c123c702ab Fix fs.Overlapping and factor fs.SameConfig 2017-01-14 09:55:53 +00:00
ncw
4aae7bcca6 Factor server side move detection 2017-01-14 09:55:53 +00:00
Brandur
3b1e0b66bb Return error on not found from ListFn
This changes `ListFn`'s implementation so that if it encounters a not
found error, instead of sending a fatal error to log, it coordinates the
return of the error between checker goroutines and sends it back to the
caller.

The main impetus here is that it allows an external program compiling
against rclone as a package to handle a not found, where it currently it
cannot.

This does change error output on a not found a little bit, we go from
this:

    2017/01/09 21:14:03 directory not found

To this:

    2017/01/09 21:13:44 Failed to ls: directory not found
2017-01-12 10:07:59 +00:00
Nick Craig-Wood
78c1f2839e Fix filters to add ** rules to directory rules
This fixes `--exclude ".*{,/**}"` to exclude all . files and
. directories.
2017-01-05 19:33:49 +00:00
Nick Craig-Wood
de2d967abd Stop --track-renames hashing matching files - fixes #984
Also only hash files of the correct size.

This speeds it up a lot.
2017-01-05 17:58:01 +00:00
Marco Paganini
6611d92e21 Only start bandwidth ticker when necessary.
- Only start the token ticker when the timetable entry has more than one
  entry.
- This fixes the "Scheduled bandwidth change" log message when no
  bwlimit is specified.
- Fixes #987
2017-01-04 19:03:49 -08:00
Nick Craig-Wood
f73ee5eade Make all config file variables be settable in the environment
These are set in the form RCLONE_CONFIG_remote_option where remote is
the uppercased remote name and option is the uppercased config file
option name.  Note that RCLONE_CONFIG_remote_TYPE must be set if
defining a new remote.

Fixes #616
2017-01-03 22:42:47 +00:00
Nick Craig-Wood
0d75d2585f Allow all options to be set from environment variables
The option names are munged changing - to _ making upper case and
prepending RCLONE_.  The values are as parsed by pflag.
2017-01-03 22:42:47 +00:00
Marco Paganini
3b0f944e23 Add time-based bandwidth limits.
- Change the --bwlimit command line parameter to accept both a limit (as
  before) or a full timetable (formatted as "hh:mm,limit
  hh:mm,limit...")
- The timetable is checked once a minute by a ticker function. A new
  tokenBucket is created every time a bandwidth change is necessary.
- This change is compatible with the SIGUSR2 change to toggle bandwidth
  limits.

This resolves #221.
2017-01-03 21:00:38 +00:00
Nick Craig-Wood
f1221b510b Change --track-renames to use the length,hash pair stored in a map
This makes it much faster in the case of many files and use less
memory.

This also detects use of --no-traverse and disables it.
2017-01-03 20:37:06 +00:00
Nick Craig-Wood
274ab349f4 sync: Only allow --track-renames if have a common hash 2017-01-03 20:35:05 +00:00
Bjørn Erik Pedersen
47d3a450a4 sync: Track and perform server-side renames
This commits adds support for tracking of file renames if `track-renames` flag is set,
and it then performs server-side renames for remotes that support it, i.e.
remotes that implement either the `Mover` or the `Copier` interface.

Fixes #888
2017-01-03 20:35:05 +00:00
Nick Craig-Wood
5b8b379feb Version v1.35 2017-01-02 15:33:06 +00:00
Nick Craig-Wood
4dd5428b13 Fix rmdirs test and integration tests which depend on each other 2017-01-02 14:15:07 +00:00
Nick Craig-Wood
64ec220d5d Fix --no-update-modtime test on remotes which don't support hashes 2016-12-31 15:19:26 +00:00
Nick Craig-Wood
cbfec0d281 Fix tests for missing config file 2016-12-20 15:05:08 +00:00
Nick Craig-Wood
1b2dda8c4c oauthutil: Reload config file off disk before updating token
This fixes the config file being overwritten when two rclones are running.

Fixes #682
2016-12-19 15:04:07 +00:00
Nick Craig-Wood
4482e75f38 Fix golint 2016-12-15 21:02:41 +00:00
Nick Craig-Wood
0117aeafbf mount: this removes the async buffering as it was killing seek performance 2016-12-15 17:08:52 +00:00
Nick Craig-Wood
8083804575 Make sure wrapped retry/fatal errors are never nil to avoid panic 2016-12-13 16:02:14 +00:00
Nick Craig-Wood
fafbcc8e2f Make server side move more obvious in debug 2016-12-09 16:57:07 +00:00
Nick Craig-Wood
48a2e3844d Add optional interface DirCacheFlush for making the tests more reliable
This is defined for the users of dircache drive, onedrive, and acd.

This helps fix the DirMove tests on acd.
2016-12-09 16:57:07 +00:00
Nick Craig-Wood
dcf53a1d12 Allows multiple --include/--exclude/--filter options - fixes #875 2016-12-07 13:37:40 +00:00
Nick Craig-Wood
3bdfa284a9 Make rclone lsd obey the filters properly 2016-12-07 11:16:36 +00:00
Nick Craig-Wood
be4fd51289 fuse: Add bandwidth accounting and buffering
This fixes rclone mount ignoring bwlimit and increases buffering which
should speed up transfers greatly.

Fixes #796
Fixes #690
2016-12-04 16:57:47 +00:00
Nick Craig-Wood
c265f451f2 Implement moveto and copyto commands for choosing a destination name on copy/move
Fixes #227
Fixes #476
2016-12-03 23:43:52 +00:00
Nick Craig-Wood
2058652fa4 Allow overlapping remotes in move when DirMove is supported 2016-12-03 09:08:40 +00:00
Nick Craig-Wood
50b3cfccb1 Factor Move out of sync.go and add remote parameter to Move and Copy 2016-12-03 09:08:40 +00:00
Nick Craig-Wood
2756900749 Fix not transferring files that don't differ in size - fixes #911
Due to a logic error files stored on remotes which support modtime but
not hashes weren't being transferred when updating with a file of the
same size but different modtime.  Instead the modtime of the remote
file was being set to that of the local file.

In practice this affected crypt with all remotes except Amazon Drive
and Dropbox.
2016-11-28 17:08:15 +00:00
Nick Craig-Wood
539853df36 Fix rmdirs test 2016-11-28 12:23:24 +00:00
Scott McGillivray
f9df545e3c add --stats-unit option and improve alignment for --stats output 2016-11-28 12:18:30 +00:00
Nick Craig-Wood
f3365dd251 Make rclone rmdirs command to delete empty directories - fixes #831 2016-11-27 11:49:31 +00:00
Nick Craig-Wood
aaa1370a36 Add directory parameter to Rmdir and Mkdir #100 #831
This will enable rclone to manage directories properly in the future.
2016-11-26 12:02:53 +00:00
Nick Craig-Wood
d95288175f Version v1.34 2016-11-06 10:18:30 +00:00
Nick Craig-Wood
01be5bff02 Fix ogier/pflag vs spf13/pflag 2016-11-05 18:18:08 +00:00
Nick Craig-Wood
e825df6448 Fix Check on crypted file systems 2016-11-05 18:17:21 +00:00
Nick Craig-Wood
ff41b0d435 Improve error message when source remote isn't found in sync #848 2016-11-05 18:03:55 +00:00
Nick Craig-Wood
441951a93b Stop removing failed upload to cloud storage remotes - fixes #559
We do remove a partially written file on local so we don't have
corrupted files lying around.
2016-11-04 21:34:25 +00:00
Nick Craig-Wood
318e335137 Remove Authorization: headers from --dump-headers output
Add in `--dump-auth` flag to put it back.
2016-11-02 15:53:43 +00:00
Nick Craig-Wood
d1759fdfa9 Add request ID to HTTP debugging to make it easier to trace concurrent flows 2016-10-31 12:01:28 +00:00
Nick Craig-Wood
e65059e431 Fix non-windows/non-unix builds for bwlimit/SIGUSR2 feature and add a mutex
The race detector complained whenever SIGUSR2 was sent to rclone so
this adds a mutex to prevent concurrent access.
2016-10-30 19:20:16 +00:00
Nick Craig-Wood
5454f2abd0 Fix race in checkServerTime 2016-10-30 19:16:27 +00:00
Marco Paganini
cc4f5ba7ba Add support to toggle bandwidth limits via SIGUSR2.
Sending rclone a SIGUSR2 signal will toggle the limiter between off and
the limit set with the --bwlimit command-line option.
2016-10-30 17:46:59 +00:00
Nick Craig-Wood
de73063977 Fix output of crypt objects in logs 2016-10-20 17:46:51 +01:00
Nick Craig-Wood
b7875fc02a rclone check: show count of hashes that couldn't be checked #700 2016-10-17 16:48:11 +01:00
Nick Craig-Wood
8a56a6836a Check server time against local time #654 2016-10-08 14:00:50 +01:00
Nick Craig-Wood
83849e0a36 Don't show encrypted password to stop confusion - fixes #656 2016-10-08 11:26:14 +01:00
Nick Craig-Wood
d42b38699b Make ResponseHeaderTimeout be --timeout not --contimeout fixes #766
This was causing a problem with Amazon Drive which often pauses for a
long time after uploads before returning the response.
2016-10-08 10:12:19 +01:00
Nick Craig-Wood
d033e92234 Stop single file and --files-from operations iterating through the source bucket.
This works by making sure directory listings that use a filter only
iterate the files provided in the filter (if any).

Single file copies now don't iterate the source or destination
buckets.

Note that this could potentially slow down very long `--files-from`
lists - this is easy to fix (with another flag probably) if it causes
anyone a problem.

Fixes #610
Fixes #769
2016-10-07 11:39:39 +01:00
Nick Craig-Wood
e94850f322 Fix timeouts not working when set to 0 and firing too often - #766 2016-10-06 10:23:23 +01:00
Nick Craig-Wood
aef2ac5c04 Add options for Open and implement Range for all remotes 2016-10-05 21:03:56 +01:00
Nick Craig-Wood
9ea20bac42 Fix accidentally committed test in move code 2016-10-03 20:16:41 +01:00
Nick Craig-Wood
945f49ab5e Make ContentType be preserved for cloud -> cloud copies - fixes #733 2016-10-03 20:02:04 +01:00
Nick Craig-Wood
6c9a258d82 Fix move command
* Delete src files which already existed in dst - fixes #751
  * Fix deletion of src file when dst file older
2016-10-03 19:58:44 +01:00
Nick Craig-Wood
f2eeb4301c Make --dump-bodies imply --dump-headers 2016-09-22 08:40:37 +01:00
Nick Craig-Wood
0caac70994 Fix build for < go1.7 2016-09-13 11:36:14 +01:00
Nick Craig-Wood
83ba59749f Make failed uploads not count as "Transferred" - fixes #708 2016-09-12 18:15:58 +01:00
Nick Craig-Wood
bbf819e2d1 Move versioncheck so it happens earlier in the compile process. 2016-09-12 17:50:19 +01:00
Nick Craig-Wood
0cb9bb3b54 Redo http Transport code
* Insert User-Agent in Transport - fixes #199
  * Update timeouts to use Context
  * Modernise transport
2016-09-12 17:50:19 +01:00
Nick Craig-Wood
2eaac80c86 b2 with crypt: fix crash when uploading large files - fixes #673 2016-09-05 18:10:01 +01:00
Nick Craig-Wood
3996bbb8cb Version v1.33 2016-08-24 23:02:05 +01:00
Nick Craig-Wood
059743a1b0 crypt: add to integration tests 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
a54806e5c1 Fix Move when underlying remote returns ErrorCantMove 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
226c2a0d83 Implement crypt for encrypted remotes - #219 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
9985fc40f4 Make Password parameters obey Optional flag and offer to generate random ones 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
b1de4c8cba Implement password Option and re-implement editing
Editing now shows all the options for the fs and asks one at a time
whether they should be changed.
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
6a4e424630 Re-implement Obscure/Reveal so they use AES-CTR encryption 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
ebb67c135e Fix listToChan passing nil objects to DeleteFile 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
037a000cc8 b2: fix stats accounting for upload - fixes #602 2016-08-22 21:19:38 +01:00
Nick Craig-Wood
84eb7031bb Implement the rclone cat command 2016-08-18 22:45:32 +01:00
Nick Craig-Wood
d7b79b4481 Mark the compiled from source version with -DEV - fixes #627 2016-08-18 21:31:10 +01:00
Nick Craig-Wood
b5faaf7116 Fix double close of abort channel - fixes #592 2016-08-18 18:56:57 +01:00
Nick Craig-Wood
0805ec051f Add BasicInfo interface shared between Dir and Object 2016-08-05 17:45:27 +01:00
Nick Craig-Wood
ae56df7d4f Add --dedupe-mode only to dedupe command 2016-08-04 21:47:14 +01:00
Nick Craig-Wood
67562081f7 Version v1.32 2016-07-13 17:32:39 +01:00
Nick Craig-Wood
63f6827a0d Version v1.31 2016-07-13 12:28:01 +01:00
Nick Craig-Wood
ac3c83f966 Fix integration tests for drive 2016-07-12 21:38:15 +01:00
Nick Craig-Wood
b9c8e61d39 Explicitly check the state in tests after writing files
...otherwise Amazon Drive will fail.
2016-07-12 21:36:39 +01:00
Nick Craig-Wood
a6056408dd Fix move command - stop it running for overlapping fses - fixes #577
* Make move command check for overlapping remotes and refuse to run
  * Do copy/delete rather than all the copies then all the deletes
  * Doesn't purge the source - this was unexpected behaviour see #512 and #416
  * Add -list-retries flag to test suite to control retries

This changes the semantics of `move` slightly.  However it now errs on
the side of not deleting stuff.
2016-07-12 10:49:37 +01:00
Nick Craig-Wood
b9479cf7ab Implement --no-update-modtime flag - fixes #511 2016-07-12 10:46:45 +01:00
Nick Craig-Wood
a20d80565b Tidy stats output - fixes #541 2016-07-11 13:04:30 +01:00
Nick Craig-Wood
4bd9932703 Fix wording in verbose copy logs - fixes #574 2016-07-09 10:11:57 +01:00
Nick Craig-Wood
13591c7c00 Redo error handling for sync/copy/move
* Factor sync/copy/move into its own file
  * Make fatal errors abort the sync
  * Make Copy return errors
  * Make Sync/Copy/Move return the last Copy error if there was one
  * Prioritise returning Fatal errors
  * NoRetry errors are returned if no other types of errors
2016-07-04 13:45:53 +01:00
Nick Craig-Wood
28f4061892 Add two more classes of error Fatal and NoRetry
These are for remotes to signal that they have a fatal error and don't
want to continue (eg cap exceeded) or that a particular file shouldn't
be retried for some reason.
2016-07-04 13:45:52 +01:00
Nick Craig-Wood
0a43ff9c13 Modify interface for accounting to take a string not an fs.Object 2016-07-02 16:58:50 +01:00
Nick Craig-Wood
9aae143833 Implement cleanup command for emptying trash / removing old versions of files 2016-07-01 16:35:36 +01:00
Nick Craig-Wood
70dc97231e Convert more tests to use assert/require 2016-06-30 15:45:30 +01:00
Nick Craig-Wood
af4ef8ad8d Implement --no-traverse flag to stop copy traversing the destination remote.
Refactor sync/copy/move
  * Don't load the src listing unless doing a sync and --delete-before
  * Don't load the dst listing if doing copy/move and --no-traverse is set

`rclone --no-traverse copy src dst` now won't load either of the
listings into memory so will use the minimum amount of memory.

This change will reduce the amount of memory rclone uses dramatically
too as in normal operations (copy without --notraverse or sync) as it
no longer loads the source file listing into memory at all.

Fixes #8
Fixes #544
Fixes #546
2016-06-29 17:38:50 +01:00
Nick Craig-Wood
13797a1fb8 Make retry logs be debug in main copy routine 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
3ad8fb8634 Make DeleteFile and DeleteFiles return errors 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
ab43005422 Make NewObject return an error
* make it return an error
  * make a canonical error fs.ErrorNotFound
  * make a test for it
  * remove logs/debugs of error
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
b1f131964e Rename NewFsObject to NewObject 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
1a87b69376 Get rid of LimitedFs - FIXME needs docs on copying single files
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.

Use this to remove the LimitedFs and just add this file to the
--files-from list.

This means that server side operations can be used also.

Fixes #518
Fixes #545
2016-06-28 08:51:43 +01:00
Nick Craig-Wood
5a3b109e25 Fix issues identified by go vet -shadow - fixes #530 2016-06-21 21:17:52 +01:00
Klaus Post
e0aa4bb492 Fix incomplete local hashes.
Fixes #533
2016-06-19 16:51:49 +02:00
Nick Craig-Wood
bd0227450e Version v1.30 2016-06-18 16:41:46 +01:00
Nick Craig-Wood
f438f1e9ef Fix stats print 2016-06-18 16:41:46 +01:00
Nick Craig-Wood
d3dd672640 Document recursion requirements for Fses 2016-06-18 14:12:47 +01:00
Nick Craig-Wood
2a46be8cf3 b2: implement large file uploading - fixes #456 2016-06-18 13:38:05 +01:00
Nick Craig-Wood
1b4370bde1 Rework retry logic when copying objects
* Fix off by one retry logic - fixes #406
  * Retry any retriable errors
  * Restructure code
2016-06-18 10:55:58 +01:00
Nick Craig-Wood
2cfb3834f2 Log errors with %v 2016-06-18 09:36:47 +01:00
Nick Craig-Wood
46135d830e Add --ignore-size flag - fixes #399 2016-06-17 17:20:08 +01:00
Nick Craig-Wood
df1092ef33 Change Fs.Put so that it must cope with existing files
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.

None of the other Fses needed adjusting.

Fixes #483
2016-06-13 19:29:10 +01:00
Nick Craig-Wood
4c5b2833b3 Convert to using github.com/pkg/errors everywhere 2016-06-13 17:43:03 +01:00
Nick Craig-Wood
7fe653c350 Unwrap errors properly for patform specific connection retry code.
Include more possible errors for Windows.

For #442
2016-06-10 13:48:41 +01:00
Nick Craig-Wood
f17cb1bf50 Fix retry of Windows wsaend errors #442
Make the test for wsaend error less specific
2016-06-09 15:34:13 +01:00
Nick Craig-Wood
67d0375b98 Audit use of log.Print and change to Debug, Log, or ErrorLog as appropriate 2016-06-06 21:23:54 +01:00
Nick Craig-Wood
108760e17b Log -v output to stdout by default - fixes #228 2016-06-04 18:49:27 +01:00
Nick Craig-Wood
e2788aa729 Display the transfer stats in more human readable form - fixes #428 2016-06-03 22:49:50 +01:00
Nick Craig-Wood
772f99fd74 Make SizeSuffix output without b suffix for more useful printouts 2016-06-03 22:49:14 +01:00
Nick Craig-Wood
a21cc161de Make 0 size files specifiable with --max-size 0b - fixes #450 2016-06-03 21:54:27 +01:00
Nick Craig-Wood
e818b7c206 Represent -1 as "off" for SIZE values 2016-06-03 21:51:39 +01:00
Nick Craig-Wood
5723d788a4 Add b suffix so we can specify bytes in --bwlimit, --min-size etc
Fixes #449
2016-06-03 21:16:48 +01:00
Nick Craig-Wood
1d6698a754 Build tweaks - fixes #484
* disable CGO for static builds everywhere
  * override Version in release build script
  * don't output symbol table in release binaries
2016-06-03 20:34:19 +01:00
Nick Craig-Wood
ccdd1ea6c4 Add --max-depth parameter
This will apply to ls/lsd/sync/copy etc

Fixes #412
Fixes #213
2016-06-03 17:05:39 +01:00
Nick Craig-Wood
ad85f6e413 Implement directory include filtering for efficiency
Fixes #395
2016-05-16 17:14:04 +01:00
Nick Craig-Wood
ac9c20b048 Make IsRetryError function 2016-05-14 17:11:19 +01:00
Nick Craig-Wood
2db35f0ce7 Dump out unexpected state in integration test 2016-05-07 21:19:26 +01:00
Nick Craig-Wood
dbfa7031d2 Factor Lister into own file, write tests and fix 2016-05-07 17:17:43 +01:00
Nick Craig-Wood
c2d0e86431 Add more tests for List() and fix resulting problems 2016-05-07 14:50:35 +01:00
Nick Craig-Wood
68ec6a9f5b Add a directory parameter to Fs.List() 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
2016-05-06 16:52:34 +01:00
Nick Craig-Wood
1752ee3c8b Retry errors which indicate the connection closed prematurely.
See discussion in #442
2016-04-29 17:29:34 +01:00
Nick Craig-Wood
5c2d8ffe33 Retry only the failing tests in the integration tests 2016-04-26 10:20:07 +01:00
Klaus Post
6c0c9abd57 Use "password:" instead of "password>" prompt
Fixes #410
2016-04-21 19:39:46 +01:00
Nick Craig-Wood
157d7d45f5 Version v1.29 2016-04-18 18:30:29 +01:00
Nick Craig-Wood
b5cba73cc3 Make test more reliable 2016-04-18 17:48:52 +01:00
Nick Craig-Wood
0f8e7c3843 Make rclone check obey the --size-only flag - fixes #419 2016-04-07 15:01:45 +01:00
Nick Craig-Wood
b835330714 Use "application/octet-stream" if mime.TypeByExtension returns invalid type
Fixes #424
2016-04-07 14:32:01 +01:00
Klaus Post
8f4d6973fb Fix missing "quit" option when there are no remotes. 2016-03-23 16:57:56 +00:00
Nick Craig-Wood
ced3a4bc19 Implement -I, --ignore-times for unconditional upload - fixes #311 2016-03-22 17:02:27 +00:00
Nick Craig-Wood
cb22583212 b2: Enable mod time syncing - fixes #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
8ef551bf9c Make dedupe remove identical copies without asking and add non interactive mode - fixes #338
* Now removes identical copies without asking
  * Now obeys `--dry-run`
  * Implement `--dedupe-mode` for non interactive running
    * `--dedupe-mode interactive` - interactive the default.
    * `--dedupe-mode skip` - removes identical files then skips anything left.
    * `--dedupe-mode first` - removes identical files then keeps the first one.
    * `--dedupe-mode newest` - removes identical files then keeps the newest one.
    * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
    * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
  * Add tests which will only run on Google Drive.
2016-03-06 18:15:01 +00:00
Nick Craig-Wood
874a64e5f6 A script to make a directory heirarchy for testing 2016-03-05 20:26:15 +00:00
Nick Craig-Wood
3c7f80f58f Version v1.28 2016-03-01 09:00:01 +00:00
Nick Craig-Wood
229ea3f86c Stop --update tests running on remotes which don't do mod time 2016-03-01 07:26:33 +00:00
Nick Craig-Wood
41eb386063 Reset password/config path in config tests to fix other tests 2016-02-29 21:43:37 +00:00
Nick Craig-Wood
dfc7cd97a3 Optionally disable gzip compression on downloads with --no-gzip-encoding - fixes #353 2016-02-29 19:48:54 +00:00
Nick Craig-Wood
280ac26464 Implement -u/--update so creation times can be used on all remotes - #226 2016-02-29 17:46:40 +00:00
Nick Craig-Wood
88cca8a6eb Simplify literals (after running gofmt -s over the code) 2016-02-29 16:57:23 +00:00
Nick Craig-Wood
3343c1afa4 Don't make directories if --dry-run set - fixes #342 2016-02-28 19:56:50 +00:00
Nick Craig-Wood
867a26fe4f Implement --low-level-retries flag - fixes #266 2016-02-25 22:58:21 +00:00
Nick Craig-Wood
3890105cdc Add -run-only flag to run_all test 2016-02-25 22:05:57 +00:00
Nick Craig-Wood
d2219a800a Fix and document the move command - fixes #334
* Don't attempt to use server side Move unless they are on the same Fs
  * Fix move in the presense of filters
2016-02-25 20:05:34 +00:00
Nick Craig-Wood
ccb59480bd Add InActive method to Filter to detect when no fiters are in use. 2016-02-25 19:58:00 +00:00
Nick Craig-Wood
95e4d837ef Make config chooser easier to understand 2016-02-21 13:40:16 +00:00
Nick Craig-Wood
e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
Nick Craig-Wood
700e47d6e2 Stub out ReadPassword on plan9 and solaris to fix compilation 2016-02-21 10:31:53 +00:00
klauspost
ef06371c93 Create separate interface for object information.
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
2016-02-19 13:31:09 +00:00
klauspost
84b00b362f Change back to original goconfig package.
Add documentation for `--ask-password`.
2016-02-17 11:45:05 +01:00
klauspost
bfd7601cf9 Add configuration file encryption
See #317 for details.

Use `rclone config` to add/change/remove password.

Tests that loads the default configuration will now fail with a better error message, and add a switch that makes it possible to disable password prompts and fail instead.

Make it possible to use the "RCLONE_CONFIG_PASS" environment variable as password for configuration.
2016-02-16 16:32:05 +01:00
Nick Craig-Wood
a6320bbad3 Fix delete command to wait until all finished - fixes missing deletes.
This also could affect deletes at the end of the sync command.
2016-02-15 16:43:59 +00:00
Nick Craig-Wood
85d99de26b Fix typo in error strings 2016-02-09 16:15:50 +00:00
Nick Craig-Wood
e62c0a58a7 Version 1.27 2016-01-31 17:50:13 +00:00
Nick Craig-Wood
0f73129ab7 dedupe command to deduplicate a remote. Useful with google drive - fixes #41 2016-01-31 16:09:42 +00:00
Nick Craig-Wood
1373efaa39 Delete command which does obey the filters - fixes #327 2016-01-31 16:06:04 +00:00
Nick Craig-Wood
5c37b777fc Make the --dry-run warnings into logs so they appear without the -v flag 2016-01-31 16:06:04 +00:00
Nick Craig-Wood
8c211fc8df Warn the user about files with same name but different case
Relates to #107 & #119.
2016-01-26 16:57:09 +00:00
Nick Craig-Wood
74a71f7824 Add tests for --delete-before, --delete-during and --delete-after 2016-01-26 16:57:09 +00:00
Nick Craig-Wood
12b51c5eb8 Remove duplicate check for filter IncludeObject 2016-01-26 16:57:09 +00:00
klauspost
14069fd8e6 Implement --delete-before, --delete-during, --delete-after - fixes #252. 2016-01-26 16:57:09 +00:00
Nick Craig-Wood
cd62f41606 Reduce number of logs and show hash type where appropriate 2016-01-24 18:06:57 +00:00
Nick Craig-Wood
109d4ee490 Prefix all test remotes with rclone-test- and make names more pronouncable 2016-01-24 12:37:46 +00:00
Nick Craig-Wood
18ebec8276 Check remote is empty between integration tests 2016-01-24 12:37:19 +00:00
Nick Craig-Wood
eda4130703 Fix integration tests so they can be run independently and out of order - fixes #291
* Make all integration tests start with an empty remote
  * Add an -individual flag so this can be a different bucket/container/directory
  * Fix up tests after changing the hashers
  * Add sha1sum test
  * Make directory checking in tests sleep more to fix acd inconsistencies
  * Factor integration tests to make more maintainable
  * Ensure remote writes have a fstest.CheckItems() before use
    * this fixes eventual consistency on the directory listings later
  * Call fs.Stats.ResetCounters() before every fs.Sync()

Note that the tests shouldn't be run concurrently as fs.Config is global state.
2016-01-23 17:02:09 +00:00
Nick Craig-Wood
27f67edb1a Fix formatting problem in sha1sum 2016-01-17 13:56:42 +00:00
Nick Craig-Wood
3ffea738e6 Make hash constants start from 1 not 2 2016-01-17 10:47:24 +00:00
Nick Craig-Wood
e8daab7971 Fix integration tests for remotes with unsupported hash schemes 2016-01-16 09:45:15 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types.
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
Nick Craig-Wood
af5f4ee724 Make --include rules add their implict exclude * at the end of the filter list
This means you can mix `--include` and `--include-from` with the
other filters (eg `--exclude`) but you must include all the files you
want in the include statement.

Fixes #280
2016-01-10 11:42:53 +00:00
klauspost
25f22ec561 Add "--ignore-existing" flag.
Add option to completely ignore existing files and not consider them for transfer.

Fixes #274
2016-01-08 08:20:04 +00:00
Nick Craig-Wood
5189231a34 Tweaks to rclone authorize
* Document the headless / remote setup procedure
  * Move Config constants into fs
  * Parse arguments in main for Authorize
2016-01-07 20:31:23 +00:00
klauspost
bcbd30bb8a Add easier headless configuration.
This will allow setting up a remote with copy&paste of values to a headless machine. It will allow copy+pasting a token into the configuration.

This requires rclone to be on a machine with a proper browser. Custom client id and secrets are supported.

To test token generation, use `rclone auth "fs type"`.
2016-01-07 20:31:23 +00:00
Nick Craig-Wood
1ce3673006 Add -clean flag to test_all.go to clean left over test directories 2016-01-03 21:49:26 +00:00
Nick Craig-Wood
3423de65fa Make canonical place for all fs in fs/all/all.go 2016-01-03 14:12:45 +00:00
Nick Craig-Wood
77fb8ac240 Version 1.26 2016-01-02 12:04:32 +00:00
dibu28
3ac4407b88 Implement Yandex storage backend - fixes #234 2015-12-30 12:11:46 +00:00
Nick Craig-Wood
8ea0d5212f Add -verbose flag to test_all and fix tries count 2015-12-30 11:34:22 +00:00
Nick Craig-Wood
33b3eea6ec Implement Backblaze B2 - fixes #224 2015-12-30 10:05:07 +00:00
Nick Craig-Wood
113624691a Add -dump-headers and -dump-bodies flags for operations test debugging 2015-12-30 09:35:35 +00:00
Nick Craig-Wood
ddf39f2d57 Replace test_all.sh with test_all.go which is cross platform and parallel 2015-12-30 09:26:34 +00:00
Nick Craig-Wood
e69e181090 Fix --min-age and --max-age when only one is present 2015-12-17 14:22:43 +00:00
Nick Craig-Wood
ac65d8369e Make fs.CheckClose public to stop duplication 2015-12-17 13:30:58 +00:00
Nick Craig-Wood
8057d668bb Fix crash in http logging - fixes #223
A nil-pointer exception was caused if the http transaction ever
resulted in a go error while using `--dump-bodies`.  Now don't ignore
the error and log it instead of the http body.
2015-12-17 13:30:58 +00:00
Nick Craig-Wood
36f1bc4a8a Make ls/lsl/md5sum/size/check obey includes and excludes - fixes #169
* run check directory listings concurrently
2015-12-17 13:30:58 +00:00
Adriano Aurélio Meirelles
3cbd57d9ad Add support to filter files based on their age 2015-12-17 09:52:38 -02:00
Nick Craig-Wood
7f0020a407 Version v1.25 2015-11-14 13:06:39 +00:00
Nick Craig-Wood
fcea3777c0 Implement Hubic storage system - fixes #200 2015-11-14 08:08:52 +00:00
Nick Craig-Wood
bed01a303f Add UnWrapper interface and implement in LimitedFs 2015-11-14 08:08:51 +00:00
Nick Craig-Wood
2c2cb84ca7 Make it so optional interface Purge can fail so it can be wrapped 2015-11-14 08:08:51 +00:00
Nick Craig-Wood
e9dda25c60 Implement Move in limited fs 2015-11-14 08:08:51 +00:00
Nick Craig-Wood
80ffbade22 Fix deletion of some excluded files without --delete-excluded #205
This only happened if the destination file was present but the source
file was missing.
2015-11-12 11:46:04 +00:00
Nick Craig-Wood
4276abc58b Version v1.24 2015-11-07 16:23:12 +00:00
Nick Craig-Wood
d42cb11b84 Fix tests to run all tests again and add onedrive 2015-11-07 11:21:15 +00:00
Nick Craig-Wood
365b4babae Make filter test files pass errcheck 2015-11-07 10:27:47 +00:00
Nick Craig-Wood
2fcb8f5db7 Add support for Microsoft One Drive - fixes #10
* Still to do
    * Copy
    * Move
    * MoveDir
2015-11-07 10:19:10 +00:00
Nick Craig-Wood
ff16e0f6df Factor common error handling into fs module 2015-11-06 15:26:58 +00:00
Nick Craig-Wood
ed72c678f8 Protect accounting from being closed twice 2015-11-06 15:26:58 +00:00
klauspost
b85ddc4e4f Extend CI tests to include formatting checks.
CI tests now tests 'go vet', 'go fmt' (via goimports) and golint.

Adds Travis experimental OSX support.
2015-11-03 13:50:29 +01:00
klauspost
b872ff0237 Add option to disable server certificate verification.
The option name mirrors the 'wget' option (also `--no-check-certificate`). The cURL equivalent is called `--insecure`, which is a bit unclear.

Put in the "developers" section in documentation with proper warnings.

Fixes #168
2015-10-29 16:42:25 +01:00
Nick Craig-Wood
1b95718460 Fix typos in filter docs and unit test assertions 2015-10-20 09:16:47 +01:00
Nick Craig-Wood
4e952af614 Allow spaces in remotes and check remote names for validity at creation time - fixes #171 2015-10-12 17:54:09 +01:00
Klaus Post
6344c3051c Add async readahead buffer
This adds an async read buffer of 4x4MB when copying files >10MB.

This fixes #164 and reduces the number of IO operations for copy/move.
2015-10-12 08:30:27 +01:00
Nick Craig-Wood
3a900e5bb7 Version v1.23 2015-10-03 16:24:07 +01:00
Nick Craig-Wood
b4d7741611 Improve output of --dump-headers 2015-10-03 16:04:51 +01:00
Nick Craig-Wood
177dbbc29a Implement rclone size for measuring remotes - fixes #153 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
57aad81b68 Version v1.22 2015-09-28 19:38:20 +01:00
Nick Craig-Wood
a91bcaaeb0 Implement rsync like include and exclude - fixes #27
* Implement include/exclude
  * Implement rsync compatible file globbing
  * Implement command line filtering flags
    * --delete-excluded - Delete files on dest excluded from sync
    * --filter - Add a file-filtering rule
    * --filter-from - Read filtering patterns from a file
    * --exclude - Exclude files matching pattern
    * --exclude-from - Read exclude patterns from file
    * --include - Include files matching pattern
    * --include-from - Read include patterns from file
    * --files-from - Read list of source-file nam
    * --min-size - Don't transfer any file smaller than this in k or suffix k|M|G
    * --max-size - Don't transfer any file larger than this in k or suffix k|M|G
  * Document
2015-09-28 19:18:21 +01:00
Nick Craig-Wood
5d4b149884 Version v1.21 2015-09-22 21:05:11 +01:00
Nick Craig-Wood
5f20ae707d Make lsl output times in localtime and fix tests - fixes #141 2015-09-22 19:04:12 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2ed158aba3 Fixes from go vet and errcheck 2015-09-22 07:31:12 +01:00
Nick Craig-Wood
e391311512 gofmt 2015-09-17 18:42:39 +01:00
klauspost
788ef76f1c Show more of the filename and align output.
Print more of the file name, and make the output aligned, so it is nicer on frequent updates.
2015-09-17 15:53:45 +02:00
klauspost
0a5870208e Display individual transfer progress
Improve progress printing by displaying individual file progress, as well
as a moving average speed with ETA. Example output:

2015/09/15 16:38:21
Transferred:    183599104 Bytes (4646.49 kByte/s)
Errors:                 0
Checks:                 1
Transferred:            0
Elapsed time:       38.5s
Transferring:
 * 01_06_14.mp3: 33% done. avg: 1280.5, cur: 1288.8 kByte/s. ETA: 1m12s
 * 01_12_15.mp3: 33% done. avg: 1002.2, cur:  943.4 kByte/s. ETA: 1m17s
 * 01_13_14.mp3: 48% done. avg: 1456.8, cur: 1425.2 kByte/s. ETA: 39s
 * 01_19_15.mp3: 28% done. avg: 1226.9, cur: 1114.4 kByte/s. ETA: 1m37s
2015-09-16 19:42:46 +01:00
Nick Craig-Wood
34193fd8d9 Version v1.20 2015-09-15 07:42:31 +01:00
Nick Craig-Wood
a288c2b3a3 Make a retry error wrapper for a plain error 2015-09-09 23:22:41 +01:00
Nick Craig-Wood
fec16b0ac8 acd: Skip test on FS which don't support ModifiedTime 2015-09-09 23:21:50 +01:00
Nick Craig-Wood
dd8717797e Implement --dump-headers and --dump-bodies debug flags 2015-09-08 21:02:48 +01:00
Nick Craig-Wood
8c3df224ef Implement Amazon Cloud Drive - fixes #45
* Optional interfaces Copier, Mover, DirMover not done
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
c98a51b26c Lightly obscure secrets 2015-09-01 22:33:34 +01:00
Nick Craig-Wood
8d33ce0154 Check for source and dest being the same in sync/copy/move 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
92745aa950 Add Root() to Fs interface 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
0168f55f3e Switch to spf13 fork of pflag - fixes #116
This supports --long value as well as --long=value which is as
expected for a unix utility.
2015-08-29 18:14:24 +01:00
Nick Craig-Wood
9ed2de3d6e Version v1.19 2015-08-28 09:47:13 +01:00
Nick Craig-Wood
59ba8f28c8 Implement move command - fixes #35
* Define Mover interface to move a single object
  * Define DirMover interface to move a directory
  * Implement DirMove operation
  * Add `rclone move` command
  * Tests for Dir Move

To Do
  * Implement Move, DirMover in local, drive, dropbox
  * unit test for Mover
  * unit test for DirMover
2015-08-28 08:49:16 +01:00
Nick Craig-Wood
5c226e91c0 Ignore rmdir in limited fs rather than throwing error - fixes #112 2015-08-25 19:16:25 +01:00
Nick Craig-Wood
a96b522958 Implement server side copies if possible - fixes #99
Add optional fs.Copier interface

Implemented for
  * swift
  * s3
  * drive
  * dropbox
  * google cloud storage
2015-08-23 21:18:38 +01:00
Nick Craig-Wood
fedf81c2b7 Add Name() to Fs interface to return name as passed to NewFs 2015-08-23 13:36:38 +01:00
Nick Craig-Wood
0c6f816a49 Implement --retries flag - fixes #109 2015-08-20 21:07:00 +01:00
Nick Craig-Wood
dfe771fb0c Correct log messages for remotes which don't support modtime/md5sum 2015-08-20 20:48:58 +01:00
Nick Craig-Wood
2fcc18779b Version v1.18 2015-08-17 17:59:37 +01:00
Nick Craig-Wood
1d9e76bb0f dropbox: remove datastore - Fixes #55 #84
This means that dropbox no longer stores MD5SUMs and modified times.

Fixup the tests so that blank MD5SUMs are ignored, and that if
Precision is set to a fs.ModTimeNotSupported, ModTimes can be ignored too.

This opens the door for other FSs which don't support metadata easily.
2015-08-17 17:26:36 +01:00
Nick Craig-Wood
771e60bd07 Show errors when reading the config file 2015-08-15 17:15:02 +01:00
Leonid Shalupov
3fcff32524 do not print stats in quiet mode - fixes #70
...unless had some errors or stats interval requested.

Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
2015-08-10 11:17:54 +01:00
Nick Craig-Wood
9d9a17547a Version v1.17 2015-06-14 15:36:16 +01:00
Leonid Shalupov
bd5f685d0a fix TestLsLong on non-UTC timezones 2015-06-14 15:23:56 +01:00
Nick Craig-Wood
539ad44757 Version v1.16 2015-06-09 18:00:33 +01:00
Nick Craig-Wood
74994a2ec1 Fix uploading big files which was causing timeouts or panics
The symtom was one of these two on upload of files only
  * panic: d.nx != 0 in crypto/md5.(*digest).checkSum
  * read tcp: i/o timeout

It turned out to be a combination of two upstream bugs

  * 5a2187309e
  * https://groups.google.com/forum/#!topic/golang-dev/0Nl6k5Sj6UU

This commit contains a work-around for the second problem, I've fixed
the first and had the change accepted upstream.
2015-06-09 17:32:45 +01:00
Nick Craig-Wood
97dced6a0b Don't check md5sum after download with --size-only - fixes #75 2015-06-09 13:18:40 +01:00
Nick Craig-Wood
e04acb09ce Version v1.15 2015-06-06 15:45:00 +01:00
Nick Craig-Wood
90744301d3 Fix package docs so they appear in godoc correctly 2015-06-06 14:24:30 +01:00
Nick Craig-Wood
e22b445cff Implement --size-only flag to sync on size not checksum & modtime - fixes #75 2015-06-06 08:49:01 +01:00
Alex Couper
7af1282375 Add --checksum flag to only discard transfers by MD5SUM - fixes #61
Useful for copying between backends where checksum fetching is fast,
ie any of s3, swift, drive or googlecloudstorage.
2015-06-05 19:46:03 +01:00
Nick Craig-Wood
d9fcc32f70 Version v1.14 2015-05-21 20:13:40 +01:00
Nick Craig-Wood
88293a4b8a Version v1.13 2015-05-10 12:39:06 +01:00
Nick Craig-Wood
1d254a3674 Implement --timeout and --conntimeout - fixes #54
NB dropbox still to do
2015-05-10 11:29:55 +01:00
Nick Craig-Wood
ba2091725e Version v1.12 2015-03-15 15:55:38 +00:00
Nick Craig-Wood
5cc5429f99 drive: switch to insert from update when a failed copy deletes the upload 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
09d71239b6 Make file size render more neatly and prevent from being < 0 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
c643e4585e core: Log duplicate files if they are detected 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
873db29391 Log all objects more informatively 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
ecb3c7bcef drive, googlecloudstorage: remove SeekWrapper after googleapi fix 2015-03-04 20:47:59 +00:00
Nick Craig-Wood
80000b904c Version v1.11 2015-03-04 17:59:31 +00:00
Nick Craig-Wood
b4a0941d4c In remote paths, change native directory separators to / - fixes #37 2015-03-02 17:04:34 +00:00
Nick Craig-Wood
46d39ebaf7 Factor Mime Type guessing into fs.MimeType() 2015-03-02 09:21:15 +00:00
Nick Craig-Wood
fe68737268 Fix niggles found by go vet 2015-02-28 15:35:54 +00:00
Nick Craig-Wood
2360bf907a Add synchronization to list output to stop corruptions - fixes #29 2015-02-28 15:30:40 +00:00
Nick Craig-Wood
a287e3ced7 Implement --bwlimit to limit data transfer bandwidth 2015-02-27 15:03:47 +00:00
Nick Craig-Wood
cf208ad21b Version v1.10 2015-02-12 18:00:20 +00:00
Nick Craig-Wood
8d1c0ad07c Fix config loop - fixes #25 2015-02-10 16:48:04 +00:00
Nick Craig-Wood
165e89c266 Version v1.09 2015-02-07 22:44:23 +00:00
Nick Craig-Wood
b4e19cfd62 windows: make tests work properly 2015-02-07 22:32:51 +00:00
Nick Craig-Wood
20ad96f3cd windows: Stop drive letters (eg C:) getting mixed up with remotes (eg drive:)
This was done by stopping the user configuring single letter remotes
and making sure we don't treat single letter remotes as a remote name,
but as a drive letter.
2015-02-07 22:32:51 +00:00
Nick Craig-Wood
5fb6f94579 drive: fix rate limit exceeded errors - fixes #20
This is done by pacing the requests to drive and backing them off
using an exponential increase.  Put and Modify operations can now be
retried also.
2015-02-07 22:32:51 +00:00
Nick Craig-Wood
18a8a61cc5 Release v1.08 2015-02-04 22:31:56 +00:00
Nick Craig-Wood
cfdb48c864 Version v1.07 2014-12-23 11:26:32 +00:00
Nick Craig-Wood
14567952b3 google cloud storage: Fix memory leak - fixes #17
This was the same problem as issue #5 (which affected google drive)
2014-12-23 11:03:34 +00:00
Nick Craig-Wood
439a126af6 Version v1.06 2014-12-12 20:13:03 +00:00
Nick Craig-Wood
af9c447146 Fix "Couldn't find home directory" on OSX - fixes #15 2014-12-12 19:18:23 +00:00
Nick Craig-Wood
839133c5e1 Version v1.05 2014-08-09 17:22:17 +01:00
Nick Craig-Wood
18439cf2d7 Move rclonetest into go tests for fs module 2014-08-03 11:18:25 +01:00
Nick Craig-Wood
3e27ff1b95 Add Root List test and fs.Limited tests for single files 2014-07-31 21:35:29 +01:00
Nick Craig-Wood
ff91698fb5 Skip tests if test remote not configured 2014-07-31 08:51:39 +01:00
Nick Craig-Wood
2f9f9afac2 fs: Document that Purger returns error on empty directory, test and fix 2014-07-29 17:18:22 +01:00
Nick Craig-Wood
17ffb0855f Fixes after running errcheck 2014-07-25 18:19:49 +01:00
Nick Craig-Wood
a5b0d88608 Make tests for each Fs
Factor tests out of rclonetest
2014-07-24 23:13:32 +01:00
Nick Craig-Wood
e09ef62d5b core: Fix race detected by go race detector 2014-07-22 23:03:14 +01:00
Nick Craig-Wood
a75bc0703f Version 1.04 2014-07-21 21:32:37 +01:00
Nick Craig-Wood
54cd46372a Version 1.03 2014-07-20 11:28:50 +01:00
Nick Craig-Wood
9aa4b6bd9b Version 1.02 2014-07-19 13:24:48 +01:00
Nick Craig-Wood
e559194fb2 fs: Verify sizes are the same after transfer in Copy() 2014-07-19 13:05:07 +01:00
Nick Craig-Wood
8a76568ea8 core: Verify MD5 sums after each transfer 2014-07-15 19:27:05 +01:00
Nick Craig-Wood
c1a245d1c8 Factor UserAgent to fs and move Version to fs 2014-07-13 19:19:58 +01:00
Nick Craig-Wood
eb2b4ea8aa rclone: Don't purge if --dry-run set 2014-07-13 10:54:30 +01:00
Nick Craig-Wood
e055ed0489 rclone: change "ls" and add "lsl" and "md5sum" commands
Changed "ls" command not to show modification time by default only
size and path.  That is because it is slow for nearly all the remotes
as it requires extra metadata lookup.  All remotes can look up files
and sizes without extra operations.

Added "lsl" which does what "ls" used to - namely show modification
time, size and path.

Added "md5sum" which produces the same output as the md5sum command -
md5sums and paths that is.
2014-07-12 12:09:20 +01:00
Nick Craig-Wood
7ebf48ef42 Fix --dry-run not working and add tests for it - fixes #3 2014-06-26 15:33:06 +01:00
Nick Craig-Wood
1d67b014cb Make compatible with go 1.1 - fixes #1 2014-06-26 15:18:48 +01:00
Nick Craig-Wood
99695d57ab Implement single file operations for all file systems 2014-05-05 22:17:57 +01:00
Nick Craig-Wood
8fd59f2e7d drive: Use o.Update and fs.Put to optimise transfers 2014-04-18 17:49:01 +01:00
Nick Craig-Wood
02afcb00e9 Factor Fs.Put into Object.Update and call Update rather than Put if possible 2014-04-18 17:49:01 +01:00
Nick Craig-Wood
d6a5bfe2d4 Get rid of fs.CopyFs and replace with fs.Sync in preparation for Object.Update 2014-04-18 17:48:46 +01:00
Nick Craig-Wood
92ec29fe3f Factor the generic code into fs and add some more intefaces 2014-03-28 22:26:42 +00:00
Nick Craig-Wood
b0064e6405 Change remote format one last time 2014-03-27 17:31:57 +00:00
Nick Craig-Wood
3eda36f6da Factor rclone main() in preparation for tests 2014-03-27 17:26:24 +00:00
Nick Craig-Wood
a628bef9c2 Use pflag for gnu style flags 2014-03-27 16:55:29 +00:00
Nick Craig-Wood
1b3a49929b Add ability for FS to have a Config helper function run 2014-03-16 13:54:43 +00:00
Nick Craig-Wood
1ea9972be7 Set strict permissions on the config file 2014-03-16 13:53:51 +00:00
Nick Craig-Wood
7d786204b4 Add -config option to specify a config file 2014-03-15 17:01:13 +00:00
Nick Craig-Wood
b3f1a45bbf Config fixes
* Fix empty config configuration
  * Alter menus when no remotes
  * Save config file after delete remote
2014-03-15 16:52:51 +00:00
Nick Craig-Wood
0a108832e2 Remove filesystem flags and put in config file with editor 2014-03-15 16:06:11 +00:00
Nick Craig-Wood
754c54b64f Fix differences print out on check 2013-07-02 08:33:17 +01:00
Nick Craig-Wood
43be26d1f8 Make local filesystem go last 2013-06-29 12:15:55 +01:00
Nick Craig-Wood
b1f681b912 Rename fs.FsX to fs.X 2013-06-28 08:57:32 +01:00
Nick Craig-Wood
6085dc1b5a Refactor into sub filesystems 2013-06-27 22:17:51 +01:00