Nick Craig-Wood
a5c3bcc9c7
fshttp: fix idle timeouts for HTTP connections #2057
...
Now we only nudge on the idle timeout after a successful Read or Write
which returns some bytes and no errors.
2018-02-16 10:35:41 +00:00
Bjørn Erik Pedersen
ab8c0a81fa
Add a delete threshold to sync (--max-delete)
...
Fixes #959
2018-02-12 11:29:58 +00:00
Nick Craig-Wood
d0d6b83a7a
fs/accounting: rework to enable accounting to work with crypt and b2
...
This removes the old system of part accounting and replaces it with a
system of popping off the accounting reader and wrapping up new ones
as necessary.
This makes it much easier to carry the context down the chain of
wrapped readers and get the limiting as near as possible to the
output. This makes the accounting more accurate and the bandwidth
limiting smoother.
Fixes #2029 and Fixes #1443
2018-02-02 15:14:41 +00:00
Nick Craig-Wood
bea02fcf52
fs/accounting: factor into separate files without changing functionality
2018-02-02 15:14:40 +00:00
Nick Craig-Wood
6fb868e00c
config: fix --log-level flag after code reorganization - fixes #2043
2018-02-02 14:07:44 +00:00
Nick Craig-Wood
8df78f2b6d
operations: ignore size of objects when they are < 0 #320
...
This allows google docs to be transferred and checked correctly.
2018-01-31 16:22:05 +00:00
Nick Craig-Wood
2eb5cfb7ad
fs: Formalize the ObjectUnWrapper interface
2018-01-31 16:21:41 +00:00
Nick Craig-Wood
fe52502f19
fs: Adjust RangeOption.Decode to return -1 for read to end
...
A Range request can never request 0 bytes however this change was made
to make a clearer signal that the limit means read to the end.
Add test and more documentation and fixup uses
2018-01-27 14:31:29 +00:00
Nick Craig-Wood
9a73688e3a
fs: Add ParseRangeOption to parse incoming Range: requests
2018-01-27 13:16:37 +00:00
Nick Craig-Wood
bc3ee977f4
fs/hash: move interface assertion to tests so it doesn't pull in spf13/flag
2018-01-26 14:35:18 +00:00
Nick Craig-Wood
a6833b68ca
local: factor RangeOption code to Decode() method and readers.LimitedReadCloser #1825
2018-01-22 19:44:00 +00:00
Nick Craig-Wood
d876392d15
onedrive: Factor code into fs.FixRangeOption
2018-01-22 17:05:00 +00:00
Nick Craig-Wood
84bc4dc142
Clarify RangeOption semantics
2018-01-21 09:51:28 +00:00
Nick Craig-Wood
ee66419a27
fs/fserrors: Add test for error from #1964
2018-01-19 17:07:40 +00:00
Nick Craig-Wood
5ad226ab54
fs: Add dir option to fs.Purge #1891
...
Purge optional interface needs fixing too.
2018-01-19 11:45:50 +00:00
Nick Craig-Wood
97c414f025
config/hash: rename more symbols after factoring into own package
2018-01-18 20:27:52 +00:00
Nick Craig-Wood
71722b5b95
config: factor Obscure and Reveal into its own package
2018-01-18 20:19:55 +00:00
Ernest Borowski
5cce74d630
flags: remove --no-traverse flag because it is obsolete - fixes #1813
...
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2018-01-18 11:00:25 +00:00
Iakov Davydov
acd55a8f65
local, fs: --exclude-if-present ignores directories which it doesn't have permission for - fixes #1959
2018-01-16 20:00:16 +00:00
Nick Craig-Wood
11da2a6c9b
Break the fs package up into smaller parts.
...
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.
The new code layout is documented in CONTRIBUTING.
2018-01-15 17:51:14 +00:00
Nick Craig-Wood
b8b620f5c2
Move all backends into backend directory
2018-01-12 17:07:38 +00:00
Nick Craig-Wood
712e6a8085
lsf: fix integration tests
2018-01-11 13:52:15 +00:00
Will Gunn
5c7e8d5a2b
fs: Add --stats-file-name-length to specify the printed file name length for stats
...
Fixes #1206
2018-01-10 20:32:36 +00:00
Nick Craig-Wood
c74c3b37da
lsf: add option to print hashes
2018-01-06 17:53:37 +00:00
Nick Craig-Wood
7c71ee1a5b
fs: fix TestListFormat on remotes which return 0 as dir size not -1
2018-01-06 17:47:42 +00:00
Jakub Tasiemski
0d041602cf
cmd: new command lsf
2018-01-06 14:39:31 +00:00
Nick Craig-Wood
b412c745a1
Start v1.39-DEV development
2017-12-23 13:40:28 +00:00
Nick Craig-Wood
f34a9116d4
Version v1.39
2017-12-23 13:07:45 +00:00
Nick Craig-Wood
7e585cda96
fs: fix TestRmdirsLeaveRoot test
2017-12-14 08:57:28 +00:00
lewapm
9c242edc10
rmdirs: add --leave-root flag - fixes #1874
2017-12-13 10:23:54 +00:00
ishuah
b05e472d2e
stats: condensed transfer output to fit 80x25 terminals
2017-12-09 10:48:36 +03:00
remusb
25b073c767
fs: add Wrap feature for FS to identify their parent FS ( #1884 )
2017-12-06 17:14:34 +02:00
Nick Craig-Wood
fa4a25a73b
fs: only test one level of cache
...
Can't test multiple caches at once as can only have 1 DB open at once
2017-12-04 15:50:59 +00:00
Ernest Borowski
934df67aef
filter: warn the user if he use --include and --exclude together fixes #1764
...
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2017-12-04 14:20:01 +00:00
Nick Craig-Wood
4b185355df
fs: rcat - use in memory object and Copy for more reliable transfers
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
7d15c33e42
fs: make Copy and Move return the destination object if possible
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
11332a19a0
fs: make an in memory object for short transfers
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
639e812789
fs: integration tests: add SUMMARY heading for log scraping
2017-11-29 15:55:37 +00:00
Nick Craig-Wood
1c6cad2252
fs: integration tests: add 30 minute timeout per test
2017-11-29 13:51:17 +00:00
ishuah
aab8051f50
move: add --delete-empty-src-dirs flag - fixes #1854
2017-11-28 11:38:19 +03:00
Nick Craig-Wood
fdb01437d8
fs: Allow the http Transport to have an optional filter request function
2017-11-24 09:07:56 +00:00
Nick Craig-Wood
02ffd43572
fs: Save the config before asking for a token - fixes #1220
...
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.
This means the post config is done after the user has confirmed the
config is OK which can't be helped.
2017-11-23 14:01:32 +00:00
Nick Craig-Wood
e53892f53b
fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use
...
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer. Since we know the size
of the chunks, pre-allocating it should be much more efficient.
RepeatableReaderBuffer uses the buffer passed in.
RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
2017-11-23 13:53:46 +00:00
ishuah
6c62fced60
move: fixed root source directories getting deleted after move - fixes #1849
2017-11-23 12:01:35 +03:00
Nick Craig-Wood
159fce0106
fs: fix --cache-dir to have some effect
2017-11-22 17:05:02 +00:00
Nick Craig-Wood
cbe5d7ce64
fs: Remove X-Auth-Token: from headers when dumping for swift
2017-11-21 17:32:07 +00:00
Nick Craig-Wood
1a65a4e769
fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters
...
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain
* headers - HTTP headers as before
* bodies - HTTP bodies as before
* requests - HTTP request bodies
* responses - HTTP response bodies
* auth - HTTP auth
* filters - Filter rexeps
Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
2017-11-21 17:32:07 +00:00
ishuah
b4aa920a3d
stats: show the amount of data transferred in kb/mb - fixes #1167
2017-11-21 12:40:02 +03:00
Nick Craig-Wood
8951875c21
vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables
2017-11-20 12:46:21 +00:00
Nick Craig-Wood
e946a8eab0
fs: Add CacheDir config variable
2017-11-20 12:00:32 +00:00
Nick Craig-Wood
46947b3b9b
rcat: fix goroutine leak
...
This was leaking goroutines in the short file case beause it wasn't
calling Close() on the Account object. This became apparent when
testing with mount.
2017-11-20 12:00:32 +00:00
Iakov Davydov
7d3abdc463
tests for --exclude-if-present
2017-11-17 21:51:11 +00:00
Iakov Davydov
538246f6c3
support exclude file in --fast-list mode
2017-11-17 21:51:11 +00:00
Iakov Davydov
557dd8f031
ListDirSorted check for excludefile
2017-11-17 21:51:11 +00:00
Iakov Davydov
37aaa19f3a
new option: --exclude-if-present
2017-11-17 21:51:11 +00:00
Iakov Davydov
cef2e3bf83
path -> startPath in walkRDirTree (we need the path package)
2017-11-17 21:51:11 +00:00
Iakov Davydov
a3a436ce16
WalkRDirTree: return error if unknown item type
2017-11-17 21:51:11 +00:00
Iakov Davydov
5d05df3124
ListContainsExcludeFile: checks for exclude file in the list
2017-11-17 21:51:11 +00:00
Iakov Davydov
421ba84e12
DirTree.Prune: deletes several directories
2017-11-17 21:51:11 +00:00
Iakov Davydov
7ae7080824
FileExists check if a file exists
2017-11-17 21:51:11 +00:00
Remus Bunduc
016abf825e
cache: first version
2017-11-15 15:23:21 +00:00
ishuah
aec2265be0
rclone: implement exit codes - #1136
2017-11-15 17:48:37 +03:00
Substantiel
2423fa40e2
config: add password sub command for setting obscured passwords
2017-11-15 14:44:45 +00:00
ishuah
c65734ee69
move: delete source directory after successful move - fixes #1642
2017-11-07 22:21:38 +00:00
Substantiel
7749157596
Add --auto-confirm flag
2017-11-05 21:56:50 +00:00
Nick Craig-Wood
cc7ed13b9b
fs: factor test running code into fstest/run.go
2017-11-03 12:59:58 +00:00
ishuah
10f27e2ff2
allow trailing+leading whitespace for passwords - #1717
...
warn users when they enter passwords with leading/trailing whitespaces
Updated config_test.go, removing deprecated test case and updated TestReveal
2017-10-20 15:46:17 +01:00
Nick Craig-Wood
115053930e
Make error messages less crypting when revealing an unobscured password - fixes #1743
2017-10-16 22:03:06 +01:00
Nick Craig-Wood
edfab09eb9
config: add sub commands for full config file management
...
Previously config sub commands were manually parsed rather than using
cobra.
Make config command have the following sub commands:
* create Create a new remote with name, type and options.
* delete Delete an existing remote <name>.
* dump Dump the config file as JSON.
* edit Enter an interactive configuration session.
* file Show path of configuration file in use.
* providers List in JSON format all the providers and options.
* show Print (decrypted) config file, or the config for a single remote.
* update Update options in an existing remote.
The following changes were made to existing commands
* listproviders was renamed to providers
* listoptions was removed in favour of providing the output in providers
* jsonconfig was renamed to create
* an optional parameter was added to the show command
2017-10-14 11:50:41 +01:00
thierry
0575623dff
Add config listproviders, listoptions, jsonconfig for automated config
...
Addition of a method listing the providers, a method listing the
options of a provider and method of manual configuration.
2017-10-13 17:17:36 +01:00
Nick Craig-Wood
fc8b13c993
moveto/copyto: Fix to allow copying to the same name - fixes #1736
2017-10-12 20:45:36 +01:00
Nick Craig-Wood
43ced30f11
fs: Add more errors to retry - fixes #1733
2017-10-10 19:51:02 +01:00
Nick Craig-Wood
3eb7f52e39
fs: Add "unexpected EOF reading trailer" as a retriable error - fixes #1730
2017-10-09 17:29:16 +01:00
Nick Craig-Wood
7f3dc9b5c4
Implement WebDAV remote #580
...
This has special knowledge of Owncloud and Nextcloud to enable more
functionality such as mod times.
2017-10-09 16:19:37 +01:00
Nick Craig-Wood
00fe6d95da
fs: fix duplicate files causing spurious copies
...
Before this fix duplicate files (on Google Drive) caused the next file
to be spuriously copied. `rclone dedupe` worked around the problem.
2017-10-02 16:52:53 +01:00
Nick Craig-Wood
9e9297838f
Implement pcloud remote - #418
2017-10-01 11:37:35 +01:00
Nick Craig-Wood
1c10497b68
Start v1.38-DEV development
2017-09-30 15:16:09 +01:00
Nick Craig-Wood
d96e45ba5b
Version v1.38
2017-09-30 14:20:43 +01:00
Nick Craig-Wood
657b3a674d
fs: fix test_all -clean to run just one cleaning thread per remote
2017-09-30 11:07:09 +01:00
Nick Craig-Wood
2857ed5c35
fs: fix --immutable tests on remotes which don't have modtime
2017-09-28 08:56:30 +01:00
Jacob McNamee
2d8e75cab4
Implement --immutable option
2017-09-24 20:00:00 +01:00
Stefan
85877f3adc
config: add show/file subcommands which print the config/its path ( fixes #1086 )
2017-09-19 17:59:19 +02:00
Nick Craig-Wood
87335de8a8
fs: fix filename normalization issues in the tests when running on OS X
2017-09-17 15:31:22 +01:00
Stefan Breunig
12405f9f41
fuse: re-use rcat to support uploads for all remotes ( fixes #1672 )
2017-09-16 22:49:08 +02:00
Stefan
234bfae0d5
b2: implement streaming upload of files with unknown length (see #1614 ) ( closes #1686 )
2017-09-16 22:43:48 +02:00
Nick Craig-Wood
4ac9a65049
fs: stop normalizing file names but do a normalized compare in the sync
...
This works by using a transform function to transform file names when
doing a compare when matching file names in a directory. rclone now
UTF-8 normalizes the file names and does a case insensitive compare if
the destination remote is case insensitive.
This deprecates the --local-no-unicode-normalization flag.
Fixes #1477
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
a8e41f081c
fs: re-implement check and cryptcheck using the same traversal as sync
...
This makes them 100% consistent with sync and also make them use less
memory as they no longer build the whole tree in memory first.
Fixes #1657
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
261c7ad9e4
fs: make syncCopyMove use context for go routine cancellation
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
fe96d5cf0a
fs: factor multiple directory traverse out of sync
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
6df12b3f00
fs: improve retriable error detection
2017-09-16 19:48:49 +01:00
Stefan Breunig
9d3f11b493
amazonclouddrive, rcat: ensure rcat integration test passes even with AmazonCloudDrive ( fixes : #1680 )
2017-09-15 18:09:04 +02:00
Nick Craig-Wood
798502b204
fs: add more errors to be considered temporary errors
...
This makes a framework for adding temporary errors identified by
syscall number or by error string.
Fixes #1660
2017-09-14 18:01:43 +01:00
Stefan Breunig
56dedc49e3
rcat: properly report if the upload fails
2017-09-13 20:21:52 +02:00
Girish Ramakrishnan
4c45cbea18
copy: error out if dst could not be listed
2017-09-12 09:29:44 +01:00
Stefan Breunig
a91448c83a
rcat: honor --dry-run even for small files
2017-09-11 22:28:16 +02:00
Stefan Breunig
80b1f2a494
rcat: configurable small files cutoff and implement proper upload verification
2017-09-11 08:26:53 +02:00
Stefan Breunig
57817397a0
rcat: directly upload small files without streaming them
2017-09-11 08:25:34 +02:00
Nick Craig-Wood
cbea06026a
Make check obey --ignore-size - fixes #1643
2017-09-01 11:20:41 +01:00
Nick Craig-Wood
2c8d6e86cc
fs: fix gofmt
2017-08-31 16:01:19 +01:00
cbruegg
bb6300b032
Fix bwlimit toggle in conjunction with schedules ( Fixes #1607 )
2017-08-31 15:33:29 +01:00
Stefan Breunig
a122b9fa7a
yandex: implement streaming uploads (see #1614 )
2017-08-19 14:07:23 +02:00
Alex McGrath Kraak
2cc9071791
http: add --user-agent option. close #1557
2017-08-17 06:49:27 +02:00
Nick Craig-Wood
265fb8a5e2
fs: Manage empty directories - fixes #100
...
During the sync we collect a list of directories which should be empty
and attempt to rmdir them at the end of the sync. If the directories
are not empty then the rmdir will fail, logging a message but not
erroring the sync.
2017-08-09 21:07:00 +01:00
Nick Craig-Wood
8a1a900733
fstest: use Feature.CanHaveEmptyDirectories to sharpen tests
...
Now we actually test whether the directories are present or not,
filtering out empty directories in the test using the
CanHaveEmptyDirectories flag.
2017-08-09 20:55:08 +01:00
Nick Craig-Wood
20ae7d562b
fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes
2017-08-09 20:55:08 +01:00
Nick Craig-Wood
ec2ea37ad2
fs: Add --disable flag to disable optional features - fixes #1551
...
Eg to disable server side copy use `--disable copy`, to see a list of
what you can disable, `--disable help`.
2017-08-07 21:34:45 +01:00
Nick Craig-Wood
bfe812ea6b
dedupe: implement merging of duplicate directories - fixes #1243
2017-08-07 15:36:41 +01:00
Nick Craig-Wood
db1995e63a
Add MergeDirs optional interface and implement it for drive
2017-08-07 15:32:47 +01:00
Nick Craig-Wood
81a2ab599f
fs: add optional ID to fs.Directory and set it in the remotes which care
2017-08-07 15:31:22 +01:00
Stefan Breunig
40f24e0ea3
config: use absolute ConfigPath to ensure newly written config is on the same mount - fixes #1569
2017-08-05 12:13:25 +02:00
Nick Craig-Wood
92d2e1f8d7
azureblob: rework and complete #801
...
* Fixup bitrot (rclone and Azure library)
* Implement Copy
* Add modtime to metadata under mtime key as RFC3339Nano
* Make multipart upload work
* Make it pass the integration tests
* Fix uploading of zero length blobs
* Rename to azureblob as it seems likely we will do azurefile
* Add docs
2017-08-04 22:56:16 +01:00
Andrei Dragomir
036fd61a50
Added Azure Blob storage support #801
2017-08-04 22:54:27 +01:00
wuyu
ec5b72f8d5
Add new QingStor remote
...
Add new package qingstor to support QingStor API.
Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.
Add new docs for it.
2017-08-04 17:25:47 +01:00
Nick Craig-Wood
f682002b84
fs: Make tests create a new bucket rather than purging the old one
...
This enables QingStor to pass the tests as it has a 2 minute lockout
on deleting the old bucket then creating it again.
2017-08-04 17:09:28 +01:00
Stefan Breunig
28a18303f3
implement rcat – fixes #230 , fixes #1001
2017-08-03 21:42:35 +02:00
Nick Craig-Wood
3e3a59768e
fs/test_all: fix after fstest factorisation
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
e01741b557
fs: Cleaning up directories in test is no longer needed
...
..as it is done in the finalise method.
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
71fe046937
fs: Add Find method to DirTree
2017-07-26 16:38:53 +01:00
Nick Craig-Wood
d5ff7104e5
fs: Implement NewDirTree for non --fast-list
2017-07-26 16:38:44 +01:00
Nick Craig-Wood
cd4895690a
fstest: Factor test initialisation into Initialise()
2017-07-26 16:38:33 +01:00
Nick Craig-Wood
1ecf2bcbd5
fs: fix typo in --bind description
2017-07-23 23:08:33 +01:00
Nick Craig-Wood
c3d6cc91ec
Fix --bind flag changes under go1.6
...
Correcting 9f24639568
2017-07-23 22:36:32 +01:00
Nick Craig-Wood
9f24639568
Add --bind flag for choosing the local addr on outgoing connections - fixes #1087
...
Supported by all remotes except FTP.
2017-07-23 16:27:39 +01:00
Nick Craig-Wood
7b81f12dad
box: add docs
...
* reorder remotes so they are in alphabetical order by full name everywhere
* update CONTRIBUTING doc
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
d279161cee
Implement box storage remote - #97
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
db6009126d
Fix test failure with new stretchr/testify - fixes #1550
2017-07-23 08:59:07 +01:00
Nick Craig-Wood
c8b5ee1e54
Start v1.37-DEV development
2017-07-22 20:43:06 +01:00
Nick Craig-Wood
a73ecec11f
Version v1.37
2017-07-22 20:04:29 +01:00
Nick Craig-Wood
0cf19ef66a
Make ListDirSorted check for subdirectories and write test
2017-07-19 09:36:27 +01:00
Nick Craig-Wood
655891170f
Check in ListDirSorted that the directory entries all belong
2017-07-18 23:39:42 +01:00
Nick Craig-Wood
6f71260acf
Add --tpslimit and --tpslimit-burst to limit transactions per second for HTTP
...
This is useful if you are being rate limited or banned by your cloud
storage provider.
2017-07-16 17:25:39 +01:00
Nick Craig-Wood
62e28d0a72
Replace token bucket limiter github.com/tsenart/tb with golang.org/x/time/rate
...
In tests tsenart/tb has proved inaccurate at low rates.
2017-07-16 16:14:44 +01:00
Nick Craig-Wood
72add5ab27
sync: state whether duplicates are objects are directories
2017-07-08 15:42:18 +01:00
Nick Craig-Wood
69ff009264
Use a stable sort for sorting directory entries
...
This is useful if there are duplicates. Assuming the remote delivers
the entries in a consistent order, this will give the best user
experience in syncing as it will consistently use the first entry for
the sync comparison.
2017-07-06 14:07:26 +01:00
Nick Craig-Wood
27b157580e
Move make_test_files.go into bin
2017-07-06 11:54:57 +01:00
Nick Craig-Wood
f0512d1a52
Fix missing fs.Dir -> fs.Directory
2017-07-06 11:31:36 +01:00
Nick Craig-Wood
575e779b55
Warn about duplicate files when syncing - fixes #1506
...
Error about unsorted directories and test thoroughly
2017-06-30 21:24:13 +01:00
Nick Craig-Wood
e7d04fc103
Create fs.Directory interface and use it everywhere
2017-06-30 14:26:59 +01:00
Nick Craig-Wood
e2d7d413ef
fs: rename BasicInfo to DirEntry
2017-06-30 14:26:58 +01:00
Nick Craig-Wood
e7e9aa0dfa
fs: Remove unused ListFser interface
2017-06-30 14:26:58 +01:00
Nick Craig-Wood
f88300a153
Don't Mkdir at the start of sync - fixes #1131
...
This is possible now that the bucket based remotes will create the
buckets on demand (9c1e703777
).
2017-06-29 12:31:53 +01:00
Nick Craig-Wood
e54087ece1
Fix config tests to save configData which fixes subsequent tests
2017-06-29 12:31:53 +01:00
Nick Craig-Wood
54561fd2bc
s3: work around eventual consistency in bucket creation
...
Deleting a bucket then testing its existence can give the wrong
result. Work around by keeping a flag as to whether we have deleted
the bucket.
2017-06-29 12:31:52 +01:00
Nick Craig-Wood
f3c7e1a9dd
Debug directory creation and removal - fixes #1192
2017-06-27 22:19:35 +01:00
Nick Craig-Wood
aa20486485
Add --stats-log-level so can see --stats without -v - fixes #1180
...
The most common use for this flag is likely to be showing the stats
without using -v by using `--stats-log-level NOTICE`.
2017-06-26 22:50:37 +01:00
Stefan Breunig
52b042971a
keep file permissions and try to keep user/group on supported systems ( fixes #1467 )
2017-06-25 09:05:24 +02:00
Stefan Breunig
2d2778eabf
don't delete remote if name does not change while renaming ( fixes #1495 )
2017-06-25 08:55:54 +02:00
Vasiliy Tolstov
6b89e6c381
add new http remote filesystem
...
Signed-off-by: Vasiliy Tolstov <v.tolstov@selfip.ru>
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
b76cd4abd2
Fix Range header option
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
4fd9570332
fs: Use an in place filter in ListDirSorted
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
5250fcdf08
core: fix data race in walk
...
This was detected by the race detector when the client of Walk() sorted entries.
2017-06-15 16:46:32 +01:00
Nick Craig-Wood
64662bef8d
Deprecate --old-sync-method it is replaced with --fast-list
...
Remove old sync method code.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
0b8d9084fc
test_all: print command line so it can be cut and pasted into bash
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
8a6a8b9623
Change List interface and add ListR optional interface
...
This simplifies the implementation of remotes. The only required
interface is now `List` which is a simple one level directory list.
Optionally remotes may implement `ListR` if they have an efficient way
of doing a recursive list.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
6fc88ff32e
Use --fast-list flag for sync/copy/move - fixes #1277
...
Redo test framework to take a -fast-list flag and test remotes with that flag.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
50928a5027
Implement --fast-list flag.
...
This is supported remotes which can do a recursive listing. It will
use more memory.
This is related to #1277 but doesn't fix that issue yet.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
53c3e5f0ab
Add placeholder support for ListR interface.
...
The ListR interface will be implemented by remotes that can do a
recursive directory listing more efficiently than just recursing
through the directories. These include the bucket based remotes.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
0edb025257
Fixup tests with dirs vs bucket based fs
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
7e20e16cff
core: Implement Walk directory listing and use in place of Lister
...
This is in preparation for removing the Lister code and replacing the
fundamental operation in the Fs with listing a single directory.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
68333d34a1
dropbox: make setting mod time on existing files work properly
...
This is a fix left over from the v2 conversion. Dropbox ignores the
client modification on an incoming file if it was identical to the
existing file. This change deletes the existing file first before
re-uploading the new one.
2017-06-13 13:58:39 +01:00
Nick Craig-Wood
740b3f6ae2
Fix problems found with ineffassign
2017-06-13 11:52:36 +01:00
Nick Craig-Wood
9a11d3efd9
Revert "Start Cat tests from 2 as onedrive doesn't support ranging from 1"
...
Now that https://github.com/OneDrive/onedrive-api-docs/issues/543 is
fixed, this can be reverted.
This reverts commit 320c53eab0
.
2017-06-10 13:48:00 +01:00
Nick Craig-Wood
ee14efd3c2
config: fix menu selection when no remotes
2017-06-10 11:39:40 +01:00
Nick Craig-Wood
52e1bfae2a
oauth: Allow auth_url and token_url to be set in the config file
...
If set in the config file, these override the ones configured into the
remote. This enables alternative oauth servers to be used for all
oauth remotes. This can only be altered by editing the config file
for the moment.
2017-06-08 20:35:32 +01:00
Nick Craig-Wood
b49821956a
Fix copyto/moveto test error (see #1261 )
2017-06-07 14:08:46 +01:00
Nick Craig-Wood
a61ba1e7c4
moveto, copyto: report transfers and checks as per move and copy
2017-06-07 13:02:21 +01:00
Nick Craig-Wood
d30cc1e119
Factor RemoteSplit into fs
2017-06-07 12:27:33 +01:00
Nick Craig-Wood
74a3dfc4e1
Fix TestHashSums
2017-06-06 23:21:47 +01:00
Nick Craig-Wood
b047402294
config: Fix save of temp file under Windows - fixes #1458
2017-06-01 16:38:19 +01:00
Stefan
9cede6b372
fully write new config file before moving to target location ( fixes #1287 )
...
* fully write new config file before moving to target location (fixes #1287 )
* do not fail if there is no previous config; print temporary config path on failure
2017-06-01 08:57:10 +01:00
Stefan Breunig
decd960867
make moveto/copyto no-ops when source and destination are the same ( fixes #1261 )
2017-05-30 23:01:19 +01:00
Nick Craig-Wood
20da3e6352
Add options to Put, PutUnchecked and Update, add HashOption and speed up local
...
* Add options to Put, PutUnchecked and Update for all Fses
* Use these to create HashOption
* Implement this in local
* Pass the option in fs.Copy
This has the effect that we only calculate hashes we need to in the
local Fs which speeds up transfers significantly.
2017-05-29 12:04:52 +01:00
Nick Craig-Wood
6381959850
dropbox: support Dropbox content hashing scheme - fixes #1302
...
* add support to hashing module
* add dbhashsum to list the hashes
* add support to dropbox module
This means objects up and downloaded to/from Dropbox will have their
hashes checked.
Note after this change local objects are calculating MD5, SHA1 and
DBHASH which is excessive and needs to be fixed.
2017-05-29 12:04:44 +01:00
Stefan Breunig
a2e3af0523
poll for Google Drive changes when mounted
2017-05-28 17:54:52 +01:00
Nick Craig-Wood
88ef475629
config: allow keys to be deleted from the config file
2017-05-25 10:15:22 +01:00
Nick Craig-Wood
ae9f8304fa
Attempt to make async buffer test more reliable
2017-05-24 16:24:06 +01:00
Nick Craig-Wood
e0ba1a2cd2
ftp: fix bitrot
2017-05-18 20:49:36 +01:00
Antonio Messina
c72fca2711
Add ftp backend - fixes #540
2017-05-18 20:49:36 +01:00
Nick Craig-Wood
50e79bc087
fs: Implement fs.Trace for tracing entry and exit of functions
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
17633f5460
Require go1.6 for building rclone
...
This is required because google.golang.org/grpc needs it.
2017-05-11 17:07:49 +01:00
Yoni Jah
cdeeff988e
Added RepetableReader to fs. used in OneDrive with io.LimitedReader to display accurate speed
2017-05-02 22:31:05 +01:00
Hraban Luyat
66e8c1600e
Print password prompts to stderr
...
This makes rclone with encrypted config better suited for use in
pipelines. E.g.:
$ rclone lsl mydrive:Some/Dir | sort -k 4
If the password prompt ("Enter configuration password") is printed to
stdout, it will be swallowed by sort. By printing it to stderr, you
still see the prompt, without sacrificing compatibility with the unix
pipeline.
2017-04-22 09:38:39 +01:00
Nick Craig-Wood
bc25190fc7
Fix misleading log message with --dry-run - fixes #1309
2017-04-10 16:07:22 +01:00
marvwatson
4810aa65a4
Update references from HTTP to HTTPS where possible
2017-03-29 05:38:34 -07:00
Nick Craig-Wood
503cd84919
Start v1.36-DEV development
2017-03-18 11:30:59 +00:00
Nick Craig-Wood
118e26f8e2
Version v1.36
2017-03-18 11:16:43 +00:00
Nick Craig-Wood
9b07d32c02
onedrive, drive, amazonclouddrive: make sure we find the root
...
This fixes copyto copying things to the wrong place - fixes #1231
2017-03-16 09:42:49 +00:00
Nick Craig-Wood
99f7fe736a
onedrive: implement Move #197
2017-03-15 14:01:08 +00:00
Nick Craig-Wood
320c53eab0
Start Cat tests from 2 as onedrive doesn't support ranging from 1
...
This has been reported here: https://github.com/OneDrive/onedrive-api-docs/issues/543
2017-03-12 14:24:33 +00:00
Nick Craig-Wood
4d5b73df85
Fix TestListDirSorted eventual consistency listing problems
2017-03-12 14:00:22 +00:00
Nick Craig-Wood
b2a4ea9304
fs/buffer: Fix panic on concurrent Read/Close - fixes #1213
2017-03-06 19:22:17 +00:00
Nick Craig-Wood
10e532bce9
Fix --files-from with an empty file copying everything - fixes #1196
2017-03-04 10:12:54 +00:00
Nick Craig-Wood
4ab7e05e02
Fix MimeType propagation
...
In fs.Copy, don't wrap objects if possible, and if not, then add a
MimeType method into the wrapped object.
2017-03-04 10:10:55 +00:00
Nick Craig-Wood
79e3c67bbd
local, yandex, dropbox: fix NewObject suceeding on a directory #1079
...
Add tests to make it consistent across all remotes
2017-02-25 11:09:57 +00:00
Nick Craig-Wood
e2f0feef3c
Add debugging to print hash values on failed hash comparison
2017-02-23 11:23:19 +00:00
Nick Craig-Wood
e59dc81658
Stop --track-renames deleting case folded source files - fixes #1094
...
What was happening is that when Move was implemented as Copy + Delete,
MoveFile was seeing the files didn't need transferring (because they
were identical) then deleted the source.
The fix uses Move instead and patches onedrive to disallow a case
folded identical copy (which errors with 500 error)
2017-02-22 19:28:22 +00:00
Nick Craig-Wood
6b0f2ef4bd
Fix --delete-before deleting files on copy - fixes #1166
2017-02-22 13:17:38 +00:00
Hisham Zarka
b3a526814e
fix --ignore-checksum
2017-02-18 13:13:53 +04:00
Nick Craig-Wood
033d1eb7af
Refactor Account interface
2017-02-17 09:15:24 +00:00
Nick Craig-Wood
ac62ef430d
Prevent double closes on async buffer
2017-02-17 08:55:24 +00:00
Nick Craig-Wood
928be0f1fd
mount: fix seek with buffering to use correct interface
...
Stop pre-cache before seeking which stops lots of excess data transfer
2017-02-17 08:55:24 +00:00
Nick Craig-Wood
6f75290678
Make async buffering start slowly to improve seek performance
2017-02-17 08:26:14 +00:00
Nick Craig-Wood
f3c5745468
Add srcRemote and dstRemote parameters to DirMove #954
2017-02-16 17:42:37 +00:00
Nick Craig-Wood
dac4bb22d3
mount: Make include and exclude filters apply to mount - fixes #1060
2017-02-15 23:28:53 +00:00
Nick Craig-Wood
b52c80e85c
sync: don't update mod times if --dry-run set - fixes #1100
2017-02-15 23:09:44 +00:00
Nick Craig-Wood
f15c6b68b6
Re-add the async buffer on seek - fixes #1137
2017-02-15 22:54:21 +00:00
Nick Craig-Wood
3f778d70f7
Add sync.Pool to async reader
2017-02-15 22:37:58 +00:00
Nick Craig-Wood
7fa687b3e1
fs: Async buffer: use ReadFill to fill the chunks and increase to 1MB
2017-02-14 22:36:37 +00:00
Nick Craig-Wood
493da54113
Add --buffer-size parameter to control buffer size for copy
2017-02-14 22:36:37 +00:00
Nick Craig-Wood
541929258b
check: Add --download flag to check all the data, not just hashes
2017-02-13 10:48:26 +00:00
Nick Craig-Wood
7047c67a5e
sync: Fix log message containing <nil>
2017-02-13 09:23:21 +00:00
Nick Craig-Wood
01c747e7db
Add cryptcheck command to check integrity of crypt remotes #1102
2017-02-12 16:30:18 +00:00
Nick Craig-Wood
666dae4229
Add --syslog flag to optionally log to syslog on capable platforms
2017-02-11 20:27:46 +00:00
Nick Craig-Wood
ac1c041377
Redo log level flags
...
* -vv or --log-level DEBUG
* -v or --log-level INFO
* --log-level NOTICE (default)
* -q --log-level ERROR
Replace Config.Verbose and Config.Quiet with Config.LogLevel
Fixes #739 Fixes #1108 Fixes #1000
2017-02-11 20:22:42 +00:00
Nick Craig-Wood
0366ea39c5
Reassign some logging levels
2017-02-11 17:56:05 +00:00
Nick Craig-Wood
80f53176d9
Rename log functions and factor into own file
2017-02-11 17:54:50 +00:00
Nick Craig-Wood
50e190ff54
cat: don't allocate buffers if not needed to reduce memory usage
2017-02-09 11:46:53 +00:00
Nick Craig-Wood
dd20a297d6
cat: Fix go routine leak
2017-02-09 11:25:36 +00:00
Nick Craig-Wood
c0ad29c06c
Clarify logging and docs for --no-traverse incompatibilities - fixes #1059
2017-02-08 22:35:12 +00:00
Nick Craig-Wood
d091d4a8bb
rclone cat: add --head, --tail, --offset, --count and --discard
...
Fixes #819
2017-02-08 08:09:41 +00:00
Nick Craig-Wood
48cdedc97b
Re-implement sync routine to work a directory at a time
...
Multiple directories (up to --checkers worth) are scanned at once.
This uses much less memory than the previous scheme - only the amount
of memory needed to hold an entire directory listing of objects.
For directory based remotes the speed is unchanged.
For bucket based remotes, instead of doing one API call to list the
whole bucket, it does multiple calls, one for each pseudo directory.
However these are done in parallel so in practice this seems to speed
up directory listings.
This replaces the existing sync method as it performs faster and uses
less memory.
The old sync method is available with the temporary --old-sync-method
flag.
Fixes #517
Fixes #439
Fixes #236
Fixes #1067
2017-02-04 10:30:25 +00:00
Nick Craig-Wood
7c6cd3a9e1
Make --delete-after the default and refactor --delete-{before,during,after} parsing
2017-02-04 10:30:25 +00:00
Nick Craig-Wood
bcdd73369f
Ignore --delete-before with --track-renames
2017-02-04 10:30:25 +00:00
Nick Craig-Wood
86bec20b56
sync: factor accumulating the rename checks
2017-02-04 10:30:25 +00:00
Nick Craig-Wood
c3b2b89473
Add ListDirSorted function to list a directory
...
* fix error return of readFilesFn also
2017-02-04 10:30:25 +00:00
Nick Craig-Wood
85f05c57d1
Clean empty directories between test runs
2017-02-04 10:30:25 +00:00
Nick Craig-Wood
726cb43be9
Complete SFTP remote #521
...
* Add unit tests
* Fix up remote so it passes tests
* Add docs
2017-02-04 10:29:46 +00:00
Nick Craig-Wood
288302c2cf
Make fallback purge delete empty directories too.
...
This was implemented to make the SFTP unit tests pass.
2017-02-04 10:29:46 +00:00
Jack Schmidt
b9a8315696
Basic SFTP support, Issue #521
2017-02-04 10:29:18 +00:00
Nick Craig-Wood
9d331ce04b
Implement --ignore-checksum flag
...
Fixes #793 Fixes #863 Fixes #981
2017-02-03 08:11:10 +00:00
Nick Craig-Wood
2192805360
rclone config: when choosing from a list, allow the value to be entered
2017-01-29 15:51:26 +00:00
Nick Craig-Wood
db0b93c0ad
rclone config: allow rename and copy of remotes - fixes #641
2017-01-29 15:37:44 +00:00
Dario Giovannetti
9d36258923
Comply with XDG Base Directory specification
...
Fixes #868
2017-01-26 10:22:08 +00:00
Nick Craig-Wood
b6848a3edb
Fix race in Lister.Finished which was causing the tests to be unreliable
2017-01-19 20:11:17 +00:00
Nick Craig-Wood
e2bf9eb8e9
Implement --suffix for use with --backup-dir only #98
...
This also makes sure we remove files we are about to override in the
--backup-dir properly.
2017-01-19 20:11:17 +00:00
Nick Craig-Wood
1fa258c2b4
Define a new Features() method for Fs
...
Optional interfaces are becoming more important in rclone,
--track-renames and --backup-dir both rely on them.
Up to this point rclone has used interface upgrades to define optional
behaviour on Fs objects. However when one Fs object wraps another it
is very difficult for this scheme to work accurately. rclone has
relied on specific error messages being returned when the interface
isn't supported - this is unsatisfactory because it means you have to
call the interface to see whether it is supported.
This change enables accurate detection of optional interfaces by use
of a Features struct as returned by an obligatory Fs.Features()
method. The Features struct contains flags and function pointers
which can be tested against nil to see whether they can be used.
As a result crypt and hubic can accurately reflect the capabilities of
the underlying Fs they are wrapping.
2017-01-16 17:33:25 +00:00
ncw
3745c526f1
Implement --backup-dir - fixes #98
...
The parameter of backup-dir specifies a remote that all deleted or
overwritten files will be copied to.
2017-01-16 17:33:25 +00:00
Nick Craig-Wood
c123c702ab
Fix fs.Overlapping and factor fs.SameConfig
2017-01-14 09:55:53 +00:00
ncw
4aae7bcca6
Factor server side move detection
2017-01-14 09:55:53 +00:00
Brandur
3b1e0b66bb
Return error on not found from ListFn
...
This changes `ListFn`'s implementation so that if it encounters a not
found error, instead of sending a fatal error to log, it coordinates the
return of the error between checker goroutines and sends it back to the
caller.
The main impetus here is that it allows an external program compiling
against rclone as a package to handle a not found, where it currently it
cannot.
This does change error output on a not found a little bit, we go from
this:
2017/01/09 21:14:03 directory not found
To this:
2017/01/09 21:13:44 Failed to ls: directory not found
2017-01-12 10:07:59 +00:00
Nick Craig-Wood
78c1f2839e
Fix filters to add ** rules to directory rules
...
This fixes `--exclude ".*{,/**}"` to exclude all . files and
. directories.
2017-01-05 19:33:49 +00:00
Nick Craig-Wood
de2d967abd
Stop --track-renames hashing matching files - fixes #984
...
Also only hash files of the correct size.
This speeds it up a lot.
2017-01-05 17:58:01 +00:00
Marco Paganini
6611d92e21
Only start bandwidth ticker when necessary.
...
- Only start the token ticker when the timetable entry has more than one
entry.
- This fixes the "Scheduled bandwidth change" log message when no
bwlimit is specified.
- Fixes #987
2017-01-04 19:03:49 -08:00
Nick Craig-Wood
f73ee5eade
Make all config file variables be settable in the environment
...
These are set in the form RCLONE_CONFIG_remote_option where remote is
the uppercased remote name and option is the uppercased config file
option name. Note that RCLONE_CONFIG_remote_TYPE must be set if
defining a new remote.
Fixes #616
2017-01-03 22:42:47 +00:00
Nick Craig-Wood
0d75d2585f
Allow all options to be set from environment variables
...
The option names are munged changing - to _ making upper case and
prepending RCLONE_. The values are as parsed by pflag.
2017-01-03 22:42:47 +00:00
Marco Paganini
3b0f944e23
Add time-based bandwidth limits.
...
- Change the --bwlimit command line parameter to accept both a limit (as
before) or a full timetable (formatted as "hh:mm,limit
hh:mm,limit...")
- The timetable is checked once a minute by a ticker function. A new
tokenBucket is created every time a bandwidth change is necessary.
- This change is compatible with the SIGUSR2 change to toggle bandwidth
limits.
This resolves #221 .
2017-01-03 21:00:38 +00:00
Nick Craig-Wood
f1221b510b
Change --track-renames to use the length,hash pair stored in a map
...
This makes it much faster in the case of many files and use less
memory.
This also detects use of --no-traverse and disables it.
2017-01-03 20:37:06 +00:00
Nick Craig-Wood
274ab349f4
sync: Only allow --track-renames if have a common hash
2017-01-03 20:35:05 +00:00
Bjørn Erik Pedersen
47d3a450a4
sync: Track and perform server-side renames
...
This commits adds support for tracking of file renames if `track-renames` flag is set,
and it then performs server-side renames for remotes that support it, i.e.
remotes that implement either the `Mover` or the `Copier` interface.
Fixes #888
2017-01-03 20:35:05 +00:00
Nick Craig-Wood
5b8b379feb
Version v1.35
2017-01-02 15:33:06 +00:00
Nick Craig-Wood
4dd5428b13
Fix rmdirs test and integration tests which depend on each other
2017-01-02 14:15:07 +00:00
Nick Craig-Wood
64ec220d5d
Fix --no-update-modtime test on remotes which don't support hashes
2016-12-31 15:19:26 +00:00
Nick Craig-Wood
cbfec0d281
Fix tests for missing config file
2016-12-20 15:05:08 +00:00
Nick Craig-Wood
1b2dda8c4c
oauthutil: Reload config file off disk before updating token
...
This fixes the config file being overwritten when two rclones are running.
Fixes #682
2016-12-19 15:04:07 +00:00
Nick Craig-Wood
4482e75f38
Fix golint
2016-12-15 21:02:41 +00:00
Nick Craig-Wood
0117aeafbf
mount: this removes the async buffering as it was killing seek performance
2016-12-15 17:08:52 +00:00
Nick Craig-Wood
8083804575
Make sure wrapped retry/fatal errors are never nil to avoid panic
2016-12-13 16:02:14 +00:00
Nick Craig-Wood
fafbcc8e2f
Make server side move more obvious in debug
2016-12-09 16:57:07 +00:00
Nick Craig-Wood
48a2e3844d
Add optional interface DirCacheFlush for making the tests more reliable
...
This is defined for the users of dircache drive, onedrive, and acd.
This helps fix the DirMove tests on acd.
2016-12-09 16:57:07 +00:00
Nick Craig-Wood
dcf53a1d12
Allows multiple --include/--exclude/--filter options - fixes #875
2016-12-07 13:37:40 +00:00
Nick Craig-Wood
3bdfa284a9
Make rclone lsd obey the filters properly
2016-12-07 11:16:36 +00:00
Nick Craig-Wood
be4fd51289
fuse: Add bandwidth accounting and buffering
...
This fixes rclone mount ignoring bwlimit and increases buffering which
should speed up transfers greatly.
Fixes #796
Fixes #690
2016-12-04 16:57:47 +00:00
Nick Craig-Wood
c265f451f2
Implement moveto and copyto commands for choosing a destination name on copy/move
...
Fixes #227
Fixes #476
2016-12-03 23:43:52 +00:00
Nick Craig-Wood
2058652fa4
Allow overlapping remotes in move when DirMove is supported
2016-12-03 09:08:40 +00:00
Nick Craig-Wood
50b3cfccb1
Factor Move out of sync.go and add remote parameter to Move and Copy
2016-12-03 09:08:40 +00:00
Nick Craig-Wood
2756900749
Fix not transferring files that don't differ in size - fixes #911
...
Due to a logic error files stored on remotes which support modtime but
not hashes weren't being transferred when updating with a file of the
same size but different modtime. Instead the modtime of the remote
file was being set to that of the local file.
In practice this affected crypt with all remotes except Amazon Drive
and Dropbox.
2016-11-28 17:08:15 +00:00
Nick Craig-Wood
539853df36
Fix rmdirs test
2016-11-28 12:23:24 +00:00
Scott McGillivray
f9df545e3c
add --stats-unit option and improve alignment for --stats output
2016-11-28 12:18:30 +00:00
Nick Craig-Wood
f3365dd251
Make rclone rmdirs command to delete empty directories - fixes #831
2016-11-27 11:49:31 +00:00
Nick Craig-Wood
aaa1370a36
Add directory parameter to Rmdir and Mkdir #100 #831
...
This will enable rclone to manage directories properly in the future.
2016-11-26 12:02:53 +00:00
Nick Craig-Wood
d95288175f
Version v1.34
2016-11-06 10:18:30 +00:00
Nick Craig-Wood
01be5bff02
Fix ogier/pflag vs spf13/pflag
2016-11-05 18:18:08 +00:00
Nick Craig-Wood
e825df6448
Fix Check on crypted file systems
2016-11-05 18:17:21 +00:00
Nick Craig-Wood
ff41b0d435
Improve error message when source remote isn't found in sync #848
2016-11-05 18:03:55 +00:00
Nick Craig-Wood
441951a93b
Stop removing failed upload to cloud storage remotes - fixes #559
...
We do remove a partially written file on local so we don't have
corrupted files lying around.
2016-11-04 21:34:25 +00:00
Nick Craig-Wood
318e335137
Remove Authorization:
headers from --dump-headers
output
...
Add in `--dump-auth` flag to put it back.
2016-11-02 15:53:43 +00:00
Nick Craig-Wood
d1759fdfa9
Add request ID to HTTP debugging to make it easier to trace concurrent flows
2016-10-31 12:01:28 +00:00
Nick Craig-Wood
e65059e431
Fix non-windows/non-unix builds for bwlimit/SIGUSR2 feature and add a mutex
...
The race detector complained whenever SIGUSR2 was sent to rclone so
this adds a mutex to prevent concurrent access.
2016-10-30 19:20:16 +00:00
Nick Craig-Wood
5454f2abd0
Fix race in checkServerTime
2016-10-30 19:16:27 +00:00
Marco Paganini
cc4f5ba7ba
Add support to toggle bandwidth limits via SIGUSR2.
...
Sending rclone a SIGUSR2 signal will toggle the limiter between off and
the limit set with the --bwlimit command-line option.
2016-10-30 17:46:59 +00:00
Nick Craig-Wood
de73063977
Fix output of crypt objects in logs
2016-10-20 17:46:51 +01:00
Nick Craig-Wood
b7875fc02a
rclone check: show count of hashes that couldn't be checked #700
2016-10-17 16:48:11 +01:00
Nick Craig-Wood
8a56a6836a
Check server time against local time #654
2016-10-08 14:00:50 +01:00
Nick Craig-Wood
83849e0a36
Don't show encrypted password to stop confusion - fixes #656
2016-10-08 11:26:14 +01:00
Nick Craig-Wood
d42b38699b
Make ResponseHeaderTimeout be --timeout
not --contimeout
fixes #766
...
This was causing a problem with Amazon Drive which often pauses for a
long time after uploads before returning the response.
2016-10-08 10:12:19 +01:00
Nick Craig-Wood
d033e92234
Stop single file and --files-from
operations iterating through the source bucket.
...
This works by making sure directory listings that use a filter only
iterate the files provided in the filter (if any).
Single file copies now don't iterate the source or destination
buckets.
Note that this could potentially slow down very long `--files-from`
lists - this is easy to fix (with another flag probably) if it causes
anyone a problem.
Fixes #610
Fixes #769
2016-10-07 11:39:39 +01:00
Nick Craig-Wood
e94850f322
Fix timeouts not working when set to 0 and firing too often - #766
2016-10-06 10:23:23 +01:00
Nick Craig-Wood
aef2ac5c04
Add options for Open and implement Range for all remotes
2016-10-05 21:03:56 +01:00
Nick Craig-Wood
9ea20bac42
Fix accidentally committed test in move code
2016-10-03 20:16:41 +01:00
Nick Craig-Wood
945f49ab5e
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-10-03 20:02:04 +01:00
Nick Craig-Wood
6c9a258d82
Fix move command
...
* Delete src files which already existed in dst - fixes #751
* Fix deletion of src file when dst file older
2016-10-03 19:58:44 +01:00
Nick Craig-Wood
f2eeb4301c
Make --dump-bodies imply --dump-headers
2016-09-22 08:40:37 +01:00
Nick Craig-Wood
0caac70994
Fix build for < go1.7
2016-09-13 11:36:14 +01:00
Nick Craig-Wood
83ba59749f
Make failed uploads not count as "Transferred" - fixes #708
2016-09-12 18:15:58 +01:00
Nick Craig-Wood
bbf819e2d1
Move versioncheck so it happens earlier in the compile process.
2016-09-12 17:50:19 +01:00
Nick Craig-Wood
0cb9bb3b54
Redo http Transport code
...
* Insert User-Agent in Transport - fixes #199
* Update timeouts to use Context
* Modernise transport
2016-09-12 17:50:19 +01:00
Nick Craig-Wood
2eaac80c86
b2 with crypt: fix crash when uploading large files - fixes #673
2016-09-05 18:10:01 +01:00
Nick Craig-Wood
3996bbb8cb
Version v1.33
2016-08-24 23:02:05 +01:00
Nick Craig-Wood
059743a1b0
crypt: add to integration tests
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
a54806e5c1
Fix Move when underlying remote returns ErrorCantMove
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
226c2a0d83
Implement crypt for encrypted remotes - #219
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
9985fc40f4
Make Password parameters obey Optional flag and offer to generate random ones
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
b1de4c8cba
Implement password Option and re-implement editing
...
Editing now shows all the options for the fs and asks one at a time
whether they should be changed.
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
6a4e424630
Re-implement Obscure/Reveal so they use AES-CTR encryption
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
ebb67c135e
Fix listToChan passing nil objects to DeleteFile
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
037a000cc8
b2: fix stats accounting for upload - fixes #602
2016-08-22 21:19:38 +01:00
Nick Craig-Wood
84eb7031bb
Implement the rclone cat command
2016-08-18 22:45:32 +01:00
Nick Craig-Wood
d7b79b4481
Mark the compiled from source version with -DEV - fixes #627
2016-08-18 21:31:10 +01:00
Nick Craig-Wood
b5faaf7116
Fix double close of abort channel - fixes #592
2016-08-18 18:56:57 +01:00
Nick Craig-Wood
0805ec051f
Add BasicInfo interface shared between Dir and Object
2016-08-05 17:45:27 +01:00
Nick Craig-Wood
ae56df7d4f
Add --dedupe-mode only to dedupe command
2016-08-04 21:47:14 +01:00
Nick Craig-Wood
67562081f7
Version v1.32
2016-07-13 17:32:39 +01:00
Nick Craig-Wood
63f6827a0d
Version v1.31
2016-07-13 12:28:01 +01:00
Nick Craig-Wood
ac3c83f966
Fix integration tests for drive
2016-07-12 21:38:15 +01:00
Nick Craig-Wood
b9c8e61d39
Explicitly check the state in tests after writing files
...
...otherwise Amazon Drive will fail.
2016-07-12 21:36:39 +01:00
Nick Craig-Wood
a6056408dd
Fix move command - stop it running for overlapping fses - fixes #577
...
* Make move command check for overlapping remotes and refuse to run
* Do copy/delete rather than all the copies then all the deletes
* Doesn't purge the source - this was unexpected behaviour see #512 and #416
* Add -list-retries flag to test suite to control retries
This changes the semantics of `move` slightly. However it now errs on
the side of not deleting stuff.
2016-07-12 10:49:37 +01:00
Nick Craig-Wood
b9479cf7ab
Implement --no-update-modtime flag - fixes #511
2016-07-12 10:46:45 +01:00
Nick Craig-Wood
a20d80565b
Tidy stats output - fixes #541
2016-07-11 13:04:30 +01:00
Nick Craig-Wood
4bd9932703
Fix wording in verbose copy logs - fixes #574
2016-07-09 10:11:57 +01:00
Nick Craig-Wood
13591c7c00
Redo error handling for sync/copy/move
...
* Factor sync/copy/move into its own file
* Make fatal errors abort the sync
* Make Copy return errors
* Make Sync/Copy/Move return the last Copy error if there was one
* Prioritise returning Fatal errors
* NoRetry errors are returned if no other types of errors
2016-07-04 13:45:53 +01:00
Nick Craig-Wood
28f4061892
Add two more classes of error Fatal and NoRetry
...
These are for remotes to signal that they have a fatal error and don't
want to continue (eg cap exceeded) or that a particular file shouldn't
be retried for some reason.
2016-07-04 13:45:52 +01:00
Nick Craig-Wood
0a43ff9c13
Modify interface for accounting to take a string not an fs.Object
2016-07-02 16:58:50 +01:00
Nick Craig-Wood
9aae143833
Implement cleanup command for emptying trash / removing old versions of files
2016-07-01 16:35:36 +01:00
Nick Craig-Wood
70dc97231e
Convert more tests to use assert/require
2016-06-30 15:45:30 +01:00
Nick Craig-Wood
af4ef8ad8d
Implement --no-traverse flag to stop copy traversing the destination remote.
...
Refactor sync/copy/move
* Don't load the src listing unless doing a sync and --delete-before
* Don't load the dst listing if doing copy/move and --no-traverse is set
`rclone --no-traverse copy src dst` now won't load either of the
listings into memory so will use the minimum amount of memory.
This change will reduce the amount of memory rclone uses dramatically
too as in normal operations (copy without --notraverse or sync) as it
no longer loads the source file listing into memory at all.
Fixes #8
Fixes #544
Fixes #546
2016-06-29 17:38:50 +01:00
Nick Craig-Wood
13797a1fb8
Make retry logs be debug in main copy routine
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
3ad8fb8634
Make DeleteFile and DeleteFiles return errors
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
ab43005422
Make NewObject return an error
...
* make it return an error
* make a canonical error fs.ErrorNotFound
* make a test for it
* remove logs/debugs of error
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
b1f131964e
Rename NewFsObject to NewObject
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
1a87b69376
Get rid of LimitedFs - FIXME needs docs on copying single files
...
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.
Use this to remove the LimitedFs and just add this file to the
--files-from list.
This means that server side operations can be used also.
Fixes #518
Fixes #545
2016-06-28 08:51:43 +01:00
Nick Craig-Wood
5a3b109e25
Fix issues identified by go vet -shadow - fixes #530
2016-06-21 21:17:52 +01:00
Klaus Post
e0aa4bb492
Fix incomplete local hashes.
...
Fixes #533
2016-06-19 16:51:49 +02:00
Nick Craig-Wood
bd0227450e
Version v1.30
2016-06-18 16:41:46 +01:00
Nick Craig-Wood
f438f1e9ef
Fix stats print
2016-06-18 16:41:46 +01:00
Nick Craig-Wood
d3dd672640
Document recursion requirements for Fses
2016-06-18 14:12:47 +01:00
Nick Craig-Wood
2a46be8cf3
b2: implement large file uploading - fixes #456
2016-06-18 13:38:05 +01:00
Nick Craig-Wood
1b4370bde1
Rework retry logic when copying objects
...
* Fix off by one retry logic - fixes #406
* Retry any retriable errors
* Restructure code
2016-06-18 10:55:58 +01:00
Nick Craig-Wood
2cfb3834f2
Log errors with %v
2016-06-18 09:36:47 +01:00
Nick Craig-Wood
46135d830e
Add --ignore-size flag - fixes #399
2016-06-17 17:20:08 +01:00
Nick Craig-Wood
df1092ef33
Change Fs.Put so that it must cope with existing files
...
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.
None of the other Fses needed adjusting.
Fixes #483
2016-06-13 19:29:10 +01:00
Nick Craig-Wood
4c5b2833b3
Convert to using github.com/pkg/errors everywhere
2016-06-13 17:43:03 +01:00
Nick Craig-Wood
7fe653c350
Unwrap errors properly for patform specific connection retry code.
...
Include more possible errors for Windows.
For #442
2016-06-10 13:48:41 +01:00
Nick Craig-Wood
f17cb1bf50
Fix retry of Windows wsaend errors #442
...
Make the test for wsaend error less specific
2016-06-09 15:34:13 +01:00
Nick Craig-Wood
67d0375b98
Audit use of log.Print and change to Debug, Log, or ErrorLog as appropriate
2016-06-06 21:23:54 +01:00
Nick Craig-Wood
108760e17b
Log -v output to stdout by default - fixes #228
2016-06-04 18:49:27 +01:00
Nick Craig-Wood
e2788aa729
Display the transfer stats in more human readable form - fixes #428
2016-06-03 22:49:50 +01:00
Nick Craig-Wood
772f99fd74
Make SizeSuffix output without b
suffix for more useful printouts
2016-06-03 22:49:14 +01:00
Nick Craig-Wood
a21cc161de
Make 0 size files specifiable with --max-size 0b
- fixes #450
2016-06-03 21:54:27 +01:00
Nick Craig-Wood
e818b7c206
Represent -1 as "off" for SIZE values
2016-06-03 21:51:39 +01:00
Nick Craig-Wood
5723d788a4
Add b
suffix so we can specify bytes in --bwlimit, --min-size etc
...
Fixes #449
2016-06-03 21:16:48 +01:00
Nick Craig-Wood
1d6698a754
Build tweaks - fixes #484
...
* disable CGO for static builds everywhere
* override Version in release build script
* don't output symbol table in release binaries
2016-06-03 20:34:19 +01:00
Nick Craig-Wood
ccdd1ea6c4
Add --max-depth parameter
...
This will apply to ls/lsd/sync/copy etc
Fixes #412
Fixes #213
2016-06-03 17:05:39 +01:00
Nick Craig-Wood
ad85f6e413
Implement directory include filtering for efficiency
...
Fixes #395
2016-05-16 17:14:04 +01:00
Nick Craig-Wood
ac9c20b048
Make IsRetryError function
2016-05-14 17:11:19 +01:00
Nick Craig-Wood
2db35f0ce7
Dump out unexpected state in integration test
2016-05-07 21:19:26 +01:00
Nick Craig-Wood
dbfa7031d2
Factor Lister into own file, write tests and fix
2016-05-07 17:17:43 +01:00
Nick Craig-Wood
c2d0e86431
Add more tests for List() and fix resulting problems
2016-05-07 14:50:35 +01:00
Nick Craig-Wood
68ec6a9f5b
Add a directory parameter to Fs.List()
2016-05-06 16:52:34 +01:00
Nick Craig-Wood
753b0717be
Refactor the List and ListDir interface
...
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.
Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.
This commit contains some code originally by Klaus Post.
Fixes #316
2016-05-06 16:52:34 +01:00
Nick Craig-Wood
1752ee3c8b
Retry errors which indicate the connection closed prematurely.
...
See discussion in #442
2016-04-29 17:29:34 +01:00
Nick Craig-Wood
5c2d8ffe33
Retry only the failing tests in the integration tests
2016-04-26 10:20:07 +01:00
Klaus Post
6c0c9abd57
Use "password:" instead of "password>" prompt
...
Fixes #410
2016-04-21 19:39:46 +01:00
Nick Craig-Wood
157d7d45f5
Version v1.29
2016-04-18 18:30:29 +01:00
Nick Craig-Wood
b5cba73cc3
Make test more reliable
2016-04-18 17:48:52 +01:00
Nick Craig-Wood
0f8e7c3843
Make rclone check obey the --size-only flag - fixes #419
2016-04-07 15:01:45 +01:00
Nick Craig-Wood
b835330714
Use "application/octet-stream" if mime.TypeByExtension returns invalid type
...
Fixes #424
2016-04-07 14:32:01 +01:00
Klaus Post
8f4d6973fb
Fix missing "quit" option when there are no remotes.
2016-03-23 16:57:56 +00:00
Nick Craig-Wood
ced3a4bc19
Implement -I, --ignore-times for unconditional upload - fixes #311
2016-03-22 17:02:27 +00:00
Nick Craig-Wood
cb22583212
b2: Enable mod time syncing - fixes #348
2016-03-22 15:56:44 +00:00
Nick Craig-Wood
414b35ea56
Change the interface of SetModTime to return an error - #348
2016-03-22 15:56:44 +00:00
Nick Craig-Wood
8ef551bf9c
Make dedupe remove identical copies without asking and add non interactive mode - fixes #338
...
* Now removes identical copies without asking
* Now obeys `--dry-run`
* Implement `--dedupe-mode` for non interactive running
* `--dedupe-mode interactive` - interactive the default.
* `--dedupe-mode skip` - removes identical files then skips anything left.
* `--dedupe-mode first` - removes identical files then keeps the first one.
* `--dedupe-mode newest` - removes identical files then keeps the newest one.
* `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
* `--dedupe-mode rename` - removes identical files then renames the rest to be different.
* Add tests which will only run on Google Drive.
2016-03-06 18:15:01 +00:00
Nick Craig-Wood
874a64e5f6
A script to make a directory heirarchy for testing
2016-03-05 20:26:15 +00:00
Nick Craig-Wood
3c7f80f58f
Version v1.28
2016-03-01 09:00:01 +00:00
Nick Craig-Wood
229ea3f86c
Stop --update tests running on remotes which don't do mod time
2016-03-01 07:26:33 +00:00
Nick Craig-Wood
41eb386063
Reset password/config path in config tests to fix other tests
2016-02-29 21:43:37 +00:00
Nick Craig-Wood
dfc7cd97a3
Optionally disable gzip compression on downloads with --no-gzip-encoding
- fixes #353
2016-02-29 19:48:54 +00:00
Nick Craig-Wood
280ac26464
Implement -u/--update so creation times can be used on all remotes - #226
2016-02-29 17:46:40 +00:00
Nick Craig-Wood
88cca8a6eb
Simplify literals (after running gofmt -s over the code)
2016-02-29 16:57:23 +00:00
Nick Craig-Wood
3343c1afa4
Don't make directories if --dry-run set - fixes #342
2016-02-28 19:56:50 +00:00
Nick Craig-Wood
867a26fe4f
Implement --low-level-retries flag - fixes #266
2016-02-25 22:58:21 +00:00
Nick Craig-Wood
3890105cdc
Add -run-only flag to run_all test
2016-02-25 22:05:57 +00:00
Nick Craig-Wood
d2219a800a
Fix and document the move command - fixes #334
...
* Don't attempt to use server side Move unless they are on the same Fs
* Fix move in the presense of filters
2016-02-25 20:05:34 +00:00
Nick Craig-Wood
ccb59480bd
Add InActive method to Filter to detect when no fiters are in use.
2016-02-25 19:58:00 +00:00
Nick Craig-Wood
95e4d837ef
Make config chooser easier to understand
2016-02-21 13:40:16 +00:00
Nick Craig-Wood
e08e35984c
Add help to remote chooser in rclone config - fixes #43
2016-02-21 13:40:16 +00:00
Nick Craig-Wood
700e47d6e2
Stub out ReadPassword on plan9 and solaris to fix compilation
2016-02-21 10:31:53 +00:00
klauspost
ef06371c93
Create separate interface for object information.
...
Take out read-only information about a Fs in a separate struct to limit access.
See discussion at #282 .
2016-02-19 13:31:09 +00:00
klauspost
84b00b362f
Change back to original goconfig package.
...
Add documentation for `--ask-password`.
2016-02-17 11:45:05 +01:00
klauspost
bfd7601cf9
Add configuration file encryption
...
See #317 for details.
Use `rclone config` to add/change/remove password.
Tests that loads the default configuration will now fail with a better error message, and add a switch that makes it possible to disable password prompts and fail instead.
Make it possible to use the "RCLONE_CONFIG_PASS" environment variable as password for configuration.
2016-02-16 16:32:05 +01:00
Nick Craig-Wood
a6320bbad3
Fix delete command to wait until all finished - fixes missing deletes.
...
This also could affect deletes at the end of the sync command.
2016-02-15 16:43:59 +00:00
Nick Craig-Wood
85d99de26b
Fix typo in error strings
2016-02-09 16:15:50 +00:00
Nick Craig-Wood
e62c0a58a7
Version 1.27
2016-01-31 17:50:13 +00:00
Nick Craig-Wood
0f73129ab7
dedupe command to deduplicate a remote. Useful with google drive - fixes #41
2016-01-31 16:09:42 +00:00
Nick Craig-Wood
1373efaa39
Delete command which does obey the filters - fixes #327
2016-01-31 16:06:04 +00:00
Nick Craig-Wood
5c37b777fc
Make the --dry-run warnings into logs so they appear without the -v flag
2016-01-31 16:06:04 +00:00
Nick Craig-Wood
8c211fc8df
Warn the user about files with same name but different case
...
Relates to #107 & #119 .
2016-01-26 16:57:09 +00:00
Nick Craig-Wood
74a71f7824
Add tests for --delete-before, --delete-during and --delete-after
2016-01-26 16:57:09 +00:00
Nick Craig-Wood
12b51c5eb8
Remove duplicate check for filter IncludeObject
2016-01-26 16:57:09 +00:00
klauspost
14069fd8e6
Implement --delete-before, --delete-during, --delete-after - fixes #252 .
2016-01-26 16:57:09 +00:00
Nick Craig-Wood
cd62f41606
Reduce number of logs and show hash type where appropriate
2016-01-24 18:06:57 +00:00
Nick Craig-Wood
109d4ee490
Prefix all test remotes with rclone-test-
and make names more pronouncable
2016-01-24 12:37:46 +00:00
Nick Craig-Wood
18ebec8276
Check remote is empty between integration tests
2016-01-24 12:37:19 +00:00
Nick Craig-Wood
eda4130703
Fix integration tests so they can be run independently and out of order - fixes #291
...
* Make all integration tests start with an empty remote
* Add an -individual flag so this can be a different bucket/container/directory
* Fix up tests after changing the hashers
* Add sha1sum test
* Make directory checking in tests sleep more to fix acd inconsistencies
* Factor integration tests to make more maintainable
* Ensure remote writes have a fstest.CheckItems() before use
* this fixes eventual consistency on the directory listings later
* Call fs.Stats.ResetCounters() before every fs.Sync()
Note that the tests shouldn't be run concurrently as fs.Config is global state.
2016-01-23 17:02:09 +00:00
Nick Craig-Wood
27f67edb1a
Fix formatting problem in sha1sum
2016-01-17 13:56:42 +00:00
Nick Craig-Wood
3ffea738e6
Make hash constants start from 1 not 2
2016-01-17 10:47:24 +00:00
Nick Craig-Wood
e8daab7971
Fix integration tests for remotes with unsupported hash schemes
2016-01-16 09:45:15 +00:00
klauspost
78c3a5ccfa
Add support for multiple hash types.
...
Add support for multiple hash types with negotiation of common hash types for comparison.
Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
Nick Craig-Wood
af5f4ee724
Make --include rules add their implict exclude * at the end of the filter list
...
This means you can mix `--include` and `--include-from` with the
other filters (eg `--exclude`) but you must include all the files you
want in the include statement.
Fixes #280
2016-01-10 11:42:53 +00:00
klauspost
25f22ec561
Add "--ignore-existing" flag.
...
Add option to completely ignore existing files and not consider them for transfer.
Fixes #274
2016-01-08 08:20:04 +00:00
Nick Craig-Wood
5189231a34
Tweaks to rclone authorize
...
* Document the headless / remote setup procedure
* Move Config constants into fs
* Parse arguments in main for Authorize
2016-01-07 20:31:23 +00:00
klauspost
bcbd30bb8a
Add easier headless configuration.
...
This will allow setting up a remote with copy&paste of values to a headless machine. It will allow copy+pasting a token into the configuration.
This requires rclone to be on a machine with a proper browser. Custom client id and secrets are supported.
To test token generation, use `rclone auth "fs type"`.
2016-01-07 20:31:23 +00:00
Nick Craig-Wood
1ce3673006
Add -clean flag to test_all.go to clean left over test directories
2016-01-03 21:49:26 +00:00
Nick Craig-Wood
3423de65fa
Make canonical place for all fs in fs/all/all.go
2016-01-03 14:12:45 +00:00
Nick Craig-Wood
77fb8ac240
Version 1.26
2016-01-02 12:04:32 +00:00
dibu28
3ac4407b88
Implement Yandex storage backend - fixes #234
2015-12-30 12:11:46 +00:00
Nick Craig-Wood
8ea0d5212f
Add -verbose flag to test_all and fix tries count
2015-12-30 11:34:22 +00:00
Nick Craig-Wood
33b3eea6ec
Implement Backblaze B2 - fixes #224
2015-12-30 10:05:07 +00:00
Nick Craig-Wood
113624691a
Add -dump-headers and -dump-bodies flags for operations test debugging
2015-12-30 09:35:35 +00:00
Nick Craig-Wood
ddf39f2d57
Replace test_all.sh with test_all.go which is cross platform and parallel
2015-12-30 09:26:34 +00:00
Nick Craig-Wood
e69e181090
Fix --min-age and --max-age when only one is present
2015-12-17 14:22:43 +00:00
Nick Craig-Wood
ac65d8369e
Make fs.CheckClose public to stop duplication
2015-12-17 13:30:58 +00:00
Nick Craig-Wood
8057d668bb
Fix crash in http logging - fixes #223
...
A nil-pointer exception was caused if the http transaction ever
resulted in a go error while using `--dump-bodies`. Now don't ignore
the error and log it instead of the http body.
2015-12-17 13:30:58 +00:00
Nick Craig-Wood
36f1bc4a8a
Make ls/lsl/md5sum/size/check obey includes and excludes - fixes #169
...
* run check directory listings concurrently
2015-12-17 13:30:58 +00:00
Adriano Aurélio Meirelles
3cbd57d9ad
Add support to filter files based on their age
2015-12-17 09:52:38 -02:00
Nick Craig-Wood
7f0020a407
Version v1.25
2015-11-14 13:06:39 +00:00
Nick Craig-Wood
fcea3777c0
Implement Hubic storage system - fixes #200
2015-11-14 08:08:52 +00:00
Nick Craig-Wood
bed01a303f
Add UnWrapper interface and implement in LimitedFs
2015-11-14 08:08:51 +00:00
Nick Craig-Wood
2c2cb84ca7
Make it so optional interface Purge can fail so it can be wrapped
2015-11-14 08:08:51 +00:00
Nick Craig-Wood
e9dda25c60
Implement Move in limited fs
2015-11-14 08:08:51 +00:00
Nick Craig-Wood
80ffbade22
Fix deletion of some excluded files without --delete-excluded #205
...
This only happened if the destination file was present but the source
file was missing.
2015-11-12 11:46:04 +00:00
Nick Craig-Wood
4276abc58b
Version v1.24
2015-11-07 16:23:12 +00:00
Nick Craig-Wood
d42cb11b84
Fix tests to run all tests again and add onedrive
2015-11-07 11:21:15 +00:00
Nick Craig-Wood
365b4babae
Make filter test files pass errcheck
2015-11-07 10:27:47 +00:00
Nick Craig-Wood
2fcb8f5db7
Add support for Microsoft One Drive - fixes #10
...
* Still to do
* Copy
* Move
* MoveDir
2015-11-07 10:19:10 +00:00
Nick Craig-Wood
ff16e0f6df
Factor common error handling into fs module
2015-11-06 15:26:58 +00:00
Nick Craig-Wood
ed72c678f8
Protect accounting from being closed twice
2015-11-06 15:26:58 +00:00
klauspost
b85ddc4e4f
Extend CI tests to include formatting checks.
...
CI tests now tests 'go vet', 'go fmt' (via goimports) and golint.
Adds Travis experimental OSX support.
2015-11-03 13:50:29 +01:00
klauspost
b872ff0237
Add option to disable server certificate verification.
...
The option name mirrors the 'wget' option (also `--no-check-certificate`). The cURL equivalent is called `--insecure`, which is a bit unclear.
Put in the "developers" section in documentation with proper warnings.
Fixes #168
2015-10-29 16:42:25 +01:00
Nick Craig-Wood
1b95718460
Fix typos in filter docs and unit test assertions
2015-10-20 09:16:47 +01:00
Nick Craig-Wood
4e952af614
Allow spaces in remotes and check remote names for validity at creation time - fixes #171
2015-10-12 17:54:09 +01:00
Klaus Post
6344c3051c
Add async readahead buffer
...
This adds an async read buffer of 4x4MB when copying files >10MB.
This fixes #164 and reduces the number of IO operations for copy/move.
2015-10-12 08:30:27 +01:00
Nick Craig-Wood
3a900e5bb7
Version v1.23
2015-10-03 16:24:07 +01:00
Nick Craig-Wood
b4d7741611
Improve output of --dump-headers
2015-10-03 16:04:51 +01:00
Nick Craig-Wood
177dbbc29a
Implement rclone size for measuring remotes - fixes #153
2015-10-03 16:04:21 +01:00
Nick Craig-Wood
57aad81b68
Version v1.22
2015-09-28 19:38:20 +01:00
Nick Craig-Wood
a91bcaaeb0
Implement rsync like include and exclude - fixes #27
...
* Implement include/exclude
* Implement rsync compatible file globbing
* Implement command line filtering flags
* --delete-excluded - Delete files on dest excluded from sync
* --filter - Add a file-filtering rule
* --filter-from - Read filtering patterns from a file
* --exclude - Exclude files matching pattern
* --exclude-from - Read exclude patterns from file
* --include - Include files matching pattern
* --include-from - Read include patterns from file
* --files-from - Read list of source-file nam
* --min-size - Don't transfer any file smaller than this in k or suffix k|M|G
* --max-size - Don't transfer any file larger than this in k or suffix k|M|G
* Document
2015-09-28 19:18:21 +01:00
Nick Craig-Wood
5d4b149884
Version v1.21
2015-09-22 21:05:11 +01:00
Nick Craig-Wood
5f20ae707d
Make lsl output times in localtime and fix tests - fixes #141
2015-09-22 19:04:12 +01:00
Nick Craig-Wood
e9c915e6fe
Fix golint warnings
2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2ed158aba3
Fixes from go vet and errcheck
2015-09-22 07:31:12 +01:00
Nick Craig-Wood
e391311512
gofmt
2015-09-17 18:42:39 +01:00
klauspost
788ef76f1c
Show more of the filename and align output.
...
Print more of the file name, and make the output aligned, so it is nicer on frequent updates.
2015-09-17 15:53:45 +02:00
klauspost
0a5870208e
Display individual transfer progress
...
Improve progress printing by displaying individual file progress, as well
as a moving average speed with ETA. Example output:
2015/09/15 16:38:21
Transferred: 183599104 Bytes (4646.49 kByte/s)
Errors: 0
Checks: 1
Transferred: 0
Elapsed time: 38.5s
Transferring:
* 01_06_14.mp3: 33% done. avg: 1280.5, cur: 1288.8 kByte/s. ETA: 1m12s
* 01_12_15.mp3: 33% done. avg: 1002.2, cur: 943.4 kByte/s. ETA: 1m17s
* 01_13_14.mp3: 48% done. avg: 1456.8, cur: 1425.2 kByte/s. ETA: 39s
* 01_19_15.mp3: 28% done. avg: 1226.9, cur: 1114.4 kByte/s. ETA: 1m37s
2015-09-16 19:42:46 +01:00
Nick Craig-Wood
34193fd8d9
Version v1.20
2015-09-15 07:42:31 +01:00
Nick Craig-Wood
a288c2b3a3
Make a retry error wrapper for a plain error
2015-09-09 23:22:41 +01:00
Nick Craig-Wood
fec16b0ac8
acd: Skip test on FS which don't support ModifiedTime
2015-09-09 23:21:50 +01:00
Nick Craig-Wood
dd8717797e
Implement --dump-headers and --dump-bodies debug flags
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
8c3df224ef
Implement Amazon Cloud Drive - fixes #45
...
* Optional interfaces Copier, Mover, DirMover not done
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
c98a51b26c
Lightly obscure secrets
2015-09-01 22:33:34 +01:00
Nick Craig-Wood
8d33ce0154
Check for source and dest being the same in sync/copy/move
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
92745aa950
Add Root() to Fs interface
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
0168f55f3e
Switch to spf13 fork of pflag - fixes #116
...
This supports --long value as well as --long=value which is as
expected for a unix utility.
2015-08-29 18:14:24 +01:00
Nick Craig-Wood
9ed2de3d6e
Version v1.19
2015-08-28 09:47:13 +01:00
Nick Craig-Wood
59ba8f28c8
Implement move command - fixes #35
...
* Define Mover interface to move a single object
* Define DirMover interface to move a directory
* Implement DirMove operation
* Add `rclone move` command
* Tests for Dir Move
To Do
* Implement Move, DirMover in local, drive, dropbox
* unit test for Mover
* unit test for DirMover
2015-08-28 08:49:16 +01:00
Nick Craig-Wood
5c226e91c0
Ignore rmdir in limited fs rather than throwing error - fixes #112
2015-08-25 19:16:25 +01:00
Nick Craig-Wood
a96b522958
Implement server side copies if possible - fixes #99
...
Add optional fs.Copier interface
Implemented for
* swift
* s3
* drive
* dropbox
* google cloud storage
2015-08-23 21:18:38 +01:00
Nick Craig-Wood
fedf81c2b7
Add Name() to Fs interface to return name as passed to NewFs
2015-08-23 13:36:38 +01:00
Nick Craig-Wood
0c6f816a49
Implement --retries flag - fixes #109
2015-08-20 21:07:00 +01:00
Nick Craig-Wood
dfe771fb0c
Correct log messages for remotes which don't support modtime/md5sum
2015-08-20 20:48:58 +01:00
Nick Craig-Wood
2fcc18779b
Version v1.18
2015-08-17 17:59:37 +01:00
Nick Craig-Wood
1d9e76bb0f
dropbox: remove datastore - Fixes #55 #84
...
This means that dropbox no longer stores MD5SUMs and modified times.
Fixup the tests so that blank MD5SUMs are ignored, and that if
Precision is set to a fs.ModTimeNotSupported, ModTimes can be ignored too.
This opens the door for other FSs which don't support metadata easily.
2015-08-17 17:26:36 +01:00
Nick Craig-Wood
771e60bd07
Show errors when reading the config file
2015-08-15 17:15:02 +01:00
Leonid Shalupov
3fcff32524
do not print stats in quiet mode - fixes #70
...
...unless had some errors or stats interval requested.
Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
2015-08-10 11:17:54 +01:00
Nick Craig-Wood
9d9a17547a
Version v1.17
2015-06-14 15:36:16 +01:00
Leonid Shalupov
bd5f685d0a
fix TestLsLong on non-UTC timezones
2015-06-14 15:23:56 +01:00
Nick Craig-Wood
539ad44757
Version v1.16
2015-06-09 18:00:33 +01:00
Nick Craig-Wood
74994a2ec1
Fix uploading big files which was causing timeouts or panics
...
The symtom was one of these two on upload of files only
* panic: d.nx != 0 in crypto/md5.(*digest).checkSum
* read tcp: i/o timeout
It turned out to be a combination of two upstream bugs
* 5a2187309e
* https://groups.google.com/forum/#!topic/golang-dev/0Nl6k5Sj6UU
This commit contains a work-around for the second problem, I've fixed
the first and had the change accepted upstream.
2015-06-09 17:32:45 +01:00
Nick Craig-Wood
97dced6a0b
Don't check md5sum after download with --size-only - fixes #75
2015-06-09 13:18:40 +01:00
Nick Craig-Wood
e04acb09ce
Version v1.15
2015-06-06 15:45:00 +01:00
Nick Craig-Wood
90744301d3
Fix package docs so they appear in godoc correctly
2015-06-06 14:24:30 +01:00
Nick Craig-Wood
e22b445cff
Implement --size-only flag to sync on size not checksum & modtime - fixes #75
2015-06-06 08:49:01 +01:00
Alex Couper
7af1282375
Add --checksum flag to only discard transfers by MD5SUM - fixes #61
...
Useful for copying between backends where checksum fetching is fast,
ie any of s3, swift, drive or googlecloudstorage.
2015-06-05 19:46:03 +01:00
Nick Craig-Wood
d9fcc32f70
Version v1.14
2015-05-21 20:13:40 +01:00
Nick Craig-Wood
88293a4b8a
Version v1.13
2015-05-10 12:39:06 +01:00
Nick Craig-Wood
1d254a3674
Implement --timeout and --conntimeout - fixes #54
...
NB dropbox still to do
2015-05-10 11:29:55 +01:00
Nick Craig-Wood
ba2091725e
Version v1.12
2015-03-15 15:55:38 +00:00
Nick Craig-Wood
5cc5429f99
drive: switch to insert from update when a failed copy deletes the upload
2015-03-15 15:27:55 +00:00
Nick Craig-Wood
09d71239b6
Make file size render more neatly and prevent from being < 0
2015-03-15 15:27:55 +00:00
Nick Craig-Wood
c643e4585e
core: Log duplicate files if they are detected
2015-03-15 15:27:55 +00:00
Nick Craig-Wood
873db29391
Log all objects more informatively
2015-03-15 15:27:55 +00:00
Nick Craig-Wood
ecb3c7bcef
drive, googlecloudstorage: remove SeekWrapper after googleapi fix
2015-03-04 20:47:59 +00:00
Nick Craig-Wood
80000b904c
Version v1.11
2015-03-04 17:59:31 +00:00
Nick Craig-Wood
b4a0941d4c
In remote paths, change native directory separators to / - fixes #37
2015-03-02 17:04:34 +00:00
Nick Craig-Wood
46d39ebaf7
Factor Mime Type guessing into fs.MimeType()
2015-03-02 09:21:15 +00:00
Nick Craig-Wood
fe68737268
Fix niggles found by go vet
2015-02-28 15:35:54 +00:00
Nick Craig-Wood
2360bf907a
Add synchronization to list output to stop corruptions - fixes #29
2015-02-28 15:30:40 +00:00
Nick Craig-Wood
a287e3ced7
Implement --bwlimit to limit data transfer bandwidth
2015-02-27 15:03:47 +00:00
Nick Craig-Wood
cf208ad21b
Version v1.10
2015-02-12 18:00:20 +00:00
Nick Craig-Wood
8d1c0ad07c
Fix config loop - fixes #25
2015-02-10 16:48:04 +00:00
Nick Craig-Wood
165e89c266
Version v1.09
2015-02-07 22:44:23 +00:00
Nick Craig-Wood
b4e19cfd62
windows: make tests work properly
2015-02-07 22:32:51 +00:00
Nick Craig-Wood
20ad96f3cd
windows: Stop drive letters (eg C:) getting mixed up with remotes (eg drive:)
...
This was done by stopping the user configuring single letter remotes
and making sure we don't treat single letter remotes as a remote name,
but as a drive letter.
2015-02-07 22:32:51 +00:00
Nick Craig-Wood
5fb6f94579
drive: fix rate limit exceeded errors - fixes #20
...
This is done by pacing the requests to drive and backing them off
using an exponential increase. Put and Modify operations can now be
retried also.
2015-02-07 22:32:51 +00:00
Nick Craig-Wood
18a8a61cc5
Release v1.08
2015-02-04 22:31:56 +00:00
Nick Craig-Wood
cfdb48c864
Version v1.07
2014-12-23 11:26:32 +00:00
Nick Craig-Wood
14567952b3
google cloud storage: Fix memory leak - fixes #17
...
This was the same problem as issue #5 (which affected google drive)
2014-12-23 11:03:34 +00:00
Nick Craig-Wood
439a126af6
Version v1.06
2014-12-12 20:13:03 +00:00
Nick Craig-Wood
af9c447146
Fix "Couldn't find home directory" on OSX - fixes #15
2014-12-12 19:18:23 +00:00
Nick Craig-Wood
839133c5e1
Version v1.05
2014-08-09 17:22:17 +01:00
Nick Craig-Wood
18439cf2d7
Move rclonetest into go tests for fs module
2014-08-03 11:18:25 +01:00
Nick Craig-Wood
3e27ff1b95
Add Root List test and fs.Limited tests for single files
2014-07-31 21:35:29 +01:00
Nick Craig-Wood
ff91698fb5
Skip tests if test remote not configured
2014-07-31 08:51:39 +01:00
Nick Craig-Wood
2f9f9afac2
fs: Document that Purger returns error on empty directory, test and fix
2014-07-29 17:18:22 +01:00
Nick Craig-Wood
17ffb0855f
Fixes after running errcheck
2014-07-25 18:19:49 +01:00
Nick Craig-Wood
a5b0d88608
Make tests for each Fs
...
Factor tests out of rclonetest
2014-07-24 23:13:32 +01:00
Nick Craig-Wood
e09ef62d5b
core: Fix race detected by go race detector
2014-07-22 23:03:14 +01:00
Nick Craig-Wood
a75bc0703f
Version 1.04
2014-07-21 21:32:37 +01:00
Nick Craig-Wood
54cd46372a
Version 1.03
2014-07-20 11:28:50 +01:00
Nick Craig-Wood
9aa4b6bd9b
Version 1.02
2014-07-19 13:24:48 +01:00
Nick Craig-Wood
e559194fb2
fs: Verify sizes are the same after transfer in Copy()
2014-07-19 13:05:07 +01:00
Nick Craig-Wood
8a76568ea8
core: Verify MD5 sums after each transfer
2014-07-15 19:27:05 +01:00
Nick Craig-Wood
c1a245d1c8
Factor UserAgent to fs and move Version to fs
2014-07-13 19:19:58 +01:00
Nick Craig-Wood
eb2b4ea8aa
rclone: Don't purge if --dry-run set
2014-07-13 10:54:30 +01:00
Nick Craig-Wood
e055ed0489
rclone: change "ls" and add "lsl" and "md5sum" commands
...
Changed "ls" command not to show modification time by default only
size and path. That is because it is slow for nearly all the remotes
as it requires extra metadata lookup. All remotes can look up files
and sizes without extra operations.
Added "lsl" which does what "ls" used to - namely show modification
time, size and path.
Added "md5sum" which produces the same output as the md5sum command -
md5sums and paths that is.
2014-07-12 12:09:20 +01:00
Nick Craig-Wood
7ebf48ef42
Fix --dry-run not working and add tests for it - fixes #3
2014-06-26 15:33:06 +01:00
Nick Craig-Wood
1d67b014cb
Make compatible with go 1.1 - fixes #1
2014-06-26 15:18:48 +01:00
Nick Craig-Wood
99695d57ab
Implement single file operations for all file systems
2014-05-05 22:17:57 +01:00
Nick Craig-Wood
8fd59f2e7d
drive: Use o.Update and fs.Put to optimise transfers
2014-04-18 17:49:01 +01:00
Nick Craig-Wood
02afcb00e9
Factor Fs.Put into Object.Update and call Update rather than Put if possible
2014-04-18 17:49:01 +01:00
Nick Craig-Wood
d6a5bfe2d4
Get rid of fs.CopyFs and replace with fs.Sync in preparation for Object.Update
2014-04-18 17:48:46 +01:00
Nick Craig-Wood
92ec29fe3f
Factor the generic code into fs and add some more intefaces
2014-03-28 22:26:42 +00:00
Nick Craig-Wood
b0064e6405
Change remote format one last time
2014-03-27 17:31:57 +00:00
Nick Craig-Wood
3eda36f6da
Factor rclone main() in preparation for tests
2014-03-27 17:26:24 +00:00
Nick Craig-Wood
a628bef9c2
Use pflag for gnu style flags
2014-03-27 16:55:29 +00:00
Nick Craig-Wood
1b3a49929b
Add ability for FS to have a Config helper function run
2014-03-16 13:54:43 +00:00
Nick Craig-Wood
1ea9972be7
Set strict permissions on the config file
2014-03-16 13:53:51 +00:00
Nick Craig-Wood
7d786204b4
Add -config option to specify a config file
2014-03-15 17:01:13 +00:00
Nick Craig-Wood
b3f1a45bbf
Config fixes
...
* Fix empty config configuration
* Alter menus when no remotes
* Save config file after delete remote
2014-03-15 16:52:51 +00:00
Nick Craig-Wood
0a108832e2
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
Nick Craig-Wood
754c54b64f
Fix differences print out on check
2013-07-02 08:33:17 +01:00
Nick Craig-Wood
43be26d1f8
Make local filesystem go last
2013-06-29 12:15:55 +01:00
Nick Craig-Wood
b1f681b912
Rename fs.FsX to fs.X
2013-06-28 08:57:32 +01:00
Nick Craig-Wood
6085dc1b5a
Refactor into sub filesystems
2013-06-27 22:17:51 +01:00