Commit Graph

750 Commits

Author SHA1 Message Date
Nick Craig-Wood
1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d846210978 fs: Add context to NewFs #3257 #4685
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
2020-11-09 18:05:54 +00:00
Adam Plánský
f89ff3872d ncdu: add toggle option for average size in directory - key 'a'
Add toggle option to show average size in directory. This toggle
function is for ncdu and is binded to key 'a'.
2020-10-30 15:33:54 +00:00
Nick Craig-Wood
bfcd4113c3 mount: implement mknod to make NFS file creation work - fixes #2115
It turns out that NFS calls mknod in FUSE even though we have create
defined. This was causing EIO errors when creating files.

This patch fixes it by implementing mknod. The way it is implemented
means that to write to an NFS file system you'll need --vfs-cache-mode
writes.
2020-10-29 15:12:36 +00:00
Nick Craig-Wood
0e7fc7613f mount: make mount be cmount under macOS #4393
This also adds an alias to the mount command so it responds as `rclone
cmount` as well as `rclone mount`.
2020-10-29 13:34:39 +00:00
Nick Craig-Wood
8ac2f52b6e mount: disable bazil/fuse based mount on macOS #4393
The library is no longer supported on macOS.
2020-10-29 13:34:39 +00:00
Josh Soref
8bc9b2b883 docs: grammar: examples are examples
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref
a15f50254a docs: grammar: if, then
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref
3e1cb8302a docs: spelling: etc.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref
e4a87f772f docs: spelling: e.g.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Adam Plánský
e00bf3d723
ncdu: add sort by average size in directory
Add keyboard shortcut 'A' which sort by average size in directory.

If files/folders have same avgSize sort by actual size

Fixes: #4699
2020-10-27 13:28:38 +00:00
Nick Craig-Wood
00a5086ff2 Remove accidentally committed binary and fix formatting
These were both committed in

b7253fc1c1 mount: docs: add note that allow-root and allow-other is not relevant on windows
2020-10-26 19:07:42 +00:00
albertony
b7253fc1c1 mount: docs: add note that allow-root and allow-other is not relevant on windows 2020-10-26 16:21:43 +00:00
Ingo
5164c3d2d0 genautocomplete: add support to output to stdout 2020-10-22 17:28:33 +01:00
David
f4dd8e3fe8 dedupe: minor clarification in docs
dedupe will not delete equal files if they are located in another folder.
2020-10-14 15:31:47 +01:00
Josh Soref
d0888edc0a Spelling fixes
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-14 15:21:31 +01:00
Nick Craig-Wood
72083c65ad cmd: make backend env vars show in help as the defaults for backend flags
Before this change

    RCLONE_DRIVE_CHUNK_SIZE=111M rclone help flags | grep drive-chunk-size

Would show the default value, not the setting of RCLONE_DRIVE_CHUNK_SIZE
as the non backend flags do.

This change makes it work as expected by setting the default of the
option to the environment variable.

Fixes #4659
2020-10-13 15:43:58 +01:00
Ameer Dawood
75a7226174
mount: docs: correction of repeated word 2020-10-07 14:25:31 +01:00
LaSombra
aac2406e19
cmd: add --progress-terminal-title to print ETA to terminal title
Adds a flag, --progress-terminal-title, that when used with --progress,
will print the string `ETA: %s` to the terminal title.

This also adds WriteTerminalTitle to lib/terminal
2020-10-06 16:34:26 +01:00
Hekmon
c58023a9ba enhance systemd integration
* log level identification
* manual activation with flag
* automatic systemd launch detection
2020-10-06 16:21:30 +01:00
Nick Craig-Wood
57261c7e97 mount: docs: remove incorrect statement about --vfs-cache-mode full
See: https://forum.rclone.org/t/is-this-documentation-correct/19376
2020-09-27 11:04:59 +01:00
Russell Cattelan
b14a58c9b8 cmd/mount2: fix the swapped UID / GID values 2020-09-23 23:06:33 +01:00
Evan Harris
e92294b482 docs: Updated mount command to reflect that it requires Go 1.13 or newer 2020-09-08 16:40:43 +01:00
edwardxml
c3d1474eb9 docs: Add full stops for consistency in rclone --help
closes #4560 closes #4561 closes #4562 closes #4563 closes #4564
2020-09-08 16:26:09 +01:00
Nick Craig-Wood
8140f67092 check: fix docs
See: https://forum.rclone.org/t/possible-issue-with-documention/18926
2020-09-07 12:10:52 +01:00
Nick Craig-Wood
4a001b8a02 check: add back missing --download flag - fixes #4565
This was accidentally removed when refactoring check and cryptcheck in

8b6f2bbb4b check,cryptcheck: add reporting of filenames for same/missing/changed #3264
2020-09-05 09:29:35 +01:00
Nick Craig-Wood
70c8566cb8 fs: Pin created backends until parents are finalized
This attempts to solve the backend lifecycle problem by

- Pinning backends mentioned on the command line into the cache
  indefinitely

- Unpinning backends when the containing structure (VFS, wrapping
  backend) is destroyed

See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834
2020-09-01 18:21:03 +01:00
Nick Craig-Wood
75d54d720c version: replace internal code with github.com/coreos/go-semver
We were already importing go-semver so it makes sense to remove the
duplicated semver parsing code and just use go-semver
2020-08-31 13:55:04 +01:00
Nick Craig-Wood
3a14b1d5a9 build: make rclone build with wasm
Needed to drop
- azureblob backend
- cache backend
- qingstor backend
- cachestats command
- ncdu command
2020-08-10 17:32:21 +01:00
David Ibarra
49cf2eb7e4 cmd/obscure: Allow obscure command to accept password on STDIN
`rclone obscure` currently only accepts a command line argument of `password` to generate
an obfuscated password. This is an issue since generating obfuscated passwords programatically
requires sending the plain text password as a shell argument, which can cause problems if the
password contains shell characters, or if the password is from an untrusted source.

This patch opens up STDIN which will allow developers to open the STDIN source and print a password
directly to `rclone obscure`, which can increase safety and convenince.
2020-08-02 11:32:47 +01:00
Nick Craig-Wood
3daa63cae8 mount: fix volume name broken in recent refactor 2020-07-29 14:23:00 +01:00
Nick Craig-Wood
421585dd72 accounting: add context to Account and propagate changes #3257
This is preparation for getting the Accounting to check the context,
buf first we need to get it in place. Since this is one of those
changes that makes lots of noise, this is in a seperate commit.
2020-07-28 16:41:17 +01:00
Nick Craig-Wood
1222b78ec4 cmount: add support for reading unknown length files using direct IO
This means that on Linux and OSX at least reading a google doc from a
mount will behave sensibly.
2020-07-28 16:23:11 +01:00
Nick Craig-Wood
0ee16b51c4 mount: On Windows don't add -o uid/gid=-1 if user supplies -o uid/gid.
Before this change if the user supplied `-o uid=XXX` then rclone would
write `-o uid=-1 -o uid=XXX` so duplicating the uid value.

After this change rclone doesn't write the default `-1` version.

This fix affects `uid` and `gid`.

See: https://forum.rclone.org/t/issue-with-rclone-mount-and-resilio-sync/14730/27
2020-07-28 16:22:29 +01:00
Nick Craig-Wood
62f0bbb598 dedupe: Make it obey the --size-only flag for duplicate detection #4321 2020-07-28 11:40:37 +01:00
Nick Craig-Wood
b2ae94de5b mount: fix mount flags not working
This was broken in the recent refactor.

See: https://forum.rclone.org/t/issue-with-allow-other-in-beta/18133
2020-07-27 15:24:28 +01:00
Jack
d64212d902
serve/restic: expose interfaces so that rclone can be used as a library from within restic
This patch enables rclone to be used as a library from within restic

- exposes NewServer
- exposes Server
- implements http.RoundTripper

Co-authored-by: Jack Deng <jackdeng@gmail.com>
2020-07-26 12:06:47 +01:00
Nick Craig-Wood
3286d1992b mount: warn macOS users that mount implementation is changing #4393 2020-07-24 15:41:31 +01:00
Nick Craig-Wood
4ac662d144 cmount: fix macOS losing directory contents #4393
Before this change when reading directories we would use the directory
handle and the Readdir(-1) call on the directory handle. This worked
fine for the first read, but if the directory was read again on the
same handle Readdir(-1) returns nothing (as per its design).

It turns out that macOS leaves the directory handle open and just
re-reads the data from it, so this problem causes directories to start
out full then subsequently appear empty.

macOS/OSXFUSE is passing an offset of 0 to the Readdir call telling
rclone to seek in the directory, but we've told FUSE that we can't
seek by always returning ofst=0 in the fill function.

This fix works around the problem by reading the directory from the
path each time, ignoring the actual handle. This should be no less
efficient.

We will return an ESPIPE if offset is ever non 0.

There are possible corner cases reading deleted directories which this
ignores.
2020-07-24 15:38:08 +01:00
Nick Craig-Wood
d73a418a55 cmount: always supply stat information in Readdir
It is cheap to make the stat information here - we give FUSE a file
type to look at least.
2020-07-24 15:12:05 +01:00
Nick Craig-Wood
306a3e0cd7 cmount: catch panics in initialization and turn into error messages 2020-07-24 15:12:05 +01:00
Nick Craig-Wood
6ff5787b40 mount: add VFS and Mount options to mount/listmounts 2020-07-24 10:48:51 +01:00
Nick Craig-Wood
3c1c6d2f01 mount: add mountOpt to mount/mount rc 2020-07-24 10:48:51 +01:00
Nick Craig-Wood
0272a7f405 mount: change interface of mount commands to take mount options
This is in preparation of being able to pass mount options to the rc
command "mount/mount"
2020-07-24 10:48:51 +01:00
Nick Craig-Wood
e1d34ef427 mount: factor Mount into mountlib and tidy signal handling
This factors common code from mount, cmount and mount2 into mountlib.

It also uses atexit for unregistering the mount.
2020-07-23 13:08:38 +01:00
Nick Craig-Wood
26b4698212 mount: make mount/mount remote control take vfsOpt option
See: https://forum.rclone.org/t/passing-mount-options-like-vfs-cache-mode-when-using-rclone-rc-mount-mount/17863
2020-07-23 12:30:41 +01:00
Nick Craig-Wood
2871268505 mount: change interface of mount commands to take VFS
This is in preparation of being able to pass options to the rc command
"mount/mount"
2020-07-23 12:30:41 +01:00
Nick Craig-Wood
8b6f2bbb4b check,cryptcheck: add reporting of filenames for same/missing/changed #3264
See: https://forum.rclone.org/t/rclone-check-v-doesnt-show-once-per-minute-update-counts/17402
2020-07-21 22:08:13 +01:00
Nick Craig-Wood
db56b1bfec serve/ftp: use refactored goftp.io/server library for binary shrink
This uses the refactored goftp library which doesn't include the minio
driver. This reduces the binary size by 1.5MB

See: https://gitea.com/goftp/server/pulls/120
2020-07-21 16:23:55 +01:00
Nick Craig-Wood
44ff766f98 mkdir: warn when using mkdir on remotes which can't have empty directories
It is a source of confusion for users that `rclone mkdir` on a remote
which can't have empty directories such as s3/b2 does nothing.

This emits a warning at NOTICE level if the user tries to mkdir a
directory not at the root for a remote which can't have empty
directories.

See: https://forum.rclone.org/t/mkdir-on-b2-consider-adding-some-output/17689
2020-07-08 17:55:58 +01:00
Nick Craig-Wood
ce065614e2 Revert "mount2,cmount: skip unreliable tests #4171"
The VFS is now reliable enough so that the mount tests don't fail.

This reverts commit 4808958f93.
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
af601575cb serve ftp: Use new facilities in goftp to fix and simplify auth proxy #4394
- Use Driver.CheckPasswd instead of server.CheckPasswd
- Make server.CheckPasswd return an error
- Remove awful findID to find parent function hack
- Remove Driver.Init as it is no longer called
- Fix backwards incompatible PublicIp -> PublicIP change

See: https://gitea.com/goftp/server/issues/117
2020-06-30 09:34:13 +01:00
Nick Craig-Wood
92cb21f0f2 serve ftp: Add error message if auth proxy fails #4394 2020-06-29 14:45:39 +01:00
Nick Craig-Wood
0031130111 serve ftp: don't compile on < go1.13 after dependency update 2020-06-29 14:45:39 +01:00
Tim Burke
5050c33162 dlna: Mark flags in docs as code
Otherwise, we get en dashes in the man page, making args more difficult
to copy/paste to a command line.

Before:

    Use –addr to specify ...

After:

    Use --addr to specify ...
2020-06-25 12:18:54 +01:00
Nick Craig-Wood
9445b12328 check: make it show stats by default 2020-06-24 11:23:34 +01:00
Nick Craig-Wood
d8eea0e397 build: run gofmt -s to simplify the code: suggested by Go Report Card 2020-06-18 18:45:39 +01:00
Chaitanya Bankanhal
68afa28b27 rc: Add mount to list if mount point was successfully created 2020-06-16 15:17:55 +01:00
Matteo Pietro Dazzi
62650a3eb3
serve dlna: Fix file list on Samsung Series 6+ TVs
This fixes the command "serve dlna" in order correctly show the list
of files on Samsung TV models starting from Series 6.
2020-06-16 14:56:02 +01:00
Nick Craig-Wood
4808958f93 mount2,cmount: skip unreliable tests #4171 2020-06-15 21:34:37 +01:00
Nick Craig-Wood
117ff1d781 serve sftp: fix race in the tests #4171 2020-06-15 21:34:33 +01:00
Nick Craig-Wood
93bd601149 touch: add ability to set nanosecond resolution times 2020-06-12 15:38:35 +01:00
Nick Craig-Wood
63e6d9d2d1 serve webdav,serve restic: Fix flags so they use environment variables
See: https://forum.rclone.org/t/serve-restic-append-only-environment-variable/17050
2020-06-11 19:28:51 +01:00
Chaitanya
d8d19072c5 mount: Add call for unmount all
mount: handle locking through single mutex.
2020-06-10 22:19:34 +01:00
Chaitanya
830ab37371 rc: Add mount list option for listing current mounts 2020-06-10 22:19:34 +01:00
Nick Craig-Wood
844b903595 docs: promote the use of -i/--interactive and "rclone sync -i" everywhere #1574 2020-06-10 12:33:53 +01:00
Nick Craig-Wood
a3b3e1f646 tree: remove -i shorthand for --noindent as it conflicts with -i/--interactive 2020-06-10 12:33:53 +01:00
Roman Kredentser
55ad1354b6 link: Add --expire and --unlink flags
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
2020-06-05 14:51:05 +01:00
Nick Craig-Wood
4c7f7582fd obscure: write more help as we are referencing it elsewhere 2020-06-05 12:48:28 +01:00
Nick Craig-Wood
e62c032184 docs: remove manually set dates and use git dates instead 2020-06-01 13:07:46 +01:00
Nick Craig-Wood
8774381e2e cmd: Note commands which need obscured input in the docs - fixes #4252 2020-05-31 12:59:09 +01:00
Nick Craig-Wood
d785942ed5 mountlib: fix rc tests when mount does not work
This fixed the Docker 1.52 build
2020-05-28 13:11:42 +01:00
Nick Craig-Wood
74b8cbfb84 docs: set unsafe HTML parsing to false and fix raw HTML insertion
This means that markdown files can't contain <thing> any more.
2020-05-27 17:31:09 +01:00
Nick Craig-Wood
050879f0ca docs: make command docs titles be one higher 2020-05-27 17:31:09 +01:00
Nick Craig-Wood
6d19bbba73 docs: add description to commands and index page 2020-05-27 17:31:08 +01:00
Nick Craig-Wood
2bd786a452 docs: fix rc docs and update anchors for new Hugo version 2020-05-27 17:31:08 +01:00
Martin Michlmayr
7a5b531bd0 docs: fix cosmetics issues 2020-05-26 13:49:09 +01:00
Nick Craig-Wood
31f21551bf mount: change maximum leaf name length to 1024 bytes - fixes #3884
This limit was previously 4k set in 59026c4761 however leaf
names above 1k now produce an IO error.

WinFSP seems to have its own method for dropping too long file names
above 255 long.
2020-05-25 15:41:11 +01:00
Martin Michlmayr
2b72c7f709 docs: move link to correct location
The link is for WinFsp, so link from "WinFsp" rather than "open
source".
2020-05-25 12:04:34 +01:00
Martin Michlmayr
041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Martin Michlmayr
4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Brandon McNama
19ff7c9302 cache: Fix Server Side Copy with Temp Upload
When wrapping a backend that supports Server Side Copy (e.g. `b2`, `s3`)
and configuring the `tmp_upload_path` option, the `cache` backend would
erroneously report that Server Side Copy/Move was not supported, causing
operations such as file moves to fail. This change fixes this issue
under these circumstances such that Server Side Copy will now be used
when the wrapped backend supports it.

Fixes #3206
2020-05-19 12:17:40 +01:00
Martin Michlmayr
fb169a8b54
doc: fix typos throughout docs 2020-05-19 12:02:44 +01:00
Nick Craig-Wood
919a180ad2 config: make config show take "remote:" as well as "remote" 2020-05-18 19:53:13 +01:00
Nick Craig-Wood
98c34e413d config: add --obscure and --no-obscure flags to config create/update
Before this change there was some ambiguity about whether passwords
were obscured on not passing them into config create or config update.

This change adds the --obscure and --no-obscure flags to make the
intent clear.

It also updates the remote control and the tests.

Fixes #3728
2020-05-15 16:41:37 +01:00
Nick Craig-Wood
4006345cfb mountlib: add tests for rc mount/mount and friends 2020-05-14 16:38:37 +00:00
Nick Craig-Wood
1319d7333c mountlib: add rc command mount/types and rename mountOption to mountType 2020-05-14 16:38:37 +00:00
Chaitanya
5f168b3b96 rc: add mount/mount command 2020-05-14 16:38:37 +00:00
Nick Craig-Wood
995cd0dc32 backend: add --json flag to always output JSON 2020-05-13 18:07:41 +01:00
Nick Craig-Wood
5eb558e058 backend: fix output of an array of strings 2020-05-13 17:55:56 +01:00
Nick Craig-Wood
1f005a82ad http: add missing comment to pacify linter 2020-05-10 18:53:38 +01:00
Maxime Suret
79d29bb41e
serve sftp: add support for multiple host keys by repeating --key flag 2020-05-09 14:43:17 +01:00
calisro
c80b6d96dd
http: improved directory listing with new template from Caddy project
This includes a new directory listing template which was originally
from the Caddy project (used with permission and copyright attribution).

This is used whenever we serve directory listings so `rclone serve
http`, `rclone serve webdav` and `rclone rcd --rc-serve`

This also modifies the tests so they work with the original template which
is easier to debug.
2020-05-08 16:15:21 +01:00
Nick Craig-Wood
bdc91eda0f serve http: add Last-Modified headers to files and directories
This means that using `rclone serve http` preserves modification times
when used with the http backend.

Fixes #4201
2020-05-05 09:41:08 +01:00
Nick Craig-Wood
ef9e6794c2 docs: make serve http/webdav template docs into a table 2020-05-04 17:36:31 +00:00
calistri
4362ca7bb9 serve http, serve webdav: Added a --template flag for user defined markup 2020-05-04 17:36:31 +00:00
Nick Craig-Wood
69888bf966 cmount: send a hint as to whether the filesystem is case insensitive or not 2020-05-04 11:38:07 +01:00
Nick Craig-Wood
d260238f99 cmount: use ReaddirPlus on Windows to improve directory listing performance
Before this change Windows would read a directory then immedately stat
every item in the directory.

After this change we return the stat information along with the
directory which stops so many callbacks.
2020-05-04 11:38:07 +01:00
Nick Craig-Wood
6ca7198f57 mount: fix disappearing cwd problem - fixes #4104
Before this change, the current working directory could disappear
according to the Linux kernel.

This was caused by mount returning different nodes with the same
information in.

This change uses vfs.Node.SetSys to cache the information so we always
return the same node.
2020-05-04 11:38:07 +01:00