Commit Graph

171 Commits

Author SHA1 Message Date
Chaitanya
d8d19072c5 mount: Add call for unmount all
mount: handle locking through single mutex.
2020-06-10 22:19:34 +01:00
Chaitanya
830ab37371 rc: Add mount list option for listing current mounts 2020-06-10 22:19:34 +01:00
Nick Craig-Wood
d785942ed5 mountlib: fix rc tests when mount does not work
This fixed the Docker 1.52 build
2020-05-28 13:11:42 +01:00
Martin Michlmayr
7a5b531bd0 docs: fix cosmetics issues 2020-05-26 13:49:09 +01:00
Nick Craig-Wood
31f21551bf mount: change maximum leaf name length to 1024 bytes - fixes #3884
This limit was previously 4k set in 59026c4761 however leaf
names above 1k now produce an IO error.

WinFSP seems to have its own method for dropping too long file names
above 255 long.
2020-05-25 15:41:11 +01:00
Martin Michlmayr
2b72c7f709 docs: move link to correct location
The link is for WinFsp, so link from "WinFsp" rather than "open
source".
2020-05-25 12:04:34 +01:00
Martin Michlmayr
041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Martin Michlmayr
4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Nick Craig-Wood
4006345cfb mountlib: add tests for rc mount/mount and friends 2020-05-14 16:38:37 +00:00
Nick Craig-Wood
1319d7333c mountlib: add rc command mount/types and rename mountOption to mountType 2020-05-14 16:38:37 +00:00
Chaitanya
5f168b3b96 rc: add mount/mount command 2020-05-14 16:38:37 +00:00
Nick Craig-Wood
fd39cbc193 vfstests: move functional tests from mountlib and make them work with VFS
The tests are now run for the mount commands and for the plain VFS.

This makes the tests much easier to debug when running with a VFS than
through a mount.
2020-04-20 10:42:33 +01:00
Nick Craig-Wood
66e08e0cc8 mount: warn if --allow-non-empty used on Windows and clarify docs 2020-03-31 12:16:03 +01:00
evileye
6ff0bb825e mount: fix fail because of too long volume name - fixes #4026 2020-03-05 13:57:20 +00:00
Nick Craig-Wood
e3c11c9ca1 mount: add --async-read flag to disable asynchronous reads
See: https://forum.rclone.org/t/constantly-high-iowait-add-log/14156
2020-02-27 16:12:33 +00:00
Nick Craig-Wood
ec1271818f mount2: hide mount2 command for the moment 2020-02-11 14:28:13 +00:00
Nick Craig-Wood
a54210a2e4 docs: restore missing mount --daemon docs
This was done as part of ebfeec9fb4 which unfortunately patched
the auto generated files.
2020-02-10 15:29:39 +00:00
Nick Craig-Wood
feee92c790 docs: restore lost mount share docs
These came from 162fdfe455 which unfortunately added the docs to
the auto generated files.
2020-02-10 15:29:39 +00:00
Nick Craig-Wood
a4bc4daf30 mounttest: fix unreliable tests on Windows CI
The failure is this which is not reproducable locally, only on the CI
servers.

    --- FAIL: TestMount/CacheMode=minimal/TestWriteFileOverwrite (1.01s)
        fs.go:351:
            Error Trace:    fs.go:351
                            write.go:65
            Error:          Received unexpected error:
                            open E:testwrite: The request could not be performed because of an I/O device error.
            Test:           TestMount/CacheMode=minimal/TestWriteFileOverwrite

The corresponding ERROR from the log is this:

    ERROR : IO error: truncate C:\Users\runneradmin\AppData\Local\rclone\vfs\local\C\Users\RUNNER~1\AppData\Local\Temp\rclone298719627\testwrite: Access is denied.

Instead of using ioutil.WriteFile this fix uses an equivalent based on
rclone's lib/file which doesn't set the exclusive flag on
Windows. This allows files to be deleted that are open.  It also
deletes existing files if an error is received and retries.
2020-01-09 11:11:49 +00:00
Nick Craig-Wood
e0d9314059 mounttest: fix occasionally failing test TestRenameOpenHandle 2019-11-11 16:20:27 +00:00
Brett Dutro
378a3f4133 mount: replace use of WriteAt with Write for cache mode >= writes and O_APPEND
os.File.WriteAt returns an error if a file was opened with O_APPEND.
This replaces it with os.File.Write if the file was opened with
O_APPEND.
2019-10-26 17:27:52 +01:00
Nick Craig-Wood
59026c4761 mount, cmount: don't pass huge filenames (>4k) to FUSE as it can't cope 2019-10-21 20:51:11 +01:00
Nick Craig-Wood
1b2ffbeca0 cmd: fix environment variables not setting command line flags
Before this fix quite a lot of the commands were ignoring environment
variables intended to set flags.
2019-10-14 17:02:09 +01:00
Nick Craig-Wood
19429083ad cmd: fix spelling of Definition 2019-10-14 17:02:09 +01:00
Nick Craig-Wood
b4b59c53f1 mount: fix "mount_fusefs: -o timeout=: option not supported" on FreeBSD
Before this change `rclone mount` would give this error on FreeBSD

    mount helper error: mount_fusefs: -o timeout=: option not supported

Because the default value for FreeBSD was set to 15m for
--daemon-timeout and that FreeBSD does not support the timeout option.

This change sets the default for --daemon-timeout to 0 on FreeBSD
which fixes the problem.

Fixes #3610
2019-10-13 11:36:51 +01:00
Brett Dutro
7d0d7e66ca vfs: move writeback of dirty data out of close() method into its own method (FlushWrites) and remove close() call from Flush()
If a file handle is duplicated with dup() and the duplicate handle is
flushed, rclone will go ahead and close the file, making the original
file handle stale. This change removes the close() call from Flush() and
replaces it with FlushWrites() so that the file only gets closed when
Release() is called. The new FlushWrites method takes care of actually
writing the file back to the underlying storage.

Fixes #3381
2019-10-09 10:07:29 +01:00
Nick Craig-Wood
6ca00c21a4 mount: update docs to show mounting from root OK for bucket based #3421 2019-08-17 10:30:41 +01:00
Nick Craig-Wood
af3c47d282 fstest: remove -subdir flag as it no longer tests anything useful #3421 2019-08-17 09:54:19 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
2019-07-28 18:47:38 +01:00
Nick Craig-Wood
4e64ee38e2 mount: default --deamon-timout to 15 minutes on macOS and FreeBSD
See: https://forum.rclone.org/t/macos-fuse-mount-contents-disappear-after-writes-while-using-vfs-cache/10566/
2019-06-25 15:30:42 +01:00
Aleksandar Jankovic
f78cd1e043 Add context propagation to rclone
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
2019-06-19 11:59:46 +01:00
Nick Craig-Wood
184a9c8da6 mountlib: clip blocks returned to 32 bit number for Windows 32 bit - fixes #2934 2019-01-27 12:04:56 +00:00
Nick Craig-Wood
571b4c060b mount: check that mountpoint and local directory to mount don't overlap
If the mountpoint and the directory to mount overlap this causes a
lockup.

Fixes #2905
2019-01-10 14:18:00 +00:00
Nick Craig-Wood
26b5f55cba Update after goimports change 2018-12-04 10:11:57 +00:00
Peter Kaminski
63a2a935fc fix typos in original files, per #2727 review request 2018-11-14 22:48:58 +00:00
Nick Craig-Wood
87e1efa997 mount, vfs: Remove EXPERIMENTAL tags
rclone mount and the --vfs-cache-mode has been tested extensively by
users now so removing the EXPERIMENTAL tag is appropriate.
2018-10-06 11:47:46 +01:00
Nick Craig-Wood
06ae4258be cmd: Fix -P not ending with a new line
Before this fix rclone didn't wait for the stats to be finished before
exiting, so the final new line was never printed.

After this change rclone will wait for the stats routine to cease
before exiting.
2018-10-03 21:46:18 +01:00
Nick Craig-Wood
174ca22936 mount,cmount: clip the number of blocks to 2^32-1 on macOS
OSX FUSE only supports 32 bit number of blocks which means that block
counts have been wrapping.  This causes f_bavail to be 0 which in turn
causes problems with programs like borg backup.

Fixes #2356
2018-08-26 17:32:59 +01:00
Alex Chen
c6c74cb869 mountlib: fix mount --daemon not working with encrypted config - fixes #2473
This passes the configKey to the child process as an Obscured temporary file with an environment variable to the
2018-08-21 09:41:16 +01:00
Oleg Kovalov
06c9f76cd2 all: fix go-critic linter suggestions 2018-08-06 21:14:03 +01:00
Nick Craig-Wood
473e3c3eb8 mount/cmount: implement --daemon-timeout flag for OSXFUSE
By default the timeout is 60s which isn't long enough for long
transactions.  The symptoms are rclone just quitting for no reason.
Supplying the --daemon-timeout flag fixes this causing the kernel to
wait longer for rclone.
2018-07-19 13:26:51 +01:00
Stefan
4009fb67c8
fs: calculate ModifyWindow each time on the fly instead of relying on global state - see #2319, #2328 2018-06-03 20:45:34 +02:00
Stefan
6607d8752c
mountlib: add testcase to ensure the ModifyWindow is calculated on Mount (see #2002) (#2319) 2018-05-28 17:49:26 +02:00
Stefan
67e9ef4547
mount: delay rename if file has open writers instead of failing outright - fixes #2130 (#2249) 2018-05-24 20:45:11 +02:00
Nick Craig-Wood
291954baba cmd: make names of argument parsing functions more consistent 2018-05-07 20:39:52 +01:00
Nick Craig-Wood
9d8d7ae1f0 mount,cmount: make --noappledouble --noapplexattr and change defaults #2287
Before this change we would unconditionally set the OSXFUSE options
noappledouble and noapplexattr.

However the noapplexattr options caused problems with copies in the
Finder.

Now the default for noapplexattr is false so we don't add the option
by default and the user can override the defaults using the
--noappledouble and --noapplexattr flags.
2018-05-07 20:37:09 +01:00
Nick Craig-Wood
6ce32e4661 mount,cmount: Add --volname flag and remove special chars from it #2287
Before this change rclone would set the volume name from the
remote:path normally.  However this has `:` and `/` in which make it
difficult to use in macOS.

Now rclone will remove the special characters and replace them with
spaces.  It also allows the volume name to be set with the --volname
flag.
2018-05-07 20:37:09 +01:00
Nick Craig-Wood
f1a43eca4d mount: make --daemon work for macOS without CGO 2018-05-05 16:23:47 +01:00
Fabian Möller
bd3ad1ac3e vfs: add option to read source files in chunks 2018-05-05 12:49:42 +01:00
Nick Craig-Wood
2b7957cc74 vfs: Only make the VFS cache if --vfs-cache-mode > Off
This stops the cache cleaner running unnecessarily and saves
resources.

This also helps with issue #2227 which was caused by a second mount
deleting objects in the first mounts cache.
2018-04-16 17:06:41 +01:00
Nick Craig-Wood
06e3fa3aba mounttest: reduce duplicated code and improve test output #2154
The written out list of tests was replaced with a nested test for
mount and cmount. The tests for each VFS cache mode were also replaced
with nested tests which makes the output and the code much cleaner.
2018-04-08 15:04:14 +01:00
Nick Craig-Wood
e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood
98a924602f mount, cmount: set --attr-timeout default to 1s - fixes #2157
This  works around these 3 problems:

  * rclone using too much memory #2157
  * rclone not serving files to samba
    * https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112
  * excessive time listing directories #2095
2018-03-23 22:42:51 +00:00
Nick Craig-Wood
66c69fe620 mount: wait longer for consistency after rm in tests 2018-03-09 23:15:38 +00:00
Nick Craig-Wood
7713acf23d mount: skip failing test TestFileModTimeWithOpenWriters on Windows 2018-03-09 23:15:38 +00:00
Nick Craig-Wood
473a388f6d mount: disable failing test TestWriteFileDoubleClose on OSX 2018-03-09 23:15:37 +00:00
Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it (#2094)
* fs: rename DirChangeNotify to ChangeNotify

* cache: switch to ChangeNotify

* ChangeNotify: keep order of notifications
2018-03-08 22:03:34 +02:00
Nick Craig-Wood
fc32fee4ad mount, cmount: add --attr-timeout to control attribute caching in kernel
This flag allows the attribute caching in the kernel to be controlled.
The default is 0s - no caching - which is recommended for filesystems
which can change outside the control of the kernel.

Previously this was at the default meaning it was 60s for mount and 1s
for cmount.  This showed strange effects when files changed on the
remote not via the kernel.  For instance Caddy would serve corrupted
files for a while when serving from an rclone mount when a file
changed on the remote.
2018-03-04 11:20:22 +00:00
ishuah
ebfeec9fb4 mount: run rclone mount in the background - fixes #723 2018-03-04 14:06:07 +03:00
Nick Craig-Wood
0f787e43b0 mount: link the nssm service manager for mount under Windows 2018-02-12 11:29:32 +00:00
Nick Craig-Wood
3a7bb7b2df mount: update docs showing --vfs-cache-mode to work around limitations 2018-02-12 11:29:32 +00:00
Jody Frankowski
5e73acd40a Clean up mount.go and vfs/help.go docs
* Title cleanups
* Typos
* `rclone mount vs rclone sync/copy` update with `File Caching`
2018-01-20 18:14:20 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts.
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
2018-01-15 17:51:14 +00:00
Nick Craig-Wood
b8b620f5c2 Move all backends into backend directory 2018-01-12 17:07:38 +00:00
Nick Craig-Wood
6d3df6f172 cmount: make tests more reliable on Windows 2017-11-28 20:39:24 +00:00
Nick Craig-Wood
af50f31f7d mounttest: wait for Release after every Read to stop using in use files under Windows 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
00b46a8b96 mounttest: wait for files to disappear from the directory listing 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
b21f227bd3 mounttest: fix crash when FUSE not present 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
e98e550021 mounttest: wait for all background Close/Release after writing a file
The filesystem does a certain amount of things asynchronously waiting
for the file to be released after writing it means everything should
be in a consistent state.
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
eb3415db50 cmount: enable more tests for Windows 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
9fbd8a6419 mounttest: fixes for running under Windows
* don't mount and unmount between cache runs - WinFSP doesn't suport it
  * use OS paths for opening things
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
981cfb1bec mounttest: retry directory listings to account for slow updates on Windows 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
57a5c67729 mounttest: run the tests for all 4 VFS cache modes 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
d7908c06c9 mountlib: ensure we don't open files with read and write intent 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
8951875c21 vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
05a1e1532b vfs,mount,cmount,serve: Add documentation for vfs caching modes 2017-11-20 12:46:21 +00:00
Fabian Möller
d8b3bf014d mount: use sdnotify to signal systemd the mount is ready
When the NOTIFY_SOCKET environment variable is set notify systemd after
the mount is ready.
2017-11-20 11:03:10 +00:00
ishuah
31d2fb4e11 mount: Fix mount breaking on Windows - fixes #1827 2017-11-16 15:20:53 +03:00
Ernest Borowski
9fbff7bcab mountlib: check if directory is not empty before mounting - fixes #1386
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2017-11-14 21:24:31 +00:00
Nick Craig-Wood
e18122e88b vfs: add tests and subsequent fixes
* Tests for VFS layer
  * Small fixes found during testing
  * Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
  * Fix nil object bugs on File
2017-11-04 10:24:10 +00:00
Nick Craig-Wood
e8883e9fdb vfs: factor flags into vfsflags and remove global variables 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
c1aaff220d Factor new vfs module out of cmd/mountlib
This is an OS style file system abstraction with directory caching
used in mount, cmount, serve webdav and serve http.
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
6da6b2556b mountlib: make directory entries be returned in sorted order 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
ca19fd2d7e mountlib: Make read/write file handles support more standard interfaces
Including Read, ReadAt, Seek, Close for read handles and Write,
WriteAt, Close for read handles.
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
2fac74b517 mountlib: store only Node in *Dir removing DirEntry struct 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
8b6daaa877 mountlib: add DirEntry() to Node interface 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
3af9d63261 mountlib: add Remove and RemoveAll methods to Node 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
c6cd2a5280 mountlib: add parent and entry to Dir 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
0bb84efe75 mountlib: Rename Remove to RemoveName 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
3ec15ac2bd mountlib: make sure Node is always set in DirEntry
This simplifies the code and makes using the DirEntry.Node usable when
using ReadDir.
2017-11-03 12:59:58 +00:00
Nick Craig-Wood
750690503e mountlib: make Node satisfy os.FileInfo interface 2017-11-03 12:59:58 +00:00
Nick Craig-Wood
54950d3423 mountlib: make more useful as a general purpose file system adaptor 2017-11-03 12:59:58 +00:00
Stefan Breunig
12405f9f41 fuse: re-use rcat to support uploads for all remotes (fixes #1672) 2017-09-16 22:49:08 +02:00
Nick Craig-Wood
ebc8361933 mount: Add notes on Windows limitations from Bill Zissimopoulos 2017-07-26 21:08:24 +01:00
Nick Craig-Wood
cd4895690a fstest: Factor test initialisation into Initialise() 2017-07-26 16:38:33 +01:00
Nick Craig-Wood
de54fd4c64 mount: add docs for windows install 2017-07-23 13:05:02 +01:00
Nick Craig-Wood
c223464cd0 mount: fix panic on renames - fixes #1533
Make sure d.items is not nil and improve locking
2017-07-22 11:00:51 +01:00
Nick Craig-Wood
e7d04fc103 Create fs.Directory interface and use it everywhere 2017-06-30 14:26:59 +01:00
Nick Craig-Wood
e2d7d413ef fs: rename BasicInfo to DirEntry 2017-06-30 14:26:58 +01:00
Nick Craig-Wood
b360527931 mount: fix hang on errored upload
In certain circumstances if an upload failed then the mount could hang
indefinitely. This was fixed by closing the read pipe after the Put
completed.  This will cause the write side to return a pipe closed
error fixing the hang.

Fixes #1498
2017-06-26 12:08:51 +01:00