Commit Graph

4758 Commits

Author SHA1 Message Date
Nick Craig-Wood
306a3e0cd7 cmount: catch panics in initialization and turn into error messages 2020-07-24 15:12:05 +01:00
Nick Craig-Wood
975a53c9e3 build: enable cmount on macOS #4393 2020-07-24 15:12:05 +01:00
Nick Craig-Wood
78fdc5805b vendor: Update github.com/billziss-gh/cgofuse to v1.4.0 #4393 2020-07-24 15:12:05 +01:00
Nick Craig-Wood
8f9d5af26d cache: remove mount tests as they aren't being run and cause maintenance issues
Before this change the cache backend contained its own routines for
mounting testing on that mount.

These tests are never run on the CI and cause a maintenance burden.

This commit removes the tests.
2020-07-24 11:57:49 +01:00
Nick Craig-Wood
6ff5787b40 mount: add VFS and Mount options to mount/listmounts 2020-07-24 10:48:51 +01:00
Nick Craig-Wood
3c1c6d2f01 mount: add mountOpt to mount/mount rc 2020-07-24 10:48:51 +01:00
Nick Craig-Wood
0272a7f405 mount: change interface of mount commands to take mount options
This is in preparation of being able to pass mount options to the rc
command "mount/mount"
2020-07-24 10:48:51 +01:00
Nick Craig-Wood
e1d34ef427 mount: factor Mount into mountlib and tidy signal handling
This factors common code from mount, cmount and mount2 into mountlib.

It also uses atexit for unregistering the mount.
2020-07-23 13:08:38 +01:00
Nick Craig-Wood
26b4698212 mount: make mount/mount remote control take vfsOpt option
See: https://forum.rclone.org/t/passing-mount-options-like-vfs-cache-mode-when-using-rclone-rc-mount-mount/17863
2020-07-23 12:30:41 +01:00
Nick Craig-Wood
2871268505 mount: change interface of mount commands to take VFS
This is in preparation of being able to pass options to the rc command
"mount/mount"
2020-07-23 12:30:41 +01:00
Nick Craig-Wood
744828a4de rc: allow JSON parameters to simplify command line usage
If the parameter being passed is an object then it can be passed as a
JSON string rather than using the `--json` flag which simplifies the
command line.

rclone rc operations/list fs=/tmp remote=test opt='{"showHash": true}'

Rather than

rclone rc operations/list --json '{"fs": "/tmp", "remote": "test", "opt": {"showHash": true}}'
2020-07-22 18:40:52 +01:00
Nick Craig-Wood
ff84351655 operations: factor Check and related functions into its own files 2020-07-21 22:08:13 +01:00
Nick Craig-Wood
8b6f2bbb4b check,cryptcheck: add reporting of filenames for same/missing/changed #3264
See: https://forum.rclone.org/t/rclone-check-v-doesnt-show-once-per-minute-update-counts/17402
2020-07-21 22:08:13 +01:00
Nick Craig-Wood
d2efb4b29b ftp: add support for --dump bodies and --dump auth
See: https://forum.rclone.org/t/rclone-copy-gives-error-connection-reset-by-peer-using-ftp/17934/27
2020-07-21 16:26:31 +01:00
Nick Craig-Wood
db56b1bfec serve/ftp: use refactored goftp.io/server library for binary shrink
This uses the refactored goftp library which doesn't include the minio
driver. This reduces the binary size by 1.5MB

See: https://gitea.com/goftp/server/pulls/120
2020-07-21 16:23:55 +01:00
Nick Craig-Wood
990a33b393 build: go mod tidy 2020-07-21 16:19:24 +01:00
Nick Craig-Wood
664c658da6 ftp: Update github.com/jlaffaye/ftp to fix interop with pure-ftpd
See: https://github.com/jlaffaye/ftp/pull/190
2020-07-21 16:17:37 +01:00
Nick Craig-Wood
d1617ce7ce Stop doing vendoring - fixes #4032 2020-07-21 16:09:53 +01:00
Nick Craig-Wood
2b50d44a2f Remove vendor directory #4032 2020-07-21 16:09:53 +01:00
Nick Craig-Wood
ddfde68140 vfs: fix directory locking caused by slow directory listings
Before this fix we took the directory lock to read the ModTime of the
directory. This was causing locking on directories which were being
re-read from the backend.

This commit gives the modtime its own lock so it can be read even when
the directory is being updated.

See: https://forum.rclone.org/t/high-cpu-load-with-rclone-mount/17604
2020-07-18 09:08:18 +01:00
Nick Craig-Wood
811b30d116 sync: fix deadlock with --track-renames-strategy modtime - fixes #4427
Before this change we could exit the popRenameMap function with the
lock held.

This fixes the problem by defer-ring the unlock.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992
2020-07-17 17:09:58 +01:00
Nick Craig-Wood
bcd362fcd5 accounting: fix documentation for speed/speedAvg
Fix the documentation for the very confusingly names speed and
speedAvg stats items.

See: https://github.com/rclone/rclone-webui-react/issues/99
2020-07-17 15:35:15 +01:00
Nick Craig-Wood
1fafcd4d28 Add Dmitry Ustalov to contributors 2020-07-15 23:15:06 +01:00
Nick Craig-Wood
2807a85f68 Add Morten Linderud to contributors 2020-07-15 23:15:06 +01:00
Nick Craig-Wood
39515acf68 Add Kevin to contributors 2020-07-15 23:15:06 +01:00
Dmitry Ustalov
aaed74fe4e
docs: workaround and policy for Google Drive API
* workaround for Google Drive API
* mention the use of Google User Data
* unified wording for user data policy
2020-07-15 23:14:39 +01:00
Nick Craig-Wood
126efaadcc vfs: fix renamed files not being uploaded with --vfs-cache-mode minimal
Before this change files that were in the cache and renamed with
--vfs-cache-mode minimal weren't renamed at all.

This fixes the problem and adds tests for all the different
combinations of cache modes and in and out of the cache.
2020-07-15 16:22:12 +01:00
Nick Craig-Wood
2adc057d95 vfs: fix very high load caused by slow directory listings
In this commit (released in v1.52.0)

6ca7198f mount: fix disappearing cwd problem

SetSys was introduced to cache node lookups.

Unfortunately taking the vfs.(*Dir) lock in SetSys causes any FUSE
operations on a directory to pile up behind slow directory listings.

In some situations this leads to very high load.

This commit fixes it by using atomic operations to read and write the
Sys value make it independent of the lock.

See: https://forum.rclone.org/t/high-cpu-load-with-rclone-mount/17604
See: #4104
2020-07-15 16:22:12 +01:00
Nick Craig-Wood
59770a4953 ftp: add note to docs about home vs root directory selection
See: https://forum.rclone.org/t/update-docs-for-ftp-path-absolute-vs-relative/17875
2020-07-15 15:27:47 +01:00
Morten Linderud
67098511db make_manual: Support SOURCE_DATE_EPOCH
The documentation contains an embedded datetime which do not read
SOURCE_DATE_EPOCH. This makes the documentation unreproducible when
distribution tries to recreate previous build of the package.

This patch ensures we attempt to read the environment variable before
default to the current build time.

Signed-off-by: Morten Linderud <morten@linderud.pw>
2020-07-15 13:23:08 +01:00
Kevin
07b2ce4ab2
Avoid comma rendered in URL in onedrive.md (#4438)
Removed comma from the end of the Azure AD Applications List Blade URL since it was not resolving and customers were opening up support tickets with the Microsoft Azure AD team.
2020-07-15 15:55:00 +08:00
Nick Craig-Wood
80d2f38192 s3: fix bucket Region auto detection when Region unset in config #2915
Previous to this fix if Region was not set and Endpoint was not set
then we set the endpoint to "https://s3.amazonaws.com/".

This is unecessary because if the Region alone isn't set then we set
it to "us-east-1" which has the same endpoint.

Having the endpoint set breaks the bucket region auto detection with
the error "Failed to update region for bucket: can't set region to
"xxx" as endpoint is set".

This fix removes that check.
2020-07-10 17:16:59 +01:00
Nick Craig-Wood
0792f4722c swift: fix purge not deleting directory markers
At some point Purge stopped deleting directory markers. We don't have
an integration test for this so it went unnoticed.

This patch fixes the problem but doesn't introduce an integration test
as we don't have a framework for making directory markers yet.
2020-07-10 15:16:11 +01:00
Nick Craig-Wood
db37360a1d swift: fix dangling large objects breaking the listing
Before this change, large objects which had had their contents deleted
would return "Object not found" and break the listing.

This change makes these objects appear as 0 sized entities so they can
be listed and deleted.
2020-07-10 11:03:08 +01:00
Nick Craig-Wood
44ff766f98 mkdir: warn when using mkdir on remotes which can't have empty directories
It is a source of confusion for users that `rclone mkdir` on a remote
which can't have empty directories such as s3/b2 does nothing.

This emits a warning at NOTICE level if the user tries to mkdir a
directory not at the root for a remote which can't have empty
directories.

See: https://forum.rclone.org/t/mkdir-on-b2-consider-adding-some-output/17689
2020-07-08 17:55:58 +01:00
Max Sum
bfa5715017 fs/accounting: sort transfers by start time 2020-07-07 15:42:06 +01:00
Max Sum
e2183ad661 fs/accounting: use transferMap instead of stringSet 2020-07-07 15:42:06 +01:00
Nick Craig-Wood
e2201689cf build: add ARMv7 to the supported builds - fixes #748 2020-07-07 12:13:20 +01:00
Nick Craig-Wood
0f72aa8a5f docs: add section about config variable precedence
See: https://forum.rclone.org/t/precedence-rules-for-config/17707
2020-07-07 11:12:35 +01:00
Nick Craig-Wood
b2f4f52b64 vfs cache: make logging consistent and remove some debug logging 2020-07-06 17:32:53 +01:00
Nick Craig-Wood
c65ed26a7e vfs: vfscache: Fix renaming of items while they are being uploaded
Previous to the fix, if an item was being uploaded and it was renamed,
the upload would fail with missing checksum errors.

This change cancels any uploads in progress if the file is renamed.
2020-07-06 17:32:53 +01:00
Nick Craig-Wood
df5dbaf49b vfs: writeback: add Rename call for renaming items in the writeback queue 2020-07-06 17:32:53 +01:00
Nick Craig-Wood
80fe1f16db b2: note that b2's encoding now allows \ but rclone's hasn't changed
See: https://forum.rclone.org/t/why-are-there-error-messages-about-non-existing-files-in-the-log/17608
2020-07-06 16:28:30 +01:00
Nick Craig-Wood
f524a4c1cc vendor: drop unused github.com/djherbis/times 2020-07-06 14:29:42 +01:00
Nick Craig-Wood
c61c3cddbd Add Evan Harris to contributors 2020-07-06 14:29:42 +01:00
Nick Craig-Wood
51767aee23 Add Garrett Squire to contributors 2020-07-06 14:29:42 +01:00
Evan Harris
cd3d7e2dca
docs: Update install.md to reflect minimum Go version
Fixes #3765
2020-07-06 13:42:47 +01:00
Garrett Squire
4f7f5404ce
build: fix file handle leak in GitHub release tool
This is a small patch to remove a defer statement found in a for loop.
It instead closes the file after it is done copying the bytes from the
tar file reader.
2020-07-04 10:51:37 +01:00
Nick Craig-Wood
d4b2709fb0 pcloud: fix oauth on European region "eapi.pcloud.com"
Pcloud appears to have opened up a new region and they are returning
the hostname in the oauth callback, thus

    GET /?code=XXX&locationid=1&hostname=api.pcloud.com&state=XXX HTTP/1.1
    GET /?code=XXX&locationid=2&hostname=eapi.pcloud.com&state=XXX HTTP/1.1

This isn't documented yet, however pCloud have confirmed that this is
the correct interpretation.

Rclone now reads the "hostname" parameter in the oauth callback and
stores it in the config file. It uses it for all subequent API calls.
2020-07-03 20:38:42 +01:00
Nick Craig-Wood
e6fdc3a932 drive: make dangling shortcuts appear in listings
Previous to this a dangling shortcut would error the directory
listing.

This patch makes dangling shortcuts appear as 0 sized objects in the
directory listing so they can be deleted. These objects can't be read
though.
2020-07-02 22:12:44 +01:00