Commit Graph

1067 Commits

Author SHA1 Message Date
Nick Craig-Wood
37d786c82a selfupdate: fix "invalid hashsum signature" error
This was caused by a change to the upstream library
ProtonMail/go-crypto checking the flags on the keys more strictly.

However the signing key for rclone is very old and does not have those
flags. Adding those flags using `gpg --edit-key` and then the
`change-usage` subcommand to remove, save, quite then re-add, save
quit the signing capabilities caused the key to work.

This also adds tests for the verification and adds the selfupdate
tests into the integration test harness as they had been disabled on
CI because they rely on external sources and are sometimes unreliable.

Fixes #7373
2023-10-18 17:55:19 +01:00
Nick Craig-Wood
29baa5888f mount: fix automount not detecting drive is ready
With automount the target mount drive appears twice in /proc/self/mountinfo.

    379 27 0:70 / /mnt/rclone rw,relatime shared:433 - autofs systemd-1 rw,fd=57,...
    566 379 0:90 / /mnt/rclone rw,nosuid,nodev,relatime shared:488 - fuse.rclone remote: rw,...

Before this fix we only looked for the mount once in
/proc/self/mountinfo. It finds the automount line and since this
doesn't have fs type rclone it concludes the mount isn't ready yet.

This patch makes rclone look through all the mounts and if any of them
have fs type rclone it concludes the mount is ready.

See: https://forum.rclone.org/t/systemd-mount-works-but-automount-does-not/42287/
2023-10-16 12:13:20 +01:00
Nick Craig-Wood
c190b9b14f serve sftp: return not supported error for not supported commands
Before this change, if a hardlink command was issued, rclone would
just ignore it and not return an error.

This changes any unknown operations (including hardlink) to return an
unsupported error.
2023-10-16 12:08:22 +01:00
Nick Craig-Wood
5afa838457 cmd: Make --progress output logs in the same format as without
See: https://forum.rclone.org/t/using-progress-change-dates-from-2023-10-05-to-2023-10-05/42173
2023-10-11 11:36:31 +01:00
Saleh Dindar
bcb3289dad nfsmount: documentation for new NFS mount feature for macOS 2023-10-06 14:08:20 +01:00
Saleh Dindar
ef2ef8ef84 nfsmount: New mount command to provide mount mechanism on macOS without FUSE
Summary:
In cases where cmount is not available in macOS, we alias nfsmount to mount command and transparently start the NFS server and mount it to the target dir.

The NFS server is started on localhost on a random port so it is reasonably secure.

Test Plan:
```
go run rclone.go mount --http-url https://beta.rclone.org :http: nfs-test
```

Added mount tests:
```
go test ./cmd/nfsmount
```
2023-10-06 14:08:20 +01:00
Saleh Dindar
c69cf46f06 serve nfs: new serve nfs command
Summary:
Adding a new command to serve any remote over NFS. This is only useful for new macOS versions where FUSE mounts are not available.
 * Added willscot/go-nfs dependency and updated go.mod and go.sum

Test Plan:
```
go run rclone.go serve nfs --http-url https://beta.rclone.org :http:
```

Test that it is serving correctly by mounting the NFS directory.

```
mkdir nfs-test
mount -oport=58654,mountport=58654 localhost: nfs-test
```

Then we can list the mounted directory to see it is working.
```
ls nfs-test
```
2023-10-06 14:08:20 +01:00
Nick Craig-Wood
a752563842 operations: add operations/check to the rc API
Fixes #7015
2023-10-04 17:52:57 +01:00
Nick Craig-Wood
9959712a06 docs: fix backend doc generator to not output duplicate config names
This was always the intention, it was just implemented wrong.

This shortens the s3 docs by 1369 bringing them down to half the size
just about.

Fixes #7325
2023-09-23 12:54:08 +01:00
Dimitri Papadopoulos Orfanos
3d473eb54e
docs: fix typos found by codespell in docs and code comments 2023-09-23 12:20:01 +01:00
Nick Craig-Wood
9370dbcc47 lsjson: make sure we set the global metadata flag too 2023-09-21 16:15:38 +01:00
Nick Craig-Wood
8c1e9a2905 rc: always report an error as JSON
Before this change, the rclone rc command wouldn't actually report the
error as a JSON blob which is inconsitent with what the HTTP API does.

This change make sure we always report a JSON error, making a
synthetic one if necessary.

See: https://forum.rclone.org/t/when-using-rclone-rc-commands-somehow-return-errors-as-parsable-json/41855
Co-authored-by: Fawzib Rojas
2023-09-20 21:57:40 +01:00
Nick Craig-Wood
e8879f3e77 docs: document release signing and verification
Fixes #7257
2023-09-11 12:28:23 +01:00
nielash
48ab67f090 bisync: fix dryRun rc parameter being ignored
Before this change, bisync ignored the dryRun parameter (only when specified
via the rc.)

This change fixes the issue, so that the dryRun rc parameter is equivalent to
the --dry-run flag.
2023-09-05 08:53:58 +01:00
nielash
089df7d977 bisync: add rc parameters for new flags
Added rc support for the flags recently introduced in #6971.

createEmptySrcDirs
ignoreListingChecksum
resilient
2023-09-05 08:53:58 +01:00
eNV25
ad724463a5
cmd: refactor and use sysdnotify in more commands
* cmd: refactor and use sysdnotify in more commands

Fixes #5117
2023-09-04 16:32:04 +01:00
Nick Craig-Wood
85c29e3629 serve dlna: fix MIME type if backend can't identify it
If the server returns the MIME type as application/octet-stream we
assume it doesn't really know what the MIME type. This patch tries
matching the MIME type from the file extension instead in this case.

This enables the use of servers (like OneDrive for Business) which
don't allow the setting of MIME types on upload and have a poor
selection of mime types.

Fixes #7259
2023-09-01 18:09:44 +01:00
eNV25
11eeaaf792
cmd/ncdu: fix add keybinding to rescan filesystem 2023-08-30 14:29:46 +01:00
eNV25
0bd0a992a4 cmd/ncdu: add keybinding to rescan filesystem
Fixes #7255
2023-08-30 09:05:58 +01:00
Nick Craig-Wood
d61328e459 serve ftp: fix race condition when using the auth proxy
In this commit we introduced a race condition when using the auth
proxy.

94a320f23c serve ftp: update to goftp.io/server v2.0.1

This was due to the re-organisation of the upstream library which made
the driver be a singleton rather than per session.

This means that when using the auth proxy we need to keep track of
which VFS to use by based on which FTP user is connected.

This also adjusts the locking so that the methods will run
concurrently.
2023-08-23 15:11:47 +01:00
Nick Craig-Wood
94a320f23c serve ftp: update to goftp.io/server v2.0.1 - fixes #7237 2023-08-22 17:24:05 +01:00
Nick Craig-Wood
7fc573db27 serve sftp: fix hash calculations with --vfs-cache-mode full
Before this change uploading files with rclone to:

    rclone serve sftp --vfs-cache-mode full

Would return the error:

    command "md5sum XXX" failed with error: unexpected non file

This patch detects that the file is still in the VFS cache and reads
the MD5SUM from there rather from the remote.

Fixes #7241
2023-08-22 13:18:36 +01:00
Roberto Ricci
50d0597d56 cmount: use atomic types 2023-08-22 12:52:13 +01:00
Nick Craig-Wood
03aab1a123 rmdirs: remove directories concurrently controlled by --checkers
See: https://forum.rclone.org/t/how-to-list-empty-directories-recursively/40995
2023-08-18 12:05:15 +01:00
nielash
0dd0d6a13e bisync: Add support for --create-empty-src-dirs - Fixes #6109
Sync creation and deletion of empty directories.
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=3.%20Bisync%20should%20create/delete%20empty%20directories%20as%20sync%20does%2C%20when%20%2D%2Dcreate%2Dempty%2Dsrc%2Ddirs%20is%20passed

Also fixed an issue causing --resync to erroneously delete empty folders and duplicate files unique to Path2
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=2.%20%2D%2Dresync%20deletes%20data%2C%20contrary%20to%20docs
2023-08-12 17:24:21 +01:00
nielash
e5bde42303 bisync: Add experimental --resilient mode to allow recovery from self-correctable errors
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=2.%20Bisync%20should%20be%20more%20resilient%20to%20self%2Dcorrectable%20errors
2023-08-12 17:24:21 +01:00
nielash
f01a50eb47 bisync: Add new --ignore-listing-checksum flag to distinguish from --ignore-checksum
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=6.%20%2D%2Dignore%2Dchecksum%20should%20be%20split%20into%20two%20flags%20for%20separate%20purposes
2023-08-12 17:24:21 +01:00
nielash
5ca61ab705 bisync: equality check before renaming (leave identical files alone)
Improved detection of false positive change conflicts (identical files are now left alone instead of renamed)
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Identical%20files%20should%20be%20left%20alone%2C%20even%20if%20new/newer/changed%20on%20both%20sides
2023-08-12 17:24:21 +01:00
nielash
4ac4ce6afd bisync: apply filters correctly during deletes
Fixed an issue causing bisync to consider more files than necessary due to overbroad filters during delete operations
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=5.%20Bisync%20reads%20files%20in%20excluded%20directories%20during%20delete%20operations
2023-08-12 17:24:21 +01:00
nielash
40a874a0d8 bisync: enforce --check-access during --resync
--check-access is now enforced during --resync, preventing data loss in certain user error scenarios
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=%2D%2Dcheck%2Daccess%20doesn%27t%20always%20fail%20when%20it%20should
2023-08-12 17:24:21 +01:00
nielash
f4dd86238d bisync: dry runs no longer commit filter changes
Fixed an issue causing dry runs to inadvertently commit filter changes
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Dry%20runs%20are%20not%20completely%20dry
2023-08-12 17:24:21 +01:00
Nick Craig-Wood
d362db2e08 rclone test info: add --check-base32768 flag to check can store all base32768 characters
Fixes #7208
2023-08-05 02:59:58 +01:00
Nick Craig-Wood
bff702a6f1 docs: group the global flags and make them appear on command and flags pages
This adds an additional parameter to the creation of each flag. This
specifies one or more flag groups. This **must** be set for global
flags and **must not** be set for local flags.

This causes flags.md to be built with sections to aid comprehension
and it causes the documentation pages for each command (and the
`--help`) to be built showing the flags groups as specified in the
`groups` annotation on the command.

See: https://forum.rclone.org/t/make-docs-for-mortals-not-only-rclone-gurus/39476/
2023-08-02 12:53:09 +01:00
Nick Craig-Wood
a1d6bbd31f Add rclone completion powershell - basic implementation only 2023-08-02 12:53:09 +01:00
Nick Craig-Wood
fb6a9dfbf3 docs: fix rclone config edit docs 2023-08-02 12:53:09 +01:00
Nick Craig-Wood
3f3c5f3ff4 build: remove unused package cmd/serve/http/data
This was superseded by lib/http/template.go
2023-08-02 12:53:09 +01:00
Nick Craig-Wood
29b1751d0e serve webdav: fix error: Expecting fs.Object or fs.Directory, got <nil>
Before this change rclone serve webdav would sometimes give this error

    Expecting fs.Object or fs.Directory, got <nil>

It turns out that when a file is being updated it doesn't have a
DirEntry and it is allowed to be <nil> so in this case we create the
mime type from the extension.

See: https://forum.rclone.org/t/webdav-union-of-onedrive-expecting-fs-object-or-fs-directory-got-nil/40298
2023-07-28 00:54:45 +01:00
gabriel-suela
9169b2b5ab cmd: fix log message typo 2023-07-23 08:43:03 +09:00
Anagh Kumar Baranwal
bb0cd76a5f
fix: mount parsing for linux 2023-07-22 17:29:20 +05:30
kapitainsky
378a2d21ee --max-transfer - add new exit code (10)
It adds dedicated exit code (10) for --max-duration flag.

Rclone will exit with exit code 10 if the duration limit is reached.

It behaves in similar fashion as --max-transfer and exit code 8.

discussed on the forum:

https://forum.rclone.org/t/max-duration-option-is-triggering-exit-with-error/39917/6
2023-07-18 09:51:31 +01:00
Nick Craig-Wood
d0d41fe847 rclone config redacted: implement support mechanism for showing redacted config
This introduces a new fs.Option flag, Sensitive and uses this along
with IsPassword to redact the info in the config file for support
purposes.

It adds this flag into backends where appropriate. It was necessary to
add oauthutil.SharedOptions to some backends as they were missing
them.

Fixes #5209
2023-07-07 16:25:14 +01:00
kapitainsky
998d1d1727 docs: listremotes also includes remotes from env vars 2023-06-24 15:46:23 +01:00
Anagh Kumar Baranwal
d06bb55f3f mount: Added _netdev to the example mount so it gets treated as a remote-fs rather than local-fs
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2023-06-23 17:37:00 +01:00
URenko
acd67edf9a docs: remove "After" in systemd mount example again 2023-06-22 18:03:04 +01:00
Nick Craig-Wood
7842000f8a backend: for command not found errors, hint to look in the underlying remote
See: https://forum.rclone.org/t/rclone-cleanup-no-way-to-delete-pending-uploads-newer-than-24-hours/38416/6
2023-06-10 14:44:01 +01:00
Nick Craig-Wood
51a468b2ba genautocomplete: rename to completion with alias to old name
This brings it into line with cobra's naming scheme and stops cobra
writing another "completion" command which doesn't work as well which
confuses users.

See: https://forum.rclone.org/t/rclone-genautocomplete-bash-vs-rclone-completion-bash-neither-works-fully/38431
2023-05-25 14:32:40 +01:00
WeidiDeng
ceb9406c2f
serve webdav: implement owncloud checksum and modtime extensions
* implement owncloud checksum and modtime extensions for webdav server
* test rclone webdav server as owncloud webdav
2023-05-15 15:38:00 +01:00
Anagh Kumar Baranwal
2520c05c4b mount2: disable xattrs
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2023-04-30 17:56:47 +01:00
Anagh Kumar Baranwal
f7f5e87632 mount2: fixed statfs
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2023-04-30 17:56:47 +01:00
Anagh Kumar Baranwal
a7e6806f26 mount2: updated go-fuse version
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2023-04-30 17:56:47 +01:00