In v2.5.0, upstream health was fixed such that whether an upstream is
considered healthy or not is mostly up to each individual handler's
config. Since "healthy" is an opinion, it is not a global value.
I unintentionally left in the "healthy" field in the API endpoint for
checking upstreams, and it is now misleading (see #4792).
However, num_requests and fails remains, so health can be determined by
the API client, rather than having it be opaquely (and unhelpfully)
determined for the client.
If we do restore this value later on, it'd need to be replicated once
per reverse_proxy handler according to their individual configs.
* reverseproxy: Improve hashing LB policies with HRW
Previously, if a list of upstreams changed, hash-based LB policies
would be greatly affected because the hash relied on the position of
upstreams in the pool. Highest Random Weight or "rendezvous" hashing
is apparently robust to pool changes. It runs in O(n) instead of
O(log n), but n is very small usually.
* Fix bug and update tests
* reverseproxy: Sync up `handleUpgradeResponse` with stdlib
I had left this as a TODO for when we bump to minimum 1.17, but I should've realized it was under `internal` so it couldn't be used directly.
Copied the functions we needed for parity. Hopefully this is ok!
* Add tests and fix godoc comments
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
Includes several breaking changes; code base updated accordingly.
- Added lots of context arguments
- Use fs.ErrNotExist
- Rename ACMEManager -> ACMEIssuer; CertificateManager -> Manager
* caddyfile: Support for raw token values, improve `map`, `expression`
* Applied code review comments
* Rename RawVal to ValRaw
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* reverseproxy: New `copy_response` handler for `handle_response` routes
Followup to #4298 and #4388.
This adds a new `copy_response` handler which may only be used in `reverse_proxy`'s `handle_response` routes, which can be used to actually copy the proxy response downstream.
Previously, if `handle_response` was used (with routes, not the status code mode), it was impossible to use the upstream's response body at all, because we would always close the body, expecting the routes to write a new body from scratch.
To implement this, I had to refactor `h.reverseProxy()` to move all the code that came after the `HandleResponse` loop into a new function. This new function `h.finalizeResponse()` takes care of preparing the response by removing extra headers, dealing with trailers, then copying the headers and body downstream.
Since basically what we want `copy_response` to do is invoke `h.finalizeResponse()` at a configurable point in time, we need to pass down the proxy handler, the response, and some other state via a new `req.WithContext(ctx)`. Wrapping a new context is pretty much the only way we have to jump a few layers in the HTTP middleware chain and let a handler pick up this information. Feels a bit dirty, but it works.
Also fixed a bug with the `http.reverse_proxy.upstream.duration` placeholder, it always had the same duration as `http.reverse_proxy.upstream.latency`, but the former was meant to be the time taken for the roundtrip _plus_ copying/writing the response.
* Delete the "Content-Length" header if we aren't copying
Fixes a bug where the Content-Length will mismatch the actual bytes written if we skipped copying the response, so we get a message like this when using curl:
```
curl: (18) transfer closed with 18 bytes remaining to read
```
To replicate:
```
{
admin off
debug
}
:8881 {
reverse_proxy 127.0.0.1:8882 {
@200 status 200
handle_response @200 {
header Foo bar
}
}
}
:8882 {
header Content-Type application/json
respond `{"hello": "world"}` 200
}
```
* Implement `copy_response_headers`, with include/exclude list support
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* opentelemetry: create a new module
* fix imports
* fix test
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddyhttp/opentelemetry/tracer.go
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* rename error ErrUnsupportedTracesProtocol
* replace spaces with tabs in the test data
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* replace spaces with tabs in the README.md
* use default values for a propagation and exporter protocol
* set http attributes with helper
* simplify code
* Cleanup modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update link in README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update documentation in README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update link to naming spec in README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Rename module from opentelemetry to tracing
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Rename span_name to span
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Rename span_name to span
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Simplify otel resource creation
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* handle extra attributes
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* update go.opentelemetry.io/otel/semconv to 1.7.0
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* update go.opentelemetry.io/otel version
* remove environment variable handling
* always use tracecontext,baggage as propagators
* extract tracer name into variable
* rename OpenTelemetry to Tracing
* simplify resource creation
* update go.mod
* rename package from opentelemetry to tracing
* cleanup tests
* update Caddyfile example in README.md
* update README.md
* fix test
* fix module name in README.md
* fix module name in README.md
* change names in README.md and tests
* order imports
* remove redundant tests
* Update documentation README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Fix grammar
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update comments
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update comments
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* update go.sum
* update go.sum
* Add otelhttp instrumentation, update OpenTelemetry libraries.
* Use otelhttp instrumentation for instrumenting HTTP requests.
This change uses context.WithValue to inject the next handler into the
request context via a "nextCall" carrier struct, and pass it on to a
standard Go HTTP handler returned by otelhttp.NewHandler. The
underlying handler will extract the next handler from the context,
call it and pass the returned error to the carrier struct.
* use zap.Error() for the error log
* remove README.md
* update dependencies
* clean up the code
* change comment
* move serveHTTP method from separate file
* add syntax to the UnmarshalCaddyfile comment
* go import the file
* admin: Write proper status on invalid requests (#4569) (fix#4561)
* update dependencies
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Vibhav Pant <vibhavp@gmail.com>
Co-authored-by: Alok Naushad <alokme123@gmail.com>
Co-authored-by: Cedric Ziel <cedric@cedric-ziel.com>
* reverseproxy: Begin refactor to enable dynamic upstreams
Streamed here: https://www.youtube.com/watch?v=hj7yzXb11jU
* Implement SRV and A/AAA upstream sources
Also get upstreams at every retry loop iteration instead of just once
before the loop. See #4442.
* Minor tweaks from review
* Limit size of upstreams caches
* Add doc notes deprecating LookupSRV
* Provision dynamic upstreams
Still WIP, preparing to preserve health checker functionality
* Rejigger health checks
Move active health check results into handler-specific Upstreams.
Improve documentation regarding health checks and upstreams.
* Deprecation notice
* Add Caddyfile support, use `caddy.Duration`
* Interface guards
* Implement custom resolvers, add resolvers to http transport Caddyfile
* SRV: fix Caddyfile `name` inline arg, remove proto condition
* Use pointer receiver
* Add debug logs
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* reverseproxy: Make shallow-ish clone of the request
* Refactor request cloning into separate function
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
If .ts.net domains are explicitly added to config,
don't try to manage a cert for them (it will fail, and our
implicit Tailscale module will
get those certs at run-time).
* Update matchers.go
* Update matchers.go
* implementation of zone_id handling
* last changes in zone handling
* give return true values instead of bool
* Apply suggestions from code review
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* changes as suggested
* Apply suggestions from code review
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Update matchers.go
* shortened the Match function
* changed mazcher handling
* Update matchers.go
* delete space
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Huge thank-you to Tailscale (https://tailscale.com) for making this change possible!
This is a great feature for Caddy and Tailscale is a great fit for a standard implementation.
* caddytls: GetCertificate modules; Tailscale
* Caddyfile support for get_certificate
Also fix AP provisioning in case of empty subject list (persist loaded
module on struct, much like Issuers, to surive reprovisioning).
And implement start of HTTP cert getter, still WIP.
* Update modules/caddytls/automation.go
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Use tsclient package, check status for name
* Implement HTTP cert getter
And use reuse CertMagic's PEM functions for private keys.
* Remove cache option from Tailscale getter
Tailscale does its own caching and we don't need the added complexity...
for now, at least.
* Several updates
- Option to disable cert automation in auto HTTPS
- Support multiple cert managers
- Remove cache feature from cert manager modules
- Minor improvements to auto HTTPS logging
* Run go mod tidy
* Try to get certificates from Tailscale implicitly
Only for domains ending in .ts.net.
I think this is really cool!
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
The TestFileListing test in tplcontext_test has one test that verifies
if directory traversal is not happening. The context root is set to
'/tmp' and then it tries to open '../../../../../etc', which gets
normalized to '/tmp/etc'.
The test then expects an error to be returned, assuming that '/tmp/etc'
does not exist on the system. When it does exist, it results in a test
failure:
```
--- FAIL: TestFileListing (0.00s)
tplcontext_test.go:422: Test 4: Expected error but had none
FAIL
FAIL
github.com/caddyserver/caddy/v2/modules/caddyhttp/templates 0.042s
```
Instead of using '/tmp' as root, use a dedicated directory created with
`os.MkdirTemp()` instead. That way, we know that the directory is empty.
* caddyhttp: Redirect HTTP requests on the HTTPS port to https://
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
This is a followup to #4407, in response to a report on the forums: https://caddy.community/t/php-fastcgi-phishing-redirection/14542
Turns out that doing `TrimRight` to remove trailing dots, _before_ cleaning the path, will cause double-dots at the end of the path to not be cleaned away as they should. We should instead remove the dots _after_ cleaning.
* caddyhttp: Enhance vars matcher
Enable "or" logic for multiple values.
Fall back to checking placeholders if not a var name.
* Fix tests (thanks @mohammed90 !)
* fastcgi: Fix a TODO, prevent zap using reflection for logging env
* Update modules/caddyhttp/reverseproxy/fastcgi/fastcgi.go
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* reverseproxy: Adjust defaults, document defaults
Related to some of the issues in https://github.com/caddyserver/caddy/issues/4245, a complaint about the proxy transport defaults not being properly documented in https://caddy.community/t/default-values-for-directives/14254/6.
- Dug into the stdlib to find the actual defaults for some of the timeouts and buffer limits, documenting them in godoc so the JSON docs get them next release.
- Moved the keep-alive and dial-timeout defaults from `reverseproxy.go` to `httptransport.go`. It doesn't make sense to set defaults in the proxy, because then any time the transport is configured with non-defaults, the keep-alive and dial-timeout defaults are lost!
- Sped up the dial timeout from 10s to 3s, in practice it rarely makes sense to wait a whole 10s for dialing. A shorter timeout helps a lot with the load balancer retries, so using something lower helps with user experience.
* reverseproxy: Make keepalive interval configurable via Caddyfile
* fastcgi: DialTimeout default for fastcgi transport too
Fixes#4428
It's best to still log handler errors at debug level so that they're hidden by default, but still accessible if additional details are necessary.
This makes it easier for users to find the default browse template if they
want to create a custom template based on that. It also makes it easier to
view the template with proper syntax highlighting.
* caddyhttp: Sanitize scheme and host on incoming requests
* reverseproxy: Sanitize the URL scheme and host before proxying
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Update tplcontext.go
Add {{ render "/path/to/file.ext" $data }} via funcRender
* Update tplcontext.go
* Refactor funcInclude, add funcImport to enable {{block}} and {{template}}
* Fix funcImport return of nil showing up in html
* Update godocs for and
* Add tests for funcInclude
* Add tests for funcImport
* os.RemoveAll -> os.Remove for TestFuncInclude and TestFuncImport
Related to (closed) Issue #2094 on template inheritance. This PR adds a new function called "import" which works like "include", except it only takes one argument and passes it to the referenced file to be used as "." in that file.
* Update tplcontext.go
Add {{ render "/path/to/file.ext" $data }} via funcRender
* Update tplcontext.go
* Refactor funcInclude, add funcImport to enable {{block}} and {{template}}
* Fix funcImport return of nil showing up in html
* Update godocs for and
* caddyhttp: Add support for triggering errors from `try_files`
* caddyhttp: Use vars instead of placeholders/replacer for matcher errors
* caddyhttp: Add comment for matcher error var key
* encode: ignore flushing until after first write (fix#4314)
The first write will determine if encoding has to be done and will add an Content-Encoding. Until then Flushing has to be delayed so the Content-Encoding header can be added before headers and status code is written. (A passthrough flush would write header and status code)
* Update modules/caddyhttp/encode/encode.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
From reading through the code, I think this code path is now obsoleted by the changes made in https://github.com/caddyserver/caddy/pull/4266.
Basically, `h.flushInterval()` will set the flush interval to `-1` if we're in a bi-directional stream, and the recent PR ensured that `h.copyResponse()` properly flushes headers immediately when the flush interval is non-zero. So now there should be no need to call Flush before calling `h.copyResponse()`.
This commit fixes the `sortByNameDirFirst` variable inside fileserver to
match what browse's default template has.
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Tweak compression settings
zstd: Limit window sizes to 128K to keep memory in control both server and client size.
zstd: Write 0 length frames. This may be needed for compatibility.
zstd: Create fewer encoders. Small memory improvement.
gzip: Allow -2 (Huffman only) and -3 (stateless) compression modes.
* Update modules/caddyhttp/encode/zstd/zstd.go
Update docs.
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
This is the more correct implementation of 23dadc0d86 (#4179)... I think. This commit effectively undoes the revert in 8848df9c5d, but with corrections to the logic.
We *do* need to use the original request path (the path the browser knows) for redirects, since they are external, and rewrites are only internal.
However, if the path was rewritten to a non-canonical path, we should not redirect to canonicalize that, since rewrites are intentional by the site owner. Canonicalizing the path involves modifying only the suffix (base element, or filename) of the path. Thus, if a rewrite involves only the prefix (like how handle_path strips a path prefix), then we can (hopefully!) safely redirect using the original URI since the filename was not rewritten.
So basically, if rewrites modify the filename, we should not canonicalize those requests. If rewrites only modify another part of the path (commonly a prefix), we should be OK to redirect.
Templates are parsed at request-time (like they are in the templates middleware) to allow live changes to the template while the server is running. Fixes race condition.
Also refactored use of a buffer so a buffer put back in the pool will not continue to be used (written to client) in the meantime.
A couple of benchmarks removed due to refactor, which is fine, since we know pooling helps here.
Also split the Caddyfile subdirective keepalive_idle_conns into two properties so the conns and conns_per_host can be set separately.
This is technically a breaking change, but probably anyone who this breaks already had a broken config anyway, and silently fixing it won't help them fix their configs.
Turns out this was an oversight, we assumed we could use `{http.response.header.*}` but that doesn't work because those are grabbed from the response writer, and we haven't copied any headers into the response writer yet.
So the fix is to set all the response headers into the replacer at a new namespace before running the handlers.
This adds the `{http.reverse_proxy.header.*}` replacer.
See https://caddy.community/t/empty-http-response-header-x-accel-redirect/12447
* caddyfile(formatter): fix nesting not decrementing
This is an extremely weird edge-case where if you had a environment variable {}
on one line, a comment on the next line, and the closing of the block on the
following line; the rest of the Caddyfile would be indented further than it
should've been.
ref; https://github.com/matthewpi/vscode-caddyfile-support/issues/13
* run gofmt
* fmt: better way of handling edge case
Followup to #4150, #4151 /cc @ueffel @polarathene
After a bit of discussion with @mholt, we decided to remove `prefer` as a subdirective and just go with using the order implicitly always. Simpler config, simpler docs, etc.
Effectively changes 7776471 and reverts a small part of f35a7fa.
* caddyhttp: Fix fallback for the error handler chain
The fix I went with in the end (after realizing some mistaken assumptions in #4131) is to just make the routes fall back to errorEmptyHandler instead of the non-error empty handler, if Terminal is true, making the routes error-aware. Ultimately this was probably just an oversight when errors was implemented at some point in the early betas of v2.
See https://caddy.community/t/problem-with-basicauth-handle-errors/12243/9 for context.
* Revert "caddyhttp: Fix fallback for the error handler chain"
This reverts commit 95b6ac44a6.
* caddyhttp: Fix via `routes.go`
* fileserver: Fix `file` matcher with empty `try_files`
Fixes https://github.com/caddyserver/caddy/issues/4146
If `TryFiles` is empty, we fill it with `r.URL.Path`. In this case, this is `/`. Then later, in `prepareFilePath()`, we run the replacer (which turns `{path}` into `/` at that point) but `file` remains the original value (and the placeholder is still the placeholder there).
So then `strings.HasSuffix(file, "/")` will be `false` for the placeholder, but `true` for the empty `TryFiles` codepath, because `file` was `/` due to being set to the actual request value beforehand.
This means that `suffix` becomes `//` in that case, so after `sanitizedPathJoin`, it becomes `./`, so `strictFileExists`'s `strings.HasSuffix(file, separator)` codepath will return true.
I think we should change the `m.TryFiles == nil` codepath to `m.TryFiles = []string{"{http.request.uri.path}"}` for consistency. (And maybe consider hoisting this to `Provision` cause there's no point doing this on every request). I don't think this "optimization" of directly using `r.URL.Path` is so valuable, cause it causes this edgecase with directories.
* Update modules/caddyhttp/fileserver/matcher.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* reverseproxy: Add `handle_response` blocks to `reverse_proxy` (#3710)
* reverseproxy: complete handle_response test
* reverseproxy: Change handle_response matchers to use named matchers
reverseproxy: Add support for changing status code
* fastcgi: Remove obsolete TODO
We already have d.Err("transport already specified") in the reverse_proxy parsing code which covers this case
* reverseproxy: Fix support for "4xx" type status codes
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* caddyhttp: Reorganize response matchers
* reverseproxy: Reintroduce caddyfile.Unmarshaler
* reverseproxy: Add comment mentioning Finalize should be called
Co-authored-by: Maxime Soulé <btik-git@scoubidou.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Below is the report using `benchstat` and cmd:
`go test -run=BenchmarkHeaderREMatcher -bench=BenchmarkHeaderREMatcher -benchmem -count=10`
```
name old time/op new time/op delta
HeaderREMatcher-16 869ns ± 1% 658ns ± 0% -24.29% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
HeaderREMatcher-16 144B ± 0% 112B ± 0% -22.22% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
HeaderREMatcher-16 7.00 ± 0% 5.00 ± 0% -28.57% (p=0.000 n=10+10)
```
* caddyhttp: reverseproxy: fix hash selection policy
Fixes: #4135
Test: go test './...' -count=1
* caddyhttp: reverseproxy: add test to catch #4135
If you revert the last commit, the test will fail.
