This is the more correct implementation of 23dadc0d86 (#4179)... I think. This commit effectively undoes the revert in 8848df9c5d, but with corrections to the logic.
We *do* need to use the original request path (the path the browser knows) for redirects, since they are external, and rewrites are only internal.
However, if the path was rewritten to a non-canonical path, we should not redirect to canonicalize that, since rewrites are intentional by the site owner. Canonicalizing the path involves modifying only the suffix (base element, or filename) of the path. Thus, if a rewrite involves only the prefix (like how handle_path strips a path prefix), then we can (hopefully!) safely redirect using the original URI since the filename was not rewritten.
So basically, if rewrites modify the filename, we should not canonicalize those requests. If rewrites only modify another part of the path (commonly a prefix), we should be OK to redirect.
Templates are parsed at request-time (like they are in the templates middleware) to allow live changes to the template while the server is running. Fixes race condition.
Also refactored use of a buffer so a buffer put back in the pool will not continue to be used (written to client) in the meantime.
A couple of benchmarks removed due to refactor, which is fine, since we know pooling helps here.
Also split the Caddyfile subdirective keepalive_idle_conns into two properties so the conns and conns_per_host can be set separately.
This is technically a breaking change, but probably anyone who this breaks already had a broken config anyway, and silently fixing it won't help them fix their configs.
While the Caddy project has had very few valid security bug reports over the years, we have a low signal-to-noise ratio with them (lots of invalid reports). Most are out of scope, and it can take too much valuable time for us to determine that. We would prefer researchers do this first. Hopefully these paragraphs spell out much more clearly what we do and don't accept.
* Force auto-renew for OCSP revoked status (maybe) (fix#4191)
* Use latest commit
* go.mod: Use CertMagic v0.14.0 (fix#4191)
Correctly replaces revoked certificates
In the Caddyfile, hosts specified for HTTP sockets (either scheme is "http" or it is on the HTTP port) should not be used as subjects in TLS automation policies (APs).
* logging: Implement dial timeout for net writer (fix#4083)
* Limit how often redials are attempted
This should cause dial blocking to occur only once every 10 seconds at most, but it also means the logger connection might be down for up to 10 seconds after it comes back online; oh well. We shouldn't block for DialTimeout at every single log emission.
* Clarify offline behavior
* caddyfile: Add parse error on site address in `{`
This is an incredibly common mistake made by users, so we should catch it earlier in the parser and give a more friendly message. Often it ends up adapting but with mistakes, or erroring out later due to other site addresses being read as directives.
There's not really ever a situation where a lone '{' is valid at the end of a site address (but I suppose there are edgecases where the user wants to use a path matcher where it ends specifically in `{`, but... why?), so this should be fine.
* Update caddyconfig/caddyfile/parse.go
Turns out this was an oversight, we assumed we could use `{http.response.header.*}` but that doesn't work because those are grabbed from the response writer, and we haven't copied any headers into the response writer yet.
So the fix is to set all the response headers into the replacer at a new namespace before running the handlers.
This adds the `{http.reverse_proxy.header.*}` replacer.
See https://caddy.community/t/empty-http-response-header-x-accel-redirect/12447
We decided that we'll use branches like `2.4` as the target for any changes that we might want to release in a `2.4.x` version like `2.4.1`, so that we can continue to merge changes targeting the next minor release (e.g. `2.5.0`) on master.
Our CI config wasn't set up for this to work properly though, since it was only running checks on PRs targeting master. This should fix it.
I couldn't find a way to do a pattern to only match digits for the branch names from Github's docs, it just looks like a pretty generic glob syntax. But this should do until we get to 3.0
* caddyfile(formatter): fix nesting not decrementing
This is an extremely weird edge-case where if you had a environment variable {}
on one line, a comment on the next line, and the closing of the block on the
following line; the rest of the Caddyfile would be indented further than it
should've been.
ref; https://github.com/matthewpi/vscode-caddyfile-support/issues/13
* run gofmt
* fmt: better way of handling edge case
Followup to #4150, #4151 /cc @ueffel @polarathene
After a bit of discussion with @mholt, we decided to remove `prefer` as a subdirective and just go with using the order implicitly always. Simpler config, simpler docs, etc.
Effectively changes 7776471 and reverts a small part of f35a7fa.
* caddyhttp: Fix fallback for the error handler chain
The fix I went with in the end (after realizing some mistaken assumptions in #4131) is to just make the routes fall back to errorEmptyHandler instead of the non-error empty handler, if Terminal is true, making the routes error-aware. Ultimately this was probably just an oversight when errors was implemented at some point in the early betas of v2.
See https://caddy.community/t/problem-with-basicauth-handle-errors/12243/9 for context.
* Revert "caddyhttp: Fix fallback for the error handler chain"
This reverts commit 95b6ac44a6.
* caddyhttp: Fix via `routes.go`
* fileserver: Fix `file` matcher with empty `try_files`
Fixes https://github.com/caddyserver/caddy/issues/4146
If `TryFiles` is empty, we fill it with `r.URL.Path`. In this case, this is `/`. Then later, in `prepareFilePath()`, we run the replacer (which turns `{path}` into `/` at that point) but `file` remains the original value (and the placeholder is still the placeholder there).
So then `strings.HasSuffix(file, "/")` will be `false` for the placeholder, but `true` for the empty `TryFiles` codepath, because `file` was `/` due to being set to the actual request value beforehand.
This means that `suffix` becomes `//` in that case, so after `sanitizedPathJoin`, it becomes `./`, so `strictFileExists`'s `strings.HasSuffix(file, separator)` codepath will return true.
I think we should change the `m.TryFiles == nil` codepath to `m.TryFiles = []string{"{http.request.uri.path}"}` for consistency. (And maybe consider hoisting this to `Provision` cause there's no point doing this on every request). I don't think this "optimization" of directly using `r.URL.Path` is so valuable, cause it causes this edgecase with directories.
* Update modules/caddyhttp/fileserver/matcher.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* reverseproxy: Add `handle_response` blocks to `reverse_proxy` (#3710)
* reverseproxy: complete handle_response test
* reverseproxy: Change handle_response matchers to use named matchers
reverseproxy: Add support for changing status code
* fastcgi: Remove obsolete TODO
We already have d.Err("transport already specified") in the reverse_proxy parsing code which covers this case
* reverseproxy: Fix support for "4xx" type status codes
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* caddyhttp: Reorganize response matchers
* reverseproxy: Reintroduce caddyfile.Unmarshaler
* reverseproxy: Add comment mentioning Finalize should be called
Co-authored-by: Maxime Soulé <btik-git@scoubidou.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Below is the report using `benchstat` and cmd:
`go test -run=BenchmarkHeaderREMatcher -bench=BenchmarkHeaderREMatcher -benchmem -count=10`
```
name old time/op new time/op delta
HeaderREMatcher-16 869ns ± 1% 658ns ± 0% -24.29% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
HeaderREMatcher-16 144B ± 0% 112B ± 0% -22.22% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
HeaderREMatcher-16 7.00 ± 0% 5.00 ± 0% -28.57% (p=0.000 n=10+10)
```
* httpcaddyfile: Fix unexpectedly removed policy
When user set on_demand tls option in a catch-all (:443) policy,
we expect other policies to not have the on_demand enabled
See ex in tls_automation_policies_5.txt
Btw, we can remove policies if they are **all** empty.
* Update caddyconfig/httpcaddyfile/tlsapp.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* caddyhttp: reverseproxy: fix hash selection policy
Fixes: #4135
Test: go test './...' -count=1
* caddyhttp: reverseproxy: add test to catch #4135
If you revert the last commit, the test will fail.
* caddyfile: Fix `import` replacing unrelated placeholders
See https://caddy.community/t/snippet-issue-works-outside-snippet/12231
So it turns out that `NewReplacer()` gives a replacer with some global defaults (like `{env.*}` and some system and time placeholders), which is not ideal when running `import` because we just want to replace `{args.*}` only, and nothing else.
* caddyfile: Add test
An idea that came up in https://caddy.community/t/save-internally-issued-wildcard-certificate-in-consul/11740, this a simple module that might be useful for anyone who uses storage modules that aren't filesystem, to let them load certs/keys externally issued for use by Caddy.
Bit goofy, since we need to fetch the certmagic.Storage during provisioning, it needs a wrapping struct instead of just being an array like `load_files`.
Future work might involve adding Caddyfile support via a subdirective of the `tls` directive maybe?
