* reverseproxy: do not parse upstream address too early if it contains replaceble parts
* remove unused method
* cleanup
* accommodate partially replaceable port
* caddyhttp: Make use of http.ResponseController
Also syncs the reverseproxy implementation with stdlib's which now uses ResponseController as well 2449bbb5e6
* Enable full-duplex for HTTP/1.1
* Appease linter
* Add warning for builds with Go 1.20, so it's less surprising to users
* Improved godoc for EnableFullDuplex, copied text from stdlib
* Only wrap in encode if not already wrapped
* added weighted round robin algorithm to load balancer
* added an adapt integration test for wrr and fixed a typo
* changed args format to Caddyfile args convention
* added provisioner and validator for wrr
* simplified the code and improved doc
* caddyhttp: Determine real client IP if trusted proxies configured
* Support customizing client IP header
* Implement client_ip matcher, deprecate remote_ip's forwarded option
* httpcaddyfile: Skip some logic if auto_https off
* Try removing this check altogether...
* Refine test timeouts slightly, sigh
* caddyhttp: Assume udp for unrecognized network type
Seems like the reasonable thing to do if a plugin registers its own
network type.
* Add comment to document my lack of knowledge
* Clean up and prepare to merge
Add comments to try to explain what happened
* httpcaddyfile: Fix `protocols` global option parsing
When checking for a block, the current nesting must be used, otherwise it returns the wrong thing.
* Adjust adapt test to cover the broken behaviour that is now fixed
* Fix some admin tests which suddenly run even with -short
* caddyhttp: Honor grace period in background
This avoids blocking during config reloads.
* Don't quit process until servers shut down
* Make tests more likely to pass on fast CI (#5045)
* caddyhttp: Even faster shutdowns
Simultaneously shut down all HTTP servers, rather than one at a time.
In practice there usually won't be more than 1 that lingers. But this
code ensures that they all Shutdown() in their own goroutine
and then we wait for them at the end (if exiting).
We also wait for them to start up so we can be fairly confident the
shutdowns have begun; i.e. old servers no longer
accepting new connections.
* Fix comment typo
* Pull functions out of loop, for readability
Previously, our "duplicate key in server block" logic was flawed because
it did not account for the site's bind address. We defer this check to
when the listener addresses have been assigned, but before we commit
a server block to its listener.
Also refined how network address parsing and joining works, which was
necessary for a less convoluted fix.
* reverseproxy: Implement retry count, alternative to try_duration
* Add Caddyfile support for `retry_match`
* Refactor to deduplicate matcher parsing logic
* Fix lint
* Make reverse proxy TLS server name replaceable for SNI upstreams.
* Reverted previous TLS server name replacement, and implemented thread safe version.
* Move TLS servername replacement into it's own function
* Moved SNI servername replacement into httptransport.
* Solve issue when dynamic upstreams use wrong protocol upstream.
* Revert previous commit.
Old commit was: Solve issue when dynamic upstreams use wrong protocol upstream.
Id: 3c9806ccb6
* Added SkipTLSPorts option to http transport.
* Fix typo in test config file.
* Rename config option as suggested by Matt
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Update code to match renamed config option.
* Fix typo in config option name.
* Fix another typo that I missed.
* Tests not completing because of apparent wrong ordering of options.
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* httpcaddyfile: Add `{vars.*}` placeholder shortcut
I'm yoinking this from my https://github.com/caddyserver/caddy/pull/4657 PR because I think we should get this in ASAP for v2.5.0 along with the new `vars` directive.
* Sort vars by matchers in reverse
Guh, this is complicated.
Fixes#4640
This also follows up on #4398 (reverting it) which made a change that technically worked, but was incorrect. It changed the condition in `hostsFromKeysNotHTTP` from `&&` to `||`, but then the function no longer did what its name said it would do, and it would return hosts even if they were marked with `http://`, if they used a non-HTTP port. That wasn't the intent of it. The test added in there was kept though, because it is a valid usecase.
The actual fix is to check _earlier_ whether all the addresses explicitly have `http://`, and if so we can short circuit and skip considering the rest.
Lots of the files were using CRLF instead of LF. Mostly my fault cause sometimes I make the files on Windows and VSCode for some reason kept making them with the wrong line endings. Sigh.
Since .txt files typically default to spaces for indentation, I'm also adding an .editorconfig to ensure they use tabs instead
* caddyfile: Support for raw token values, improve `map`, `expression`
* Applied code review comments
* Rename RawVal to ValRaw
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* reverseproxy: New `copy_response` handler for `handle_response` routes
Followup to #4298 and #4388.
This adds a new `copy_response` handler which may only be used in `reverse_proxy`'s `handle_response` routes, which can be used to actually copy the proxy response downstream.
Previously, if `handle_response` was used (with routes, not the status code mode), it was impossible to use the upstream's response body at all, because we would always close the body, expecting the routes to write a new body from scratch.
To implement this, I had to refactor `h.reverseProxy()` to move all the code that came after the `HandleResponse` loop into a new function. This new function `h.finalizeResponse()` takes care of preparing the response by removing extra headers, dealing with trailers, then copying the headers and body downstream.
Since basically what we want `copy_response` to do is invoke `h.finalizeResponse()` at a configurable point in time, we need to pass down the proxy handler, the response, and some other state via a new `req.WithContext(ctx)`. Wrapping a new context is pretty much the only way we have to jump a few layers in the HTTP middleware chain and let a handler pick up this information. Feels a bit dirty, but it works.
Also fixed a bug with the `http.reverse_proxy.upstream.duration` placeholder, it always had the same duration as `http.reverse_proxy.upstream.latency`, but the former was meant to be the time taken for the roundtrip _plus_ copying/writing the response.
* Delete the "Content-Length" header if we aren't copying
Fixes a bug where the Content-Length will mismatch the actual bytes written if we skipped copying the response, so we get a message like this when using curl:
```
curl: (18) transfer closed with 18 bytes remaining to read
```
To replicate:
```
{
admin off
debug
}
:8881 {
reverse_proxy 127.0.0.1:8882 {
@200 status 200
handle_response @200 {
header Foo bar
}
}
}
:8882 {
header Content-Type application/json
respond `{"hello": "world"}` 200
}
```
* Implement `copy_response_headers`, with include/exclude list support
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* opentelemetry: create a new module
* fix imports
* fix test
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddyhttp/opentelemetry/tracer.go
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* rename error ErrUnsupportedTracesProtocol
* replace spaces with tabs in the test data
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* replace spaces with tabs in the README.md
* use default values for a propagation and exporter protocol
* set http attributes with helper
* simplify code
* Cleanup modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update link in README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update documentation in README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update link to naming spec in README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Rename module from opentelemetry to tracing
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Rename span_name to span
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Rename span_name to span
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Simplify otel resource creation
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* handle extra attributes
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* update go.opentelemetry.io/otel/semconv to 1.7.0
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* update go.opentelemetry.io/otel version
* remove environment variable handling
* always use tracecontext,baggage as propagators
* extract tracer name into variable
* rename OpenTelemetry to Tracing
* simplify resource creation
* update go.mod
* rename package from opentelemetry to tracing
* cleanup tests
* update Caddyfile example in README.md
* update README.md
* fix test
* fix module name in README.md
* fix module name in README.md
* change names in README.md and tests
* order imports
* remove redundant tests
* Update documentation README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Fix grammar
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update comments
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update comments
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* update go.sum
* update go.sum
* Add otelhttp instrumentation, update OpenTelemetry libraries.
* Use otelhttp instrumentation for instrumenting HTTP requests.
This change uses context.WithValue to inject the next handler into the
request context via a "nextCall" carrier struct, and pass it on to a
standard Go HTTP handler returned by otelhttp.NewHandler. The
underlying handler will extract the next handler from the context,
call it and pass the returned error to the carrier struct.
* use zap.Error() for the error log
* remove README.md
* update dependencies
* clean up the code
* change comment
* move serveHTTP method from separate file
* add syntax to the UnmarshalCaddyfile comment
* go import the file
* admin: Write proper status on invalid requests (#4569) (fix#4561)
* update dependencies
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Vibhav Pant <vibhavp@gmail.com>
Co-authored-by: Alok Naushad <alokme123@gmail.com>
Co-authored-by: Cedric Ziel <cedric@cedric-ziel.com>
* reverseproxy: Begin refactor to enable dynamic upstreams
Streamed here: https://www.youtube.com/watch?v=hj7yzXb11jU
* Implement SRV and A/AAA upstream sources
Also get upstreams at every retry loop iteration instead of just once
before the loop. See #4442.
* Minor tweaks from review
* Limit size of upstreams caches
* Add doc notes deprecating LookupSRV
* Provision dynamic upstreams
Still WIP, preparing to preserve health checker functionality
* Rejigger health checks
Move active health check results into handler-specific Upstreams.
Improve documentation regarding health checks and upstreams.
* Deprecation notice
* Add Caddyfile support, use `caddy.Duration`
* Interface guards
* Implement custom resolvers, add resolvers to http transport Caddyfile
* SRV: fix Caddyfile `name` inline arg, remove proto condition
* Use pointer receiver
* Add debug logs
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
The `net.JoinHostPort()` function has some naiive logic for handling IPv6, it just checks if the host part has a `:` and if so it wraps the host part with `[ ]` but this causes our network type prefix to get wrapped as well, which is invalid for `caddy.NetworkAddress`. Instead, we can just concatenate the host and port manually here to avoid this side-effect.
* caddyhttp: Enhance vars matcher
Enable "or" logic for multiple values.
Fall back to checking placeholders if not a var name.
* Fix tests (thanks @mohammed90 !)
* reverseproxy: Adjust defaults, document defaults
Related to some of the issues in https://github.com/caddyserver/caddy/issues/4245, a complaint about the proxy transport defaults not being properly documented in https://caddy.community/t/default-values-for-directives/14254/6.
- Dug into the stdlib to find the actual defaults for some of the timeouts and buffer limits, documenting them in godoc so the JSON docs get them next release.
- Moved the keep-alive and dial-timeout defaults from `reverseproxy.go` to `httptransport.go`. It doesn't make sense to set defaults in the proxy, because then any time the transport is configured with non-defaults, the keep-alive and dial-timeout defaults are lost!
- Sped up the dial timeout from 10s to 3s, in practice it rarely makes sense to wait a whole 10s for dialing. A shorter timeout helps a lot with the load balancer retries, so using something lower helps with user experience.
* reverseproxy: Make keepalive interval configurable via Caddyfile
* fastcgi: DialTimeout default for fastcgi transport too
We realized we made some mistakes with the directive ordering, so we're making some minor adjustments.
`abort` and `error` don't really make sense to be after other handler directives, because you would expect to be able to "fail-fast" and throw an error before falling through to some `file_server` or `respond` typically. So we're moving them up to just before `respond`, i.e. before the common handler directives.
This is also more consistent with our existing examples in the docs, which actually didn't work due to the directive ordering. See https://caddyserver.com/docs/caddyfile/directives/error#examples
Also, `push` doesn't quite make sense to be after `handle`/`route`, since its job is to read from response headers to push additional resources if necessary, and `handle`/`route` may be terminal so push would not be reached if it was declared outside those. And also, it would make sense to be _before_ `templates` because a template _could_ add a `Link` header to the response dynamically.
* httpcaddyfile: ensure hosts to skip can always be collected
Previously, some hosts that should be skipped in logging would
be missed as the current logic would only collect them after
encountering the first server that would log. This change makes sure
the ServerLogConfig is initialized before iterating over the server
blocks.
* httpcaddyfile: add test case for skip hosts behavior
