If use gzip and templates at the same time, the response body will
be gzipped data. And in this case, the Content-Type header won't be
set by Caddy code. Then Go http package will set "Content-Type" to
wrong value "application/x-gzip" which is determined by response body.
So the header Contenty-Type should be set in templates middleware.
This commit removes _almost_ all instances of hard-coded ports 80 and
443 strings, and now allows the user to define what the HTTP and HTTPS
ports are by the -http-port and -https-ports flags.
(One instance of "80" is still hard-coded in tls.go because it cannot
import httpserver to get access to the HTTP port variable. I don't
suspect this will be a problem in practice, but one workaround would be
to define an exported variable in the caddytls package and let the
httpserver package set it as well as its own HTTPPort variable.)
The port numbers required by the ACME challenges HTTP-01 and TLS-SNI-01
are hard-coded into the spec as ports 80 and 443 for good reasons,
but the big question is whether they necessarily need to be the HTTP
and HTTPS ports. Although the answer is probably no, they chose those
ports for convenience and widest compatibility/deployability. So this
commit also assumes that the "HTTP port" is necessarily the same port
on which to serve the HTTP-01 challenge, and the "HTTPS port" is
necessarily the same one on which to serve the TLS-SNI-01 challenge. In
other words, changing the HTTP and HTTPS ports also changes the ports
the challenges will be served on.
If you change the HTTP and HTTPS ports, you are responsible for
configuring your system to forward ports 80 and 443 properly.
Closes#918 and closes#1293. Also related: #468.
* Use RequestURI when redirecting to canonical path.
Caddy may trim a request's URL path when it starts with the path that's
associated with the virtual host. This change uses the path from the request's
RequestURI when performing a redirect.
Fix issue #1327.
* Rename redirurl to redirURL.
* Redirect to the full URL.
The scheme and host from the virtual host's site configuration is used
in order to redirect to the full URL.
* Add comment and remove redundant check.
* Store the original URL path in request context.
By storing the original URL path as a value in the request context,
middlewares can access both it and the sanitized path. The default
default FileServer handler will use the original URL on redirects.
* Replace contextKey type with CtxKey.
In addition to moving the CtxKey definition to the caddy package, this
change updates the CtxKey references in the httpserver, fastcgi, and
basicauth packages.
* httpserver: Fix reference to CtxKey
Timeouts are important for mitigating slowloris, yes. But after a number
of complaints and seeing that default timeouts are a sore point of
confusion, we're disabling them now. However, the code that sets
default timeouts remains intact; the defaults are just the zero value.
While Caddy aims to be secure by default, Caddy also aims to serve a
worldwide audience. Even my own internet here in Utah is poor at times,
with bad WiFi signal, causing some connections to take over 10s to
be established. Many use the Internet while commuting on slower
connection speeds. Latency across country borders is another concern.
As such, disabling default timeouts will serve a greater population of
users than enabling them, as slowloris is easy to mitigate and does
not seem to be reported often (I've only seen it once). It's also very
difficult sometimes to distinguish slowloris from genuine slow networks.
That decision is best left to the site owner for now.
* tls: Refactor TLS config innards with a few minor syntax changes
muststaple -> must_staple
"http2 off" -> "alpn" with list of ALPN values
* Fix typo
* Fix QUIC handler
* Inline struct field assignments
* Store name of authenticated user in basicauth for use by upstream middleware such as fastcgi and cgi.
* Use request context to transfer name of authorized user from basicauth to upstream middleware. Test retrieval of name from context.
* Remove development code that was inadvertently left in place
* Use keys of type httpserver.CtxKey to access Context values
* WIP: Implement HTTPS interception detection by Durumeric, et. al.
Special thanks to @FiloSottile for guidance with the custom listener.
* Add {{.IsMITM}} context action and {mitm} placeholder
* Improve MITM detection heuristics for Firefox and Edge
* Add tests for MITM detection heuristics
* Improve Safari heuristics for interception detection
* Read ClientHello during first Read() instead of during Accept()
As far as I can tell, reading the ClientHello during Accept() prevents
new connections from being accepted during the read. Since Read() should
be called in its own goroutine, this keeps Accept() non-blocking.
* Clean up MITM detection handler; make possible to close connection
* Use standard lib cipher suite values when possible
* Improve Edge heuristics and test cases
* Refactor MITM checking logic; add some debug statements for now
* Fix bug in MITM heuristic tests and actual heuristic code
* Fix gofmt
* Remove debug statements; preparing for merge
* fixing panic when root is symlink
checking root path is a symlink before os.Stat which panics
* fixing formatting
* adding test to verify symlink root path check
* fixing typo
* Add request placeholder support for querying request cookies.
This adds the ability to query the request cookies for placeholders
using the syntax "@cookiename".
For example, this would allow rewriting based on a cookie:
rewrite {
if @version is 'dev'
to /dev/index.html
}
* Switch cookie special char from @ to :
* Switch special char for cookies from : to ~
* Fix data race for max connection limiting in proxy directive.
The Conns and Unhealthy fields are updated concurrently across all active
requests. Because of this, they must use atomic operations for reads and
writes.
Prior to this change, Conns was incremented atomically, but read unsafely.
Unhealthly was updated & read unsafely. The new test
TestReverseProxyMaxConnLimit exposes this race when run with -race.
Switching to atomic operations makes the race detector happy.
* oops, remove leftover dead code.
* Use new subdirectives and flatten rolling config
* Set default rotate config
* Set default rolling config (hopefully) errwhere
* Make private
* Flatten errors directive and remove c.IncrNest()
* Don't skip first error log roller subdirective we see
* Remove hadBlock
* Try lumberjack import
* Unname import
The standard lib pprof library doesn't set its own Content-Type header
properly. If pprof is used with gzip, the index endpoint will be
interpreted as a .gz file; so we force its hand and set the header.
* proxy: use a new context for the outgoing request
fix issue #1345
Signed-off-by: Tw <tw19881113@gmail.com>
* proxy: add test for canceling the request
Signed-off-by: Tw <tw19881113@gmail.com>
* Generate meta elements from useful front matters.
Limited to the default template and specific elements.
* Rerun gofmt
* Add "keywords" and remove "language" to/from the list of meta tags.
* Add a simple positive list test for the meta tag generation.
* Move the meta tag list to a var at the begin of the file.
Seperate the Meta tags from the other front matters:
- Don't override user settings with name `meta`
- Cleaner Code.
* Remove the uneccessary `[:]` in the []Bytes to String casting.
@mholt was right ;)
* One minor refinement. Combining two statements.
Because of this commit(6e36811c37399d60cbce587b7c48e611009c5aec) on go tip,
it will probe the request's body to determine whether to use chunked transfer
encoding which trailers depend on it.
So we just offer a non empty body to make trailers work.
fix issue #1359
Signed-off-by: Tw <tw19881113@gmail.com>
* Add {whenISO} to record timestamp in ISO 8601 format in UTC.
ISO 8601 is the standard time format and is easy to parse.
This change assumes users desiring ISO 8016 generally prefer UTC for simplicity.
This results in {whenISO} to be significantly shorter than {when}:
{when} = "02/Jan/2006:15:04:05 +0000"
{whenISO} = "2006-01-02T15:04:12Z"
Add unit test to verify both, as there was no unit test for {when}.
* Rename {whenISO} to {when_iso}
If only one upstream is defined we don't need to buffer the body.
Instead we directly stream the body to the upstream host,
which reduces memory usage as well as latency.
Furthermore this enables different kinds of HTTP streaming
applications like gRPC for instance.
If a site owner protects a path with basicauth, no need
to use the Authorization header elsewhere upstream, especially since it
contains credentials.
If this breaks anyone, it means they're double-dipping. It's usually
good practice to clear out credentials as soon as they're not needed
anymore. (Note that we only clear credentials after they're used,
they stay for any other reason.)
* Added path cleanup functions with masking to preserve certain patterns + unit tests, #1298
* Use custom PathClean function instead of path.Clean to apply masks to preserve protocol separator in the path
* Indentation corrected in the test data map to pass the lint
* Fixing ineffassign of a temporary string variable
* Improved variable naming and documentation
* Improved variable naming
* Added benchmarks and improved variable naming in tests
* Removed unnecessary value capture when iterating over a map for keys
* A typo correction
This issue was caused by connHijackerTransport trying to record HTTP
response headers by "hijacking" the Read() method of the plain net.Conn.
This does not simply work over TLS though since this will record the TLS
handshake and encrypted data instead of the actual content.
This commit fixes the problem by providing an alternative transport.DialTLS
which correctly hijacks the overlying tls.Conn instead.
* Fixed HTTP/2 support for the proxy middleware
http.Transport instances whose TLSClientConfig, Dial, or DialTLS field
is non-nil will be configured without HTTP/2 support by default.
This commit adds the proper calls to http2.ConfigureTransport()
everywhere a http.Transport is created and thus fixes HTTP/2 in the
proxy middleware whenever insecure_skip_verify or keepalive is provided.
* Added HTTP/2 support check to TestReverseProxyInsecureSkipVerify
* Feature #1282 - Support pre-gzipped files
* Fix broken test cases
* Support brotli encoding as well
* Fix for #1276 - support integers and floats as metadata in markdown (#1278)
* Fix for #1276
* Use strconv.Format
* Use map[string]interface{} as variables
* One more file
* Always run all tests before commit
* Get rid of DocFlags
* Fix syntax in caddy.conf
* Update to Go 1.7.4
* Add send_timeout property to fastcgi directive.
* Convert rwc field on FCGIClient from type io.ReadWriteCloser to net.Conn.
* Return HTTP 504 to the client when a timeout occurs.
* In Handler.ServeHTTP(), close the connection before returning an HTTP
502/504.
* Refactor tests and add coverage.
* Return HTTP 504 when FastCGI connect times out.
* test: add unit test for #1283 (#1288)
* After review fixes
* Limit the number of restarts with systemd
* Prevent fd leak
* Prevent fd leak
* Refactor loops
* gofmt
* Convert rwc field on FCGIClient from type io.ReadWriteCloser to net.Conn.
* Return HTTP 504 to the client when a timeout occurs.
* In Handler.ServeHTTP(), close the connection before returning an HTTP
502/504.
* Refactor tests and add coverage.
* Fix for #1276
* Use strconv.Format
* Use map[string]interface{} as variables
* One more file
* Always run all tests before commit
* Get rid of DocFlags
By setting the read deadline in streamReader.Read(), the deadline was
extended by the read timeout on each subsequent call. To avoid this, the
deadline is set in FCGIClient.Request(), before the first read occurs.
See #1094.
* Filter empty headers
Some web servers (e.g. Jetty 9.3) don’t like HTTP headers with empty values. This commit filters header replacements with zero length.
* Extend tests to verify removal of empty headers
* Handle add-header case
* Change - Use short variable assignment
* Make fastcgi load balanceable too
* Address one more corner case - invalid configuration fastcgi /
* After review fixes
* Simplify conditions
* Error message
* New fastcgi syntax
* golint will be happy
* Change syntax
* Add Files action to template context. (#1198)
* Fixes to testFiles().
- Set os.ModePerm on directories created during test.
- Use filepath.Join() to create directory path.
- Use Fatalf, not Fatal.
* Make additional fixes to test cases.
* Fix test cases to use correct path format.
Dir.Open() in net/http requires '/'-separated paths while
filepath.Join() may produce paths with different separator.
* Remove directory created by test at end of loop.
* Close the FileSystem before returning.
* Initialize names slice to the number of entries.
Also, do not call os.RemoveAll() unless the path to the directory
is a valid one.
* Very simple fix for #1153
* Prevent Caddy-Rewrite-Original-URI being added as an HTTP ENV variable passed to FastCGI
part of fix for #1153
* Changes to Markdown to fix travis CI build.
#1955.2
* Revert "Changes to Markdown to fix travis CI build."
This reverts commit 4a01888839.
* fail fast and fmt changes
* Create test for existance of Caddy-Rewrite-Original-URI header value #1153
* updated test comment
* const moved outside function so available to tests
* Remove the eager check in the browse middleware, whether the root directory exists.
Caddy will start and throw a 404-error until the directory will be created.
* Add the complimentary test.
- Tests the startup of the browse middleware if the site root is inexistent and browse is pointing to the site root.
* Some minor stylistic tweaks.
* keep fastcgi connection open
* poor mans serialisation to make up for the lack of demuxing
* pointing includes to echse's repo
* Revert "pointing includes to echse's repo"
This reverts commit 281daad8d4.
* switch for persistent fcgi connections on/off added
* fixing ineffectual assignments
* camel case instead of _
* only activate persistent sockets on windows (and some naming conventions/cleanup)
* gitfm import sorting
* Revert "fixing ineffectual assignments"
This reverts commit 79760344e7.
# Conflicts:
# caddyhttp/staticfiles/fileserver.go
* added another mutex and deleting map entries. thx to mholts QA comments!
* thinking about it, this RW lock was not a good idea here
* thread safety
* I keep learning about mutexs in go
* some cosmetics
* adding persistant fastcgi connections switch to directive
* Support for configurable connection pool.
* ensure positive integer pool size config
* abisofts pool fix + nicer logging for the fastcgi_test
* abisoft wants to have dialer comparison in _test instead of next to struct
* Do not put dead connections back into pool
* Fix fastcgi header error
* Do not put dead connections back into pool
* some code style improvements from the discussion in https://github.com/mholt/caddy/pull/1134
* abisofts naming convention
* keep fastcgi connection open
* poor mans serialisation to make up for the lack of demuxing
* pointing includes to echse's repo
* Revert "pointing includes to echse's repo"
This reverts commit 281daad8d4.
* switch for persistent fcgi connections on/off added
* fixing ineffectual assignments
* camel case instead of _
* only activate persistent sockets on windows (and some naming conventions/cleanup)
* gitfm import sorting
* Revert "fixing ineffectual assignments"
This reverts commit 79760344e7.
# Conflicts:
# caddyhttp/staticfiles/fileserver.go
* added another mutex and deleting map entries. thx to mholts QA comments!
* thinking about it, this RW lock was not a good idea here
* thread safety
* I keep learning about mutexs in go
* some cosmetics
This function should not be used outside of development. It destroys the
absolute ordering and guarantees of correctness. Multiple uses of it
may work fine, but maybe not if they overlap, causing non-deterministic
builds which is bad. However, this can be convenient when developing
a plugin by calling it from an init() function, since you don't have
to modify the Caddy source code just to try your plugin.
Removing quic protocol headers from being persisted during proxy requests.
Not removing them could lead to the client attempting to connect to the wrong port.
This makes the quic headers consistent with other protocol headers.
* * Added support for environment variables to 'templates' module.
* Fixed flaw in test caused by environment variable ordering during testing on CI.
* Updated some local variables to camel-case.
* Reverted changes to replacer as environment variables are processed elsewhere.
* Removed PrintEnv functionality in favour of documenting using template range.
* Fix for stripping of 'Content-Disposition' and other headers from 'X-Accel-Redirect' redirect scripts.
* Added test case for header manipulation of redirect response.
First, great job on the 0.9 release! It seems caddy's path lead into a bright future. Thanks also for including the locale plugin.
Trying it, I've figured out, that there might be a problem with the order of the directives. In the typical use case, the result of the locale detection might be used in the `rewrite` and `log` plugin. If I'm not mistaken, it makes sense to put the `locale` directive before those.
* Allow for UDP servers
Extend the Server interface with ServePacket and ListenPacket - this is
in the same vein as the net package.
Plumb the packetconn through the start and restart phases.
Rename RestartPair to RestartTriple as it now also contains a Packet.
Not that these can now be nil, so we need to check for that when
restarting.
* Update the documentation
If we listen on 127.0.0.1:80 for `localhost` but :80 for everything else,
then a hostname in the hosts file that resolves to 127.0.0.1 will be
served on :80 (unless the bind directive is used) but the OS will use
the socket listening at 127.0.0.1:80, thus giving a "No such site" error
even though the site is there, but it's on the other listener at :80.
Two ways to fix this: 1) Leave as-is and require the user to set "bind
127.0.0.1" in their Caddyfile for all sites that are resolved in the
hosts file, or 2) Take out this special case and let localhost sites
listen on :80 (unless the user changes that with the bind directive, of
course). Having localhost bind to any interface is a little annoying
(unsettling?) but probably best in the long run.
https://forum.caddyserver.com/t/wildcard-virtual-domains-with-wildcard-roots/221/9?u=matt
OnStartup and OnShutdown callbacks now run as part of restarts, too.
The startup and shutdown directives only run their commands NOT as part
of restarts, as before. Some middleware that use OnStartup may need to
switch to OnFirstStartup and implement OnFinalShutdown to do any cleanup
as needed.
