* perf: bypass header key normalization for canonical literals
http.Header's Get, Set, Add and Del run textproto.CanonicalMIMEHeaderKey
on every call. When the key is a canonical literal the result is the key
itself, so the scan is pure overhead. Access the map directly instead.
Only hot, allocation-free paths are converted, and only where the key is
a canonical literal. Call sites taking a key from config, a template or
the request itself still go through the normalizing methods, since they
need it for correctness.
allHeaderValues and lastHeaderValue dropped their http.CanonicalHeaderKey
call for the same reason: all callers pass canonical literals.
benchstat, Apple M5 Pro, go1.27.1:
PrepareRequest-15 659.4n ± 2% -> 457.7n ± 2% -30.58% (p=0.000 n=12)
AcceptedEncodings-15 330.8n ± 5% -> 309.4n ± 7% -6.44% (p=0.019 n=10)
ResponseWriterWriteHeader/event-stream-15 1007.5n ± 3% -> 732.5n ± 1% -27.30%
ResponseWriterWriteHeader/informational-15 60.0n ± 3% -> 48.0n ± 2% -20.01%
ResponseWriterWriteHeader/not-modified-15 214.3n ± 6% -> 175.3n ± 4% -18.20%
ResponseWriterWriteHeader/connect-15 60.2n ± 3% -> 51.9n ± 4% -13.85%
ResponseWriterWriteHeader/html-15 206.5n ± 4% -> 197.9n ± 2% -4.19%
geomean -14.38%
Allocations are unchanged.
* tests: verify http.Header map keys are canonical
Indexing http.Header directly skips the canonicalization Get/Set/Add/Del
perform, which also puts the key outside what CI checks. canonicalheader
only inspects method calls. staticcheck's SA1008 covers reads, but stops
descending as soon as an assignment writes to a header, so it misses the
write, the delete builtin and the read-modify-write append. A key that is
not canonical in those positions is a silent lookup miss, not a compile
error.
Walk the loaded packages and verify every string constant used as a key in
an http.Header map operation matches http.CanonicalHeaderKey. A helper that
takes the key from its caller would hide the literal again, so a parameter
named with a canonical prefix opts its arguments into the same check.
The WebSocket headers Caddy deliberately writes with RFC 6455 casing are
allowed explicitly; they are only ever written out, never looked up against
a parsed header map.
Covers 64 operations, runs in under a second, and promotes golang.org/x/tools
from an indirect to a direct dependency at the version already in the module
graph.
* headercheck: resolve aliases of http.Header
* headercheck: check http.Header composite literal keys
* headercheck: note overlap with canonicalheader
* headercheck: drop parameter-name check
Helpers now take the indexed values, so every key sits in an index expression the analyzer and the linter already check.
* headercheck: scope RFC 6455 exceptions to their call sites
A key-based allow-list also accepted reads such as h["Sec-WebSocket-Key"], which miss because parsed headers are canonical. Honor //nolint:canonicalheader on the line instead, which the linter will use too.
* encode: read every Accept-Encoding field line
AcceptedEncodings read only the first field line, so a client that split
the list across several lines had the rest ignored. RFC 9110 section 5.3
treats repeated list-field lines as one comma-joined list. Iterate over
the items of every line instead of joining them, and skip empty items,
which used to yield an empty encoding. file_server's precompressed
selection uses the same function and is fixed with it.
Fixes https://github.com/caddyserver/caddy/issues/8272
Co-authored-by: Kağan Küçük <kaankck@gmail.com>
* encode: read every Cache-Control and Content-Encoding field line
Only the first field line was read, but RFC 9110 section 5.3 makes
multiple lines of a list-based field equivalent to one comma-joined
value:
- Cache-Control: no-transform on a later line was ignored, so the
request or response was compressed anyway.
- Content-Encoding: an empty first line followed by a real coding let
the handler encode the body a second time.
* encode: parse Cache-Control no-transform per RFC 9111
The substring match missed No-Transform, though RFC 9111 section 5.2 compares directive names case-insensitively, and fired on text that is not the directive: longer tokens such as x-no-transform-ext, quoted arguments such as community="no-transform", and semicolon-separated values, which are not valid Cache-Control syntax. Split the field on commas outside quoted strings and match only a well-formed no-transform element.
* encode: keep accepted encoding preferences on the stack
AcceptedEncodings counted commas to size the preference slice, and the variable capacity put it on the heap. A constant capacity covers typical Accept-Encoding headers and lets the compiler keep the slice on the stack; append still moves it to the heap for longer ones. BenchmarkAcceptedEncodings: 151.8ns -> 120.5ns (-21%), 192 -> 64 B/op, 2 -> 1 allocs/op (n=8).
* encode: build the encoded ETag without fmt
fmt.Sprintf boxed both arguments and parsed the format on every encoded response with a strong ETag. Concatenation: 47ns -> 16.5ns, 56 -> 24 B, 3 -> 1 allocs.
* reverseproxy: access response Server and Via headers directly
finalizeResponse runs on every proxied response; skip the key normalization as prepareRequest already does on the request side.
* encode: access headers directly in Write, WriteHeader and Close
These run once per response; convert the remaining literal-key Get, Set and Add calls like the rest of the handler.
* encode: match Cache-Control no-transform with httpguts
httpguts.HeaderValuesContainsToken already compares list elements case-insensitively across field lines, so the hand-written list parser is unnecessary. A no-transform inside a comma-separated quoted argument now also disables encoding, which only errs toward not transforming.
* encode: inline the no-transform check
isEncodeAllowed only wrapped one httpguts call. Its unit test now drives the response directive through ServeHTTP instead.
---------
Co-authored-by: Kağan Küçük <kaankck@gmail.com>
* encode: Don't write an empty encoded stream for HEAD responses
When nothing was written (e.g. a HEAD request), Close() still closed the
encoder, which wrote the empty stream trailer to the client. The server
then set Content-Length to that size (20 for gzip, 13 for zstd) instead
of omitting it as for the matching GET response.
* encode: Check resp.ContentLength in the HEAD test
* encode: Track body writes to decide whether to close the encoder
* encode: flush headers immediately for server-sent events responses
The encode middleware withholds the response header until the first body
write so it can sniff content-type and apply the minimum_length threshold.
For a text/event-stream response the upstream typically writes headers and
flushes to establish the event stream before any event body is available,
so the client never received the headers and the stream stalled; the same
buffering also delayed individual events.
When WriteHeader sees a text/event-stream content type, initialize encoding
and write the header through immediately. Forcing the header out also marks
the response as started, so subsequent event writes bypass the minimum_length
buffering and stream to the client as they arrive.
Fixes#6293
* encode: add WriteHeader benchmark covering SSE fast path
* encode: replace mime.ParseMediaType with bound-checked SSE check
WriteHeader runs an SSE Content-Type check on every call once headers
haven't been written yet. mime.ParseMediaType parses the full media
type, including parameters, even when nothing matches, which shows up
on the hot header-write path.
Replace it with a bound-checked manual prefix/boundary check (isSSE),
skipping parameter parsing for the common non-SSE case.
* encode: reject content types with junk after text/event-stream
isSSE accepted any suffix after a space, so a value like
"text/event-stream nonsense" was treated as an SSE response. After the
media type, skip optional whitespace and require either the end of the
value or a parameter separator. The check remains allocation-free, so
the hot-path motivation for the manual matcher is preserved.
---------
Co-authored-by: SillyZir <269283839+SillyZir@users.noreply.github.com>
Co-authored-by: Kévin Dunglas <kevin@les-tilleuls.coop>
AcceptedEncodings runs for every request when response compression is enabled,
so its allocations contribute directly to per-request garbage and GC pressure.
While the gains are modest, this cuts allocations by two-thirds and bytes
nearly in half on a hot path that executes on every compressed request. Fewer
allocations means less work for the garbage collector, which at high request
rates translates into lower GC frequency and steadier tail latency, not just
faster execution of this one function in isolation.
For each token in the Accept-Encoding header it called strings.Split(accepted,
";"), allocating a throwaway slice per encoding, and grew the prefs slice from
empty. This commit replaces the per-token Split with strings.Cut (zero
allocation) and presize prefs from the comma count.
As requested per the policy, this commit adds a benchmark to exercise the
change. With header "gzip, deflate, br;q=0.9, zstd;q=0.8":
AcceptedEncodings
- sec/op 1158.5n -> 771.8n (-33.38%)
- B/op 408 -> 216 (-47.06%)
- allocs/op 9 -> 3 (-66.67%)
Co-authored-by: @dunglas
Header.Get/Set re-canonicalize and allocate whenever the passed key
isn't already in canonical MIME header form. Sec-WebSocket-Key,
WWW-Authenticate, and content-type all miss the fast path; switch to
their canonical forms (Sec-Websocket-Key, Www-Authenticate,
Content-Type).
Enable the canonicalheader linter to catch future regressions,
excluded in test files since those already use non-canonical casing
in several places without a perf-sensitive path behind them.
* fix(encode): prioritize zstd and br over gzip in content negotiation
* test(encode): update unit tests to reflect new default priority ties
* fix(encode): move default preferences to dynamic encode handler and restore generic negotiation helper
* test(encode): call real Provision function in served-response test
* test(encode): rename served-response test to TestServeHTTPDefaultEncodingPreference
* refactor(encode): use slices.SortStableFunc and httptest.NewRecorder as recommended
* refactor(encode): simplify sorting with cmp.Compare and check request error in test
* test(encode): fix variable redeclaration in TestServeHTTPDefaultEncodingPreference
Fix 'no new variables on left side of :=' error by changing 'err :=' to 'err ='
on line 347, since err was already declared on line 332.
This fixes the build failure in the encode module tests.
* encode: write status immediate for success response for CONNECT requests
* fix compile
* fix test
* fix lint
* treat first write and flush for encode response writer to CONNECT request as success if status is not set explicitly
* caddyhttp: Make use of http.ResponseController
Also syncs the reverseproxy implementation with stdlib's which now uses ResponseController as well 2449bbb5e6
* Enable full-duplex for HTTP/1.1
* Appease linter
* Add warning for builds with Go 1.20, so it's less surprising to users
* Improved godoc for EnableFullDuplex, copied text from stdlib
* Only wrap in encode if not already wrapped
* fix encode handler header manipulation
also avoid implementing ReadFrom because it breaks when io.Copied to directly
* strconv.Itoa should be tried as a last resort
WriteHeader during Close
* encode: ignore flushing until after first write (fix#4314)
The first write will determine if encoding has to be done and will add an Content-Encoding. Until then Flushing has to be delayed so the Content-Encoding header can be added before headers and status code is written. (A passthrough flush would write header and status code)
* Update modules/caddyhttp/encode/encode.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* encode: implement prefer setting
* encode: minimum_length configurable via caddyfile
* encode: configurable content-types which to encode
* file_server: support precompressed files
* encode: use ReponseMatcher for conditional encoding of content
* linting error & documentation of encode.PrecompressedOrder
* encode: allow just one response matcher
also change the namespace of the encoders back, I accidently changed to precompressed >.>
default matchers include a * to match to any charset, that may be appended
* rounding of the PR
* added integration tests for new caddyfile directives
* improved various doc strings (punctuation and typos)
* added json tag for file_server precompress order and encode matcher
* file_server: add vary header, remove accept-ranges when serving precompressed files
* encode: move Suffix implementation to precompressed modules
* ci: Use golangci's github action for linting
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix most of the staticcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the prealloc lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the misspell lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the varcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the errcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the bodyclose lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the deadcode lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the unused lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the gosec lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the gosimple lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the ineffassign lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the staticcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Revert the misspell change, use a neutral English
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Remove broken golangci-lint CI job
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Re-add errantly-removed weakrand initialization
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* don't break the loop and return
* Removing extra handling for null rootKey
* unignore RegisterModule/RegisterAdapter
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* single-line log message
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Fix lint after a1808b0dbf209c615e438a496d257ce5e3acdce2 was merged
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Revert ticker change, ignore it instead
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Ignore some of the write errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Remove blank line
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Use lifetime
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* close immediately
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Preallocate configVals
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddytls/distributedstek/distributedstek.go
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
This commit goes a long way toward making automated documentation of
Caddy config and Caddy modules possible. It's a broad, sweeping change,
but mostly internal. It allows us to automatically generate docs for all
Caddy modules (including future third-party ones) and make them viewable
on a web page; it also doubles as godoc comments.
As such, this commit makes significant progress in migrating the docs
from our temporary wiki page toward our new website which is still under
construction.
With this change, all host modules will use ctx.LoadModule() and pass in
both the struct pointer and the field name as a string. This allows the
reflect package to read the struct tag from that field so that it can
get the necessary information like the module namespace and the inline
key.
This has the nice side-effect of unifying the code and documentation. It
also simplifies module loading, and handles several variations on field
types for raw module fields (i.e. variations on json.RawMessage, such as
arrays and maps).
I also renamed ModuleInfo.Name -> ModuleInfo.ID, to make it clear that
the ID is the "full name" which includes both the module namespace and
the name. This clarity is helpful when describing module hierarchy.
As of this change, Caddy modules are no longer an experimental design.
I think the architecture is good enough to go forward.
Kévin Dunglas
Jay Hemnani
Hitanshu
SillyZir
Julien Voisin
Sam Ottenhoff
Muhammad Syafri, S.Kom
Zen Dodd
Matthew Holt
ledigang
wyrapeseed
Siomachkin
aro-lew
joemicky
WeidiDeng
Francis Lavoie
Yolan Romailler
Christoph
darkweak
Steffen Brüheim
Dave Henderson
Success Go