caddy/caddytest/integration/caddyfile_adapt
Francis Lavoie c7d6c4cbb9
reverseproxy: copy_response and copy_response_headers for handle_response routes (#4391)
* reverseproxy: New `copy_response` handler for `handle_response` routes

Followup to #4298 and #4388.

This adds a new `copy_response` handler which may only be used in `reverse_proxy`'s `handle_response` routes, which can be used to actually copy the proxy response downstream. 

Previously, if `handle_response` was used (with routes, not the status code mode), it was impossible to use the upstream's response body at all, because we would always close the body, expecting the routes to write a new body from scratch.

To implement this, I had to refactor `h.reverseProxy()` to move all the code that came after the `HandleResponse` loop into a new function. This new function `h.finalizeResponse()` takes care of preparing the response by removing extra headers, dealing with trailers, then copying the headers and body downstream.

Since basically what we want `copy_response` to do is invoke `h.finalizeResponse()` at a configurable point in time, we need to pass down the proxy handler, the response, and some other state via a new `req.WithContext(ctx)`. Wrapping a new context is pretty much the only way we have to jump a few layers in the HTTP middleware chain and let a handler pick up this information. Feels a bit dirty, but it works.

Also fixed a bug with the `http.reverse_proxy.upstream.duration` placeholder, it always had the same duration as `http.reverse_proxy.upstream.latency`, but the former was meant to be the time taken for the roundtrip _plus_ copying/writing the response.

* Delete the "Content-Length" header if we aren't copying

Fixes a bug where the Content-Length will mismatch the actual bytes written if we skipped copying the response, so we get a message like this when using curl:

```
curl: (18) transfer closed with 18 bytes remaining to read
```

To replicate:

```
{
	admin off
	debug
}

:8881 {
	reverse_proxy 127.0.0.1:8882 {
		@200 status 200
		handle_response @200 {
			header Foo bar
		}
	}
}

:8882 {
	header Content-Type application/json
	respond `{"hello": "world"}` 200
}
```

* Implement `copy_response_headers`, with include/exclude list support

* Apply suggestions from code review

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2022-03-09 11:00:51 -07:00
..
auto_https_disable_redirects.txt
auto_https_ignore_loaded_certs.txt httpcaddyfile: Add `auto_https ignore_loaded_certs` (#4077) 2021-05-02 12:11:27 -06:00
auto_https_off.txt
bind_ipv6.txt httpcaddyfile: Fix incorrect handling of IPv6 bind addresses (#4532) 2022-01-18 11:27:43 -07:00
encode_options.txt encode: Drop `prefer` from Caddyfile (#4156) 2021-05-10 11:12:59 -06:00
error_example.txt httpcaddyfile: Reorder some directives (#4311) 2021-08-26 14:31:55 -06:00
file_server_disable_canonical_uris.txt fileserver: Add `disable_canonical_uris` Caddyfile subdirective (#4222) 2021-07-01 17:22:16 -06:00
file_server_pass_thru.txt fileserver: Add `pass_thru` Caddyfile option (#4613) 2022-03-04 20:50:05 -07:00
file_server_precompressed.txt encode,staticfiles: Content negotiation, precompressed files (#4045) 2021-03-29 18:47:19 -06:00
file_server_status.txt fileserver: Add status code override (#4076) 2021-04-08 11:09:12 -06:00
global_options.txt httpcaddyfile: Disabling OCSP stapling for both managed and unmanaged (#4589) 2022-02-19 14:20:38 -07:00
global_options_acme.txt caddyfile: make renew_interval option configurable (#4451) 2021-11-28 17:22:26 -05:00
global_options_admin.txt
global_options_default_bind.txt httpcaddyfile: Add `default_bind` global option (#4531) 2022-01-18 11:29:07 -07:00
global_options_log_and_site.txt caddyconfig: add global option for configuring loggers (#4028) 2021-03-12 13:00:02 -07:00
global_options_log_basic.txt caddyconfig: add global option for configuring loggers (#4028) 2021-03-12 13:00:02 -07:00
global_options_log_custom.txt caddyhttp: Split up logged remote address into IP and port (#4403) 2021-11-29 01:18:35 -05:00
global_options_log_multi.txt caddyconfig: add global option for configuring loggers (#4028) 2021-03-12 13:00:02 -07:00
global_options_preferred_chains.txt httpcaddyfile: Don't put localhost in public APs (fix #4220) 2021-06-25 11:28:32 -06:00
global_options_skip_install_trust.txt httpcaddyfile: Add pki app `root` and `intermediate` cert/key config (#4514) 2022-01-18 12:18:31 -07:00
global_server_options_multi.txt httpcaddyfile: Support explicitly turning off `strict_sni_host` (#4592) 2022-03-01 20:02:39 -05:00
global_server_options_single.txt caddyhttp: Move HTTP redirect listener to an optional module (#4585) 2022-02-19 15:36:36 -07:00
handle_path.txt
handle_path_sorting.txt
header.txt headers: Fix `+` in Caddyfile to properly append rather than set (#4506) 2022-01-04 10:10:11 -07:00
http_only_hostnames.txt
http_only_on_any_address.txt
http_only_on_domain.txt
http_only_on_hostless_block.txt httpcaddyfile: Take into account host scheme/port (fix #4113) 2021-04-16 11:17:22 -06:00
http_only_on_localhost.txt
http_only_on_non_standard_port.txt
https_on_domain.txt
import_args_file.txt
import_args_snippet.txt
import_args_snippet_env_placeholder.txt caddyfile: Fix `import` replacing unrelated placeholders (#4129) 2021-04-22 18:29:04 -06:00
log_except_catchall_blocks.txt
log_filters.txt logging: add support for hashing data (#4434) 2021-12-02 13:51:37 -07:00
log_roll_days.txt logging: Add `roll_local_time` Caddyfile option (#4583) 2022-02-19 15:12:28 -07:00
log_skip_hosts.txt httpcaddyfile: Ensure hosts to skip for logs can always be collected (#4258) 2021-08-02 14:15:27 -06:00
matcher_syntax.txt caddyhttp: Enhance vars matcher (#4433) 2021-12-13 13:59:58 -07:00
matchers_in_route.txt
method_directive.txt rewrite: Add `method` Caddyfile directive (#4528) 2022-01-18 12:17:35 -07:00
metrics_disable_om.txt
metrics_syntax.txt
not_block_merging.txt
php_fastcgi_expanded_form.txt
php_fastcgi_handle_response.txt fastcgi: Fix Caddyfile parsing when `handle_response` is used (#4342) 2021-09-11 14:12:21 -06:00
php_fastcgi_index_off.txt
php_fastcgi_matcher.txt
php_fastcgi_subdirectives.txt
php_fastcgi_try_files_override.txt fastcgi: Implement `try_files` override in Caddyfile directive (#4347) 2021-09-17 08:23:06 -06:00
portless_upstream.txt
request_body.txt
request_header.txt headers: Fix Caddyfile parsing for `request_header` with matchers (#4085) 2021-03-29 10:55:29 -06:00
reverse_proxy_dynamic_upstreams.txt reverseproxy: Dynamic upstreams (with SRV and A/AAAA support) (#4470) 2022-03-06 17:43:39 -07:00
reverse_proxy_empty_non_http_transport.txt
reverse_proxy_h2c_shorthand.txt
reverse_proxy_handle_response.txt reverseproxy: copy_response and copy_response_headers for handle_response routes (#4391) 2022-03-09 11:00:51 -07:00
reverse_proxy_health_headers.txt reverseproxy: Fix incorrect `health_headers` Caddyfile parsing (#4485) 2021-12-17 08:53:11 -07:00
reverse_proxy_health_path_query.txt reverseproxy: Implement health_uri, deprecate health_path, supports query (#4050) 2021-03-29 18:36:40 -06:00
reverse_proxy_options.txt reverseproxy: Dynamic upstreams (with SRV and A/AAAA support) (#4470) 2022-03-06 17:43:39 -07:00
reverse_proxy_trusted_proxies.txt reverseproxy: Implement trusted proxies for `X-Forwarded-*` headers (#4507) 2022-03-06 18:51:55 -05:00
reverse_proxy_upstream_placeholder.txt reverseproxy: Fix upstreams with placeholders with no port (#4046) 2021-03-03 10:12:31 -07:00
shorthand_parameterized_placeholders.txt
site_block_sorting.txt
sort_directives_with_any_matcher_first.txt
sort_directives_within_handle.txt httpcaddyfile: Fix sorting edgecase for nested `handle_path` (#4477) 2021-12-13 13:42:08 -05:00
tls_acme_preferred_chains.txt httpcaddyfile: Add `preferred_chains` global option and issuer subdirective (#4192) 2021-06-08 14:10:37 -06:00
tls_automation_policies_1.txt
tls_automation_policies_2.txt
tls_automation_policies_3.txt
tls_automation_policies_4.txt httpcaddyfile: Fix automation policy consolidation again (fix #4161) 2021-05-11 15:26:07 -06:00
tls_automation_policies_5.txt httpcaddyfile: Fix unexpectedly removed policy (#4128) 2021-04-29 10:56:01 -06:00
tls_automation_policies_6.txt httpcaddyfile: Fix automation policy consolidation again (fix #4161) 2021-05-11 15:26:07 -06:00
tls_automation_policies_7.txt httpcaddyfile: Don't add HTTP hosts to TLS APs (fix #4176 and fix #4198) 2021-06-09 14:35:09 -06:00
tls_automation_policies_8.txt httpcaddyfile: Empty tls policy for internal http localhost (#4398) 2021-10-26 13:54:19 -06:00
tls_automation_policies_global_email_localhost.txt httpcaddyfile: Don't put localhost in public APs (fix #4220) 2021-06-25 11:28:32 -06:00
tls_client_auth_cert_file.txt
tls_client_auth_inline_cert.txt
tls_conn_policy_consolidate.txt
tls_internal_options.txt caddytls: Add internal Caddyfile `lifetime`, `sign_with_root` opts (#4513) 2022-01-18 12:19:50 -07:00
tls_propagation_timeout.txt caddytls: Add Caddyfile support for `propagation_timeout` (#4178) 2021-06-07 12:25:12 -06:00
tracing.txt tracing: New OpenTelemetry module (#4361) 2022-03-08 12:18:32 -07:00