Hi There,
We are using Jetty 9.4.14.v20181114.
Our app uses a servlet filter to get requests from Jetty and handles
everything else from there. As a result, we use Jetty as a Web server
with only HTTP 1.1 support (no WebSockets, H2, JSP, or anything else).
We don't have any Jetty config files; we configure it all on startup
of our app via the embedded API.
Now, we need to start proxying some HTTP requests. To this end, we are
trying to use Jetty's ProxyServlet. This is challenging because we
don't have any servlets in our app and the servlet request/responses
are pretty obscured from the programming model of our request
handlers. Despite this, we have manged to get the proxy running and
mostly working.
The issue that we're running into, however, is that proxied requests
do not contain all of the headers from the origin server. Some of
them, Content-Type, Content-Length, etc. are not present in the
response that the proxy provides to the HTTP client (curl, Postman,
etc.).
In our request handler, this code does the proxying:
_proxyServlet.service(servletRequest, servletResponse);
servletResponse.flushBuffer();
In this snippet, _proxyServlet is an anonymous subclass of
ProxyServlet.Transparent:
_proxyServlet = new ProxyServlet.Transparent()
{
protected String filterServerResponseHeader(HttpServletRequest clientRequest, Response serverResponse,
String headerName, String headerValue)
{
if (_filteredResponseHeaders.contains(StringUtils.lowerCase(headerName, Locale.US)))
{
return null;
}
return headerValue;
}
@Override
protected Set<String> findConnectionHeaders(HttpServletRequest clientRequest)
{
ImmutableSet.Builder<String> builder = ImmutableSet.builder();
@Nullable Set<String> connectionHeaders = super.findConnectionHeaders(clientRequest);
if (connectionHeaders != null)
{
builder.addAll(connectionHeaders.stream().filter(StringUtils::isNotEmpty).collect(Collectors.toSet()));
}
builder.addAll(_filteredRequestHeaders);
return builder.build();
}
};
_proxyServlet.init(new ReverseProxyServletConfig(prefix, proxyTo));
Those filter sets are just these:
Set<String> _filteredResponseHeaders = ImmutableSet.of("server", "date");
Set<String> _filteredRequestHeaders = ImmutableSet.of("cookie", "accept-encoding");
The config of the servlet has some issues ATM, but I don't think
that's the cause of the issue (might be though). Here's how it's being
configured when init is called on the proxy servlet:
private static class ReverseProxyServletConfig implements ServletConfig
{
private final Map<String, String> _config;
private final StaticContext _context;
ReverseProxyServletConfig(String prefix, String proxyTo)
{
URI proxyToUri = URI.create(proxyTo);
_config = ImmutableMap.of(
"proxyTo", proxyTo,
"whiteList", String.format("%s:%d", proxyToUri.getHost(), proxyToUri.getPort()),
"prefix", prefix,
"maxThreads", "16" // TODO: Remove so executor set below is used instead
);
_context = new StaticContext(); // Same as Jetty's except that getContextPath returns "" instead of null, so rewrite works without "null" being prepended
_context.setAttribute("org.eclipse.jetty.server.Executor", ForkJoinPool.commonPool()); // TODO: Change executor
}
@Override
public String getServletName()
{
return ReverseProxyController.class.getName();
}
@Override
public ServletContext getServletContext()
{
return _context;
}
@Nullable
@Override
public String getInitParameter(String configParameterName)
{
return _config.get(configParameterName);
}
@Override
public Enumeration<String> getInitParameterNames()
{
return Collections.enumeration(_config.keySet());
}
}
If I make HTTP requests with Postman or curl, I don't get all the
response headers. BUT if I have a debugger attached when
_proxyServlet.service is called, I have seen on some occasions that
the headers will show up! This makes me think that there some async
issues here. When I read the docs, it says about ProxyServlet that
"The request processing is asynchronous, but the I/O is blocking." So,
this made me think that I just needed to wait for request processing
to complete. To do this, I tried to add this right after the call to
_proxyServlet.service:
servletRequest.getAsyncContext().complete();
That didn't work though, and in fact resulted in no body content as
well as no HTTP headers.
I also tried overriding the ProxyServlet class's service method. In my
override, I removed the async processing of the request. This also
resulted in no HTTP headers and no body content.
With the code above, and this request
curl -X POST \
https://localhost:6749/admin/api/httpbin/post \
-H 'Content-Type: application/json' \
-H 'Origin:
example.com' \
-d '{
"test": 44
}'
I see these messages in the logs:
rewriting:
https://localhost:6749/admin/api/httpbin/post ->
http://httpbin.org/postResponse headers HttpResponse[HTTP/1.1 200 OK]@613a2308
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 17 Nov 2018 08:23:40 GMT
Content-Type: application/json
Content-Length: 628
Access-Control-Allow-Origin:
example.comAccess-Control-Allow-Credentials: true
Via: 1.1 vegur
.ReverseProxyController:612 67028742 proxying to downstream:
HttpResponse[HTTP/1.1 200 OK]@613a2308
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 17 Nov 2018 08:23:40 GMT
Content-Type: application/json
Content-Length: 628
Access-Control-Allow-Origin:
example.comAccess-Control-Allow-Credentials: true
Via: 1.1 vegur
There's those missing headers! Why aren't those being returned to my
HTTP client? I also see the following log messages after these:
org.eclipse.jetty.client.HttpReceiver:467 Request/Response succeeded: Result[HttpRequest[POST /post HTTP/1.1]@22dbb42a > HttpResponse[HTTP/1.1 200 OK]@613a2308] null
.ReverseProxyController:631 67028742 proxying successful
org.eclipse.jetty.server.HttpChannelState:669 complete HttpChannelState@792941c2{s=ASYNC_WAIT a=STARTED i=false r=IDLE w=false}
org.eclipse.jetty.server.HttpChannel:314 HttpChannelOverHttp@1c3d6aa2{r=4,c=true,a=ASYNC_WOKEN,uri=
https://localhost:6749/admin/api/httpbin/post,age=499} handle
https://localhost:6749/admin/api/httpbin/postorg.eclipse.jetty.server.HttpChannelState:219 handling HttpChannelState@792941c2{s=ASYNC_WOKEN a=COMPLETE i=false r=IDLE w=false}
org.eclipse.jetty.server.HttpChannel:327 HttpChannelOverHttp@1c3d6aa2{r=4,c=true,a=COMPLETING,uri=
https://localhost:6749/admin/api/httpbin/post,age=499} action COMPLETE
org.eclipse.jetty.server.HttpChannelState:860 onComplete HttpChannelState@792941c2{s=COMPLETING a=COMPLETE i=false r=IDLE w=false}
org.eclipse.jetty.server.HttpChannelState:1294 onEof HttpChannelState@792941c2{s=COMPLETED a=NOT_ASYNC i=false r=IDLE w=false}
Shortly after this, I start to see a few of these errors:
org.eclipse.jetty.io.FillInterest:134 onFail FillInterest@54bf29e8{SSLC.NBReadCB@3b86e603{SslConnection@3b86e603::SocketChannelEndPoint@45a8ac84{/0:0:0:0:0:0:0:1:54495<->/0:0:0:0:0:0:0:1:6749,OPEN,fill=FI,flush=-,to=30005/30000}{io=1/1,kio=1,kro=1}->SslConnection@3b86e603{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint@6bd593aa{/0:0:0:0:0:0:0:1:54495<->/0:0:0:0:0:0:0:1:6749,OPEN,fill=FI,flush=-,to=30006/30000}=>HttpConnection@1d3bb59c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@18d2abc6{s=START}]=>HttpChannelOverHttp@1c3d6aa2{r=4,c=false,a=IDLE,uri=null,age=0}}}
java.util.concurrent.TimeoutException: Idle timeout expired: 30004/30000 ms
at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
org.eclipse.jetty.io.WriteFlusher:471 ignored: WriteFlusher@2c599ac5{IDLE}->null
java.nio.channels.ClosedChannelException: null
at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:502) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:353) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:216) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:192) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:175) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.close(HttpConnectionOverHTTP.java:195) [jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onIdleExpired(HttpConnectionOverHTTP.java:145) [jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:401) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
Does anyone have any tips on what more to try or clues as to what may
be going wrong?
--
Regards,
Travis Spencer
