Re: [jetty-users] [eclipse/jetty.project] Lockup processing POST request

[Silvio Bierman <sbierman@xxxxxxxxxxxxxxxxxx>]

    Hello Joakim,
    
    You talk about the code in the alternative servlet TestServlet2 that
    treats the POST body as a ZIP. As posted the code however uses
    TestServlet1 that simply uses input.transferTo to read and discard
    all bytes of the POST body without making any assumptions about the
    content. This results in the same lockup for larger files.
    
    So although I understand your point about reading the complete body
    when parsing input in some format while reading the request that is
    not the cause of the issue. I merely posted the second servlet
    because it demonstrates that the first part of the request body is
    read properly, printing out the name/size of several initial entries
    in a posted ZIP file and then stalling.
    
    Kind regards,
    
    Silvio

On 14-09-2023 14:55, Joakim Erdfelt wrote:

You have to finish reading from the HTTP stream.
The ZipInputStream does not read to EOF.
Just closing the request.getInputStream is insufficient in many cases, as it can leave data on the connection that is unread (breaking HTTP/1 connection persistence in the process).
What if the transfer-encoding was chunked? and the terminal chunk hasn't been read from the network buffers yet?
What if the transfer-encoding final chunk hasn't even been sent by the client yet?
What if the stream contained superflous data, unrelated to the Zip stream after the zip stream?

All your code is doing is iterating through the Zip directory/dictionary/index for entries, and reading the entry content.
How this is done is by reading the entire stream until it finds the bytes indicating the start of directory section at the end of the stream, then reading each entry of the directory to know where in the previous parts of the stream the ZipEntry contents are located (and what size they are).
So you are basically reading ALL of the file contents first, then slowly iterating through the dictionary, and then going backwards in the zip stream to each entry contents, once you reach an entry in the Zip directory section that cannot be represented by java ZipEntry it stops. You haven't actually read the whole stream yet if that last entry is something structural that cannot be represented by a java ZipEntry.

How do you fix this?
You have to satisfy the basics of HTTP here.
Even libraries that read / parse JSON and XML have this same issue.
Every modern JSON and XML library have options to continue reading to EOF.
They can even be configured to behave differently depending on what is found after the content.
Eg: in many XML and JSON parsers there's an option to fail the document parse if any non-whitespace is discovered after the document.

See TODO in the code snippet below.

		val in = request.getInputStream
		val zip = new ZipInputStream(in)

		var entry = zip.getNextEntry
		while (entry != null)
		{
			val out = new CountingOutputStream(new NullOutputStream)
			zip.transferTo(out)
			out.close
			System.err.println(entry.getName + ":" + out.bytesWritten)
			entry = zip.getNextEntry
		}
		zip.close
                // TODO: you need to finish reading from `in` here, the HTTP stream is not yet complete, just the ZIP content is.
                // You have to choose a behavior here, 
                // eg: if there is content, do you fail the upload? or ignore it the extra content? warn the user?
                // small amounts of content ok? large amounts of extra content show a problem?

If you don't want to do this, you can send the Zip stream in a multipart/form-data, which puts the HTTP stream behavior requirement on the Server, not you.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: <eclipse/jetty.project/issues/10513/1719399968@github.com>