On 5/31/23 17:48, Bill Ross wrote:
Can you swap in another httpclient to test? I assume swapping jetty server would be too much, given something works. :-)
I can't do anything about the Jetty server without upgrading Solr. I really want to get them upgraded, but it's not up to me.
I tried to use the legacy SolrJ clients that utilize Apache HttpClient 4.x, but for an unknown reason I was not able to get those clients to work. I am not using any http client directly, I use SolrJ. Layers upon layers. I am completely shielded from any direct interaction with the Jetty client by SolrJ.
From faster result on smaller batch size: are you monitoring memory use? I'd try even smaller, looking at the perf profile for clues.
The queries gathering documents from the source are faster with a filter in place, so they feed data to the queue faster. I think this is probably because it is only sorting a few million documents for each document batch instead of the full 30 million.
I was running my program with a 1GB heap. With a queue size of 100000 or 150000, that worked well.
I later bumped the queue size to 200000 and had to bump the heap because I got OOME. The space is consumed by the SolrInputDocument objects on the queue. I set the heap to 2GB for the 200K queue size. Now the max queue size is 500K and the heap is 5GB. A larger queue evens out the transfer of data from the query thread to the indexing threads and keeps the migration from stalling.
I'm using ZGC to optimize for latency. The code is compiled for Java 11. Thanks, Shawn
_______________________________________________
jetty-users mailing list jetty-users@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
