Community
Participate
Working Groups
SYMPTOM: In my company's environment, invoking "Help->Software Updates->Find an Install" is very slow (a minute and longer until the next window opens). I have several update sites installed. The proxy settings are filled in. Java is 1.5.0_05 DIAGNOSIS: I made an ethereal dump (attached), and it showed that eclipse tries to make a DNS resolution for the update sites, which eventually fails. This takes quite some time (a seperate issue, but not for here). I understand that this only is a problem if DNS Resolution fails for real world IP's, and it is slow, anyway I think it's a bug. WORKAROUND: putting the host names in question into the hosts file and assigning a fake and invalid ip helped in my case
Created attachment 31867 [details] ethereal dump during invokation This is the network dump my diagnosis is based on. It's based on a fresh Eclipse installation, so you only see on update site.
I detected the same problem. This problem didn't occur under eclipse 3.0, so there must be a significant change in update handling during 3.0 and 3.1.
I made a bug report to java.sun.com and got an internal review ID of: 618611 This review ID is NOT visible on on Sun Developer Network (SDN). I will post results ASAP.
(In reply to comment #3) > I made a bug report to java.sun.com and got an internal review ID of: 618611 > This review ID is NOT visible on on Sun Developer Network (SDN). > I will post results ASAP. > Here is the response from Sun: Re: (Incident Review ID: 618737) Eclipse 3.1 UpdateManager behind Firewall/Proxy makes DNS Lookup and crash "I have reviewed the Eclipse Bug report you have posted and it's possibly a bug in Eclipse 3.1 and not in the Java runtime. However, I will keep a close watch on this bug report. If it is identified that this could be triggered by a bug in the java.net package, we will investigate this further."
As far as I know all that we do is simply use java.net to connect to update site. All of this resolution happens inside java.net package. Eclipse can not do anything until we are either given exception or connection by java.net. So, i can not see how DNS resolution is eclipse update problem and not java.net problem.
I debugged my configuration and intercepted the blocking operation; here is my stack trace (partially): ... InetAddress.getByName(String) line: not available Handler(URLStreamHandler).getHostAddress(URL) line: not available Handler(URLStreamHandler).hostsEqual(URL, URL) line: not available Handler(URLStreamHandler).sameFile(URL, URL) line: not available Handler(URLStreamHandler).equals(URL, URL) line: not available URL.equals(Object) line: not available DiscoveryFolder.contains(SiteBookmark) line: 139 DiscoveryFolder.addBookmarks(IFeature) line: 130 DiscoveryFolder.initialize() line: 70 The problem occurs at the URL.equals(URL): this is specified as blocking, since it does an ip resolution. In the described configuration, the dns lookup will fail only after some long delay.
In my case this situation is even worse. When I try to invoke the dialog to install/update features, it can take several dozens of minutes (often longer than hour) to come up. Once I select sites and it goes to search those sites, each site can take 10-30 minutes just to display me the dialog to select one of the few possible sites to search. This is on really fast corporate network with Eclipse 3.2 Running on Fedora Core 5 even with fresh workspace and installation using JD 1.5.07.
I have just updated to version 3.2.1 and situation did improved. Now instead of hours for the first dialog to appear it takes roughly 20 - 30 minutes. So it doesn't appear to be encessarily JDK bug since nothing in my system has changed other that I have installed 3.2.1. Once the first dialog appears, it takes roughly 5 minutes for each of the other dialogs to appear. Once I select all the features I want to install, the download and install of roughly 100MB of new features took only couple of minutes. So whatever Eclipse is doing to show the first dialog (where I choose to either intalls updates or new features) is causing the slowness.
I have the same problem here with Eclipse 3.2.1 and Sun JRE 1.5.0.09 on Linux. When I invoke the dialog to install/update features, it takes several dozens of minutes before I can see or do anything. The same happens when I select an item. I had the same problem before with Eclipse 3.1, 3.2 and Sun JRE 1.4 as well as IBM JRE 1.4.
(In reply to comment #6) > InetAddress.getByName(String) line: not available > Handler(URLStreamHandler).getHostAddress(URL) line: not available > Handler(URLStreamHandler).hostsEqual(URL, URL) line: not available > Handler(URLStreamHandler).sameFile(URL, URL) line: not available > Handler(URLStreamHandler).equals(URL, URL) line: not available > URL.equals(Object) line: not available > DiscoveryFolder.contains(SiteBookmark) line: 139 > DiscoveryFolder.addBookmarks(IFeature) line: 130 > DiscoveryFolder.initialize() line: 70 > Branko, as you can see on previous stack trace (and URL class javadoc), the equals method provokes a DNS resolution. It appears that the main problem is that update manager should NOT use URL instances internally, but another type of structure. Perhaps use simple Strings and just build the URL object when trying to connect.
Also see this post: http://michaelscharf.blogspot.com/2006/11/javaneturlequals-and-hashcode-make.html
I've not had a chance to delve into the source, but it appears that DiscoveryFolder is either extending or aggregating a Set, so whenever a SiteBookmark is added to it, all existing elements must be compared with equals to see if it already exists in the Set. Additionally it appears that SiteBookmark.equals delegates to URL.equals. The DNS lookup could be broken at that point if it did URL.toString().equals. And similarly for SiteBookmark.hashCode.
*** Bug 156077 has been marked as a duplicate of this bug. ***
seems to have vanished with europe. any other opinions?
> seems to have vanished with europe. Well, it is not vanished but the delay is significantly less. Now I have to wait only 30 seconds and not 5 minutes :) But it is still slower than without sitting behind a proxy.
The improved performance is probably related to the use of update site digests. The rest of the slowdown is outside of Update scope. Most of the network magic happens in java.net - Update is just a customer (particularly now that Update is using the platform network support).
Some of the network magic that happens in java.net (like URL.equals(URL)) is documented as blocking, so the Update component shouldn't be calling it from the EDT, or at all if possible, as SiteBookmark.equals(...) does. Software Updates > Find and Install... used to do this a lot, but this bug seems to be fixed in Eclipse 3.3. The new 30 second UI freeze on my PC comes with no network activity and very high CPU use in org.eclipse.update.core.Feature.getIncludedFeatureReferences(), so it looks like a separate bug in Update that has nothing to do with java.net.
I can also confirm blocking waits in the update manager with Europa Fall-edition, behind my companies Proxy. I have to run Updates in the background because searching can take a long time, and performance inside the update manager, selecting packages, is extremely slow, blocking the UI while it downloads additional information. I suppose this will be fixed by the new update manager in Ganymede, but I'd prefer a fix for 3.3 as we cannot change to new major releases all too quickly.
There is no improvement with Ganymede RC3. I think it is even worse than with Europa.
Axel, could you please attach some stack dumps to help us figure out what is going on? Does it eventually succeed of fail?
Reply to comment #20: Help->Software updates->Available Software->Manage Sites will take about one minute before the UI responses. netstat reveals a UDP connection to our DNS nameserver on port 53 during this time. Adding or removing a site (local or remote) is a pain as you have to wait every time one minute.
(In reply to comment #20) > Axel, could you please attach some stack dumps to help us figure out what is > going on? Does it eventually succeed of fail? > Pascal, do you really need stack dumps? java.net.URL hashCode() and equals() are operations involving blocking network IO. In work environments with proxies, they are very expensive for common failure cases. The Eclipse proxy preferences don't get used. As such, these should not be used in Set/Maps etc. Can you key them on the String that is used to build the URL originally?
For those who want to install a proxy server locally to see the issue, try FreeProxy from http://www.handcraftedsoftware.org/index.php
Yuri, we do need stack dumps, because we can't crawl all over the code hunting this, unless of course you want :). Also the code between UM and p2 is completely different which is why I'm a bit surprised about the same characteristics. Anyway like with any bug report, the more the merrier.
(In reply to comment #24) > Yuri, we do need stack dumps, because we can't crawl all over the code hunting > this, unless of course you want :). > Also the code between UM and p2 is completely different which is why I'm a bit > surprised about the same characteristics. Anyway like with any bug report, the > more the merrier. > Fair comment. I am still using 3.3, but looking forward to the issue being fixed finally in 3.4. Thanks.
Axel, do you use manual proxy configuration (preferences -> General -> Network Connections) or system proxy configuration?
(In reply to comment #26) I use manual proxy configuration from the Eclipse preferences page. What do mean with "system proxy manual proxy configuration"? I have set the environment variable http_proxy as well (btw, I am using Linux).
Probably everyone is aware of: http://michaelscharf.blogspot.com/2006/11/javaneturlequals-and-hashcode-make.html I ran findbugs on p2. It reported a few classes using URL.equals() and URL.hashCode(). I tried to track down the usages of this classes. There are places avoiding the use of URL.equal/hashCode by using the URL.toExternalForm as the key for a map. However, the following places use URL.equals/hashCode. URLUtil.sameURL(URL, URL) invokes java.net.URL.equals(Object) line 30 -> AbstractArtifactRepository.equals(Object) -> HashSet: Util.getAggregatedBundleRepository(IProfile, int).bundleRepositories -> ArtifactRepositoryManager.getRepository(URL), iterates over a list of repositories and uses sameURL -> MetadataRepositoryManager.getRepository(URL), same as above -> ArtifactRepositoryManager.getRepositoryProperty(URL, String), same as above -> MetadataRepositoryManager.getRepositoryProperty(URL, String), same as above -> ArtifactRepositoryManager.isEnabled(URL), same as above -> MetadataRepositoryManager.isEnabled(URL), same as above FileTransferID.namespaceEquals(BaseID) line 37 FileTransferID.namespaceHashCode() line 45 -> seems to be not used that often SiteFeature.sameURL(URL, URL) invokes java.net.URL.equals(Object) line 54 -> only used in Stack and ArrayList within MetaDatagenerator RepositoryReference.equals(Object) invokes java.net.URL.equals(Object) line 40 RepositoryReference.hashCode() invokes java.net.URL.hashCode(), which line 50 -> Set: MetadataParser.RepositoryReferenceHandler(..) Than I started remote debugging a running Eclipse IDE (both RC4), having set breakpoints URL and URLUtils. When opening Help > Software Updates: - MetadataRepositoryManager.isEnabled(URL) called URLUtil ~ 40 times - MetadataRepositoryManager.getRepositoryProperty(URL, String) called URLUtil >>> 40 times I removed the URLUtil breakpoints because it became to hard to debug. As a result so far URLUtil.sameURL() and so URL.equals() became called very, very (!) often until the UI appears. Than I selected the Ganymede Update Site: - MetadataParser.RepositoryReferenceHandler(..) is called often (stopped counting) uses a Set<RepositoryReference> which does URL.equals/hashCode in its equals/hashCode methods Could that cause the delays? Stefan
Yes, we would like to move to using URI in 3.5 to avoid the URL class altogether. At one point I looked at changing references to URL#equals with URL.toExternalForm().equals(). However, I ran some benchmarks and found that URL#equals was actually 3 times *faster* than URL.toExternalForm().equals(). so I didn't make the change. The performance characteristics may be different in a proxy/firewall environment.
Created attachment 105167 [details] URL equals benchmark I would be curious to see the results of running this benchmark in other environments. The result on my machine is: Default equals 1437ms String equals 4985ms
URL.equals is expensive because of the call to resolve the host address. After being called once the value is cached which explains why the benchmark performs well in a loop. See URLStreamHandler.getHostAddress
Created attachment 105289 [details] URL performance test I created a little test that tests a list of URLs (almost 100), each URL is tested once - not looped to avoid caching effects - and with different usages (equals, toExternalForm,..). Each usage test runs with a new program start to again avoid any caching. I ran these tests in an environment without a proxy and within a companies corporate lan (with proxy). Comparing two equal urls take about 90 ms without a proxy and about 2 sec (!) with a proxy! Performing the comparison on the URL.toExternalForm strings takes in both environments about 2 ms (the urls are not that long). URL.equals() does for both urls a InetAddress.getByName() which results in a DNS lookup that most of the proxies do not fulfill. This is the reason for this delays. John´s test without proxy: Default equals 3234ms String equals 18313ms and with proxy: sorry still runs (!) I post the result tomorrow.
Could people that are seeing this specify in more detail the proxy environments that are showing the slowdown? That is, I suspect that such slowdowns are limited to only a few proxy environments...and not in all proxy environments. It would therefore be helpful to characterize the proxy environments where the problem exists. e.g. proxy used, configuration, etc.
(In reply to comment #33) > Could people that are seeing this specify in more detail the proxy environments > that are showing the slowdown? That is, I suspect that such slowdowns are > limited to only a few proxy environments...and not in all proxy environments. > It would therefore be helpful to characterize the proxy environments where the > problem exists. e.g. proxy used, configuration, etc. When I originally described the bug, I described that it depends on DNS resolution. The crucial factor seems to be that the DNS resolution for names outside the proxy doesn't work in the LAN behind the proxy. Unfortunately this seems to be a security concept widely used. So just using a proxy doesn't help, you also have to disable DNS lookups to the outside in your LAN in order to reproduce the problem. The DNS dependency is why the workaround with the hosts file works. Since also the timing of the failing DNS depends on a lot of parameters (network, local OS/IP stack, Java version, firewall settings...) the effect is varying a lot. The best way to avoid it, is using findbugs and eliminate URL.equals, resp. don't use the URL class as a key in maps regards, Gernot
(In reply to comment #34) > (In reply to comment #33) <stuff deleted> > When I originally described the bug, I described that it depends on DNS > resolution. The crucial factor seems to be that the DNS resolution for names > outside the proxy doesn't work in the LAN behind the proxy. Unfortunately this > seems to be a security concept widely used. > So just using a proxy doesn't help, you also have to disable DNS lookups to the > outside in your LAN in order to reproduce the problem. > The DNS dependency is why the workaround with the hosts file works. > Since also the timing of the failing DNS depends on a lot of parameters > (network, local OS/IP stack, Java version, firewall settings...) the effect is > varying a lot. Thanks, this is helpful. > The best way to avoid it, is using findbugs and eliminate URL.equals, resp. > don't use the URL class as a key in maps Although possible in the longer term, I don't believe this is possible in 3.4 timeframe (i.e. it's a done deal). So I think the most that can be done for 3.4 is to characterize the problematic proxies/proxy configs as precisely as possible, and identify workarounds for those situations (e.g. hosts file for some configs, etc).
In reply to comment 34 and comment 35 > > The DNS dependency is why the workaround with the hosts file works. > So I think the most that can be done for 3.4 is to characterize the problematic > proxies/proxy configs as precisely as possible, and identify workarounds for > those situations (e.g. hosts file for some configs, etc). A lot of our customers are in the area of finance and assurance. Their security settings are extremely restrictive, i.e. they don´t like workarounds like modifying host files. However, as Gernot said we have to eliminate URL.equals/hashCode.
(In reply to comment #36) > In reply to comment 34 and comment 35 > > > > The DNS dependency is why the workaround with the hosts file works. > > > So I think the most that can be done for 3.4 is to characterize the problematic > > proxies/proxy configs as precisely as possible, and identify workarounds for > > those situations (e.g. hosts file for some configs, etc). > > A lot of our customers are in the area of finance and assurance. Their security > settings are extremely restrictive, i.e. they don´t like workarounds like > modifying host files. I'm sympathetic, but I don't think there's anything else but workarounds to do for 3.4. > However, as Gernot said we have to eliminate URL.equals/hashCode. > Would be nice, but seems far too late in cycle for anything like this for 3.4. But I'm not the steward, author, maintainer of the code in question here, so it's not my call.
Created attachment 105378 [details] URLUtil.sameURL() uses internaly URI + unit test In reply to comment 37: Yes Scott, you are right. I guess it is now too late. Nevertheless, the attached patch ´fixes´ URLUtil.sameURL() by using internally URIs for the comparison, but I am not sure how general this ´fix´ is. As described in comment 28 URLUtil.sameURL() is used very often.
In reply to comment 32: Just for completeness John´s test in our ´proxied´ LAN results in: Default equals 329674ms String equals 13203ms
Stefan, does the patch helps significantly when running in proxy environments?
Pascal, I did the following in a ´proxied´ LAN with the patched (URLUtil.sameURL()) and the original RC4: Help > Software Updates > Available Software and than expanded "Ganymede Update Site. Expanding took in the original RC4 about 2 mins and with the patched RC4 10 secs.! Also RepositoryReference could be easily made faster with this trick without changing its interface. This is not part of patch.
Forgot to mention: With the patched version I could install the CDT without any problems.
Then we may want to consider for 3.4.1 if the patch is not too scary.
I have released Stefan's fix into the 3.4.x and HEAD streams. There is a lot of general discussion in this bug report about other potential problem areas, but we should open specific bugs reports for outstanding issues after this fix is released. From Stefan's testing this fix alone results in an order of magnitude improvement in performance. There is also a general bug 237776 to replace our use of URL with URI throughout the p2 code base.
*** Bug 164211 has been marked as a duplicate of this bug. ***
Hi John, I'm evaluating the M20080903-2000 (3.4.1 RC2) build for our lab and see that the patch from comment 38 is not a part of URLUtil class. Actually, the org.eclipse.equinox.p2.core plugin version is 1.0.0.v20080530-1237 which is a month *before* the patch. Is there any configuration issue with p2 versions included into 3.4.1 RC2??? I guess at least you can't mark this bug as resolved/fixed with the target 3.4.1, but I would very appreciate any attempt to get the fix into the 3.4.1 final, as we are also sitting behind the firewall...
You're correct, the fix was released into the maintenance stream but never tagged. It is now too late for 3.4.1 but I'll tag and release for 3.4.2. Mea culpa.
Released for 3.4.2.
To clarify what was released, I released the patch from comment #38 (attachment 105378 [details]), but only the URLUtil portion. The added test case from the patch was not released.
