Community
Participate
Working Groups
In the usage we are doing of the http connection, would there be any advantage in switching to use the apache http client in the SDK, the Installer and the Admin UI? Size wise, a rough estimate makes it look like it would bring in an additional 400 K
Just for everyone's info, to use the httpclient filetransfer provider would mean: 1) including 3 Orbit bundles: a) httpclient 3.1.0 - 299k b) commons codec - 1.2.0 - 35k c) commons logging - 1.0.4 - 43k 2) including 1 additional ECF bundle (the ECF provider) org.eclipse.ecf.provider.filetransfer.httpclient - 22k That makes a total of ~400k as Pascal indicated Also FYI...we have a workaround/fix for the connect and/or read timeout problem for the JRE in bug 219368. FWIW, I think it would be a good idea to start using/distributing/fully testing the apache httpclient 3.1 ECF provider. Although bug 219368 deals with the connect timeout problem (albeit in an ungraceful way), I suspect that there could be other JRE-induced issues with http transfer...at least on some VMs, and I think the httpclient implementation is likely to ultimately be more robust. But, I could be wrong...and there may be other unexpected problems with the httpclient implementation (at least in that version). From what I can tell and find, however, it seems pretty robust.
While evaluating different http client solutions for smartup (i.e. compeople´s p2) a while ago I had problems with the apache http client when using it inside corporate networks (windows!). It seems that the support for proxy authentication/NTLM in a windows environment is not that good as with sun´s http connection. @Scott: Do you have any experiences concerning proxy authentication/NTLM with the htpp client?
(In reply to comment #2) > While evaluating different http client solutions for smartup (i.e. compeople´s > p2) a while ago I had problems with the apache http client when using it inside > corporate networks (windows!). It seems that the support for proxy > authentication/NTLM in a windows environment is not that good as with sun´s > http connection. > > @Scott: Do you have any experiences concerning proxy authentication/NTLM with > the htpp client? > No...unfortunately no direct experience. Although I did see in the httpclient codebase that NTLM support was there...I just haven't been able to test it (I don't have access to such proxy servers). For reference, I found this: http://hc.apache.org/httpclient-3.x/authentication.html#NTLM Would you be able to test both the JRE-based and the httpclient 3.1.0-based impls in that environment? Which httpclient version did you see this problem with? Is it possible that the problems were fixed in httpclient in 3.1.0?
Yes, I could run such tests. Scott, could you provide me with test applications? This comparison http://www.oaklandsoftware.com/product_http/compare.html might also be interesting. Their http client seems also to be quite good, but it is not for free. See also Apache http client´s documented limitations: http://hc.apache.org/httpclient-3.x/authentication.html#Known_limitations_and_problems I think I was evaluating http client 3.0/3.0.1 in spring 2006. Regarding p2 I would propose to not stick to a certain http client/transport implementation. I am aware of the abstract Transport class in p2, that would allow using different implementations. Currently p2 uses ECF which itself can be configured for different transports. But I don´t know how easy that is.
Thanks for the links Stefan. To be clear here, the proposal is not to drop ECF in favor of a direct use of the apache http client. Today ECF uses the apache http client when it is available (when the appropriate ECF file transfer provider is here). The question is "does apache http client buy us anything over what the JREs offer"? Also it is interesting to note that a commercial application could choose to ship p2 with the oakland http client by writing an ecf provider. What are the EE requirements of apache http client and dependencies?
(In reply to comment #4) > Yes, I could run such tests. Scott, could you provide me with test > applications? Yes. There is a set of tests in the following location /cvsroot/technology/org.eclipse.ecf/tests In the following plugins: org.eclipse.ecf.tests.filetransfer These are Junit plugin tests. We have been using these to test against the proxy configurations that we have access to. > This comparison http://www.oaklandsoftware.com/product_http/compare.html might > also be interesting. Their http client seems also to be quite good, but it is > not for free. > See also Apache http client´s documented limitations: > http://hc.apache.org/httpclient-3.x/authentication.html#Known_limitations_and_problems > I think I was evaluating http client 3.0/3.0.1 in spring 2006. > > Regarding p2 I would propose to not stick to a certain http client/transport > implementation. I am aware of the abstract Transport class in p2, that would > allow using different implementations. Currently p2 uses ECF which itself can > be configured for different transports. But I don´t know how easy that is. The answer to this is that it's as easy of installing a new plugin. There is an extension point: org.eclipse.ecf.provider.filetransfer.retrieveFileTransferProtocolFactory that allows plugins to define themselves as handlers for a given protocol (e.g. http, etc). If no other protocol providers are defined, then the JRE URLConnection is used by the org.eclipse.ecf.provider.filetransfer plugin directly. The httpclient provider (org.eclipse.ecf.provider.filetransfer.httpclient) uses this extension point to define itself as the protocol handler for http and https. So by simply installing the httpclient provider bundle in the environment it will be used. This can/could be done with other providers as well.
(In reply to comment #5) > Thanks for the links Stefan. > To be clear here, the proposal is not to drop ECF in favor of a direct use of > the apache http client. Today ECF uses the apache http client when it is > available (when the appropriate ECF file transfer provider is here). Yes, as described in comment #6. >The > question is "does apache http client buy us anything over what the JREs offer"? > Also it is interesting to note that a commercial application could choose to > ship p2 with the oakland http client by writing an ecf provider. Yes, exactly. > > What are the EE requirements of apache http client and dependencies? > According to the manifest in the Orbit projects, its currently set to CDC 1.0/Foundation 1.0, JDK 1.2 for httpclient itself and CDC 1.0/Foundation 1.0, JDK 1.3 for codec and commons logging (which httpclient depends upon). I don't know if the 1.3 in codec and commons logging is actually required or just a typo.
I never thought of dropping ECF. It was just for completeness. @Scott: I will try to run the tests as soon as possible. I can not predict when. Shall I post the results here?
(In reply to comment #8) > I never thought of dropping ECF. It was just for completeness. > > @Scott: I will try to run the tests as soon as possible. I can not predict > when. Shall I post the results here? > Hi Stephan...we have bug 208012 to record proxy testing, so would appreciate it if you could post results there. Thanks Stephan.
*** Bug 219498 has been marked as a duplicate of this bug. ***
adding dependency on 224196, which is being worked. With resolution of 224196 the httpclient 3.0.1-based provider will be ready for testing and limited distribution as part of 3.5M3
(In reply to comment #11) > adding dependency on 224196, which is being worked. With resolution of 224196 > the httpclient 3.0.1-based provider will be ready for testing and limited > distribution as part of 3.5M3 > We are now very close to resolving bug #224196, so I have created a p2 repository for installing the current ECF httpclient-based provider and made it available as a zip on this page: http://wiki.eclipse.org/ECF_Apache_Httpclient-Based_Provider For those that are able to test, please follow the instructions on that page to download the zip, unzip to local filesystem, and install the repositories (in the repos directory) via the p2 update manager. Nevermind the incorrect feature meta-data in the repository. Please report any bugs discovered to technology/ECF/ecf.filetransfer component, with [filetransfer][httpclient] in the title of bug. Thanksinadvance for help in testing this.
I've created a metadata repository for httpclient installation via p2 at the following location: http://download.eclipse.org/technology/ecf/httpclientrepo The replication is currently *off* due to 3.4.1 service release, so the replication will not occur immediately. On dev.eclipse.org it's available on /home/data/httpd/download.eclipse.org/technology/ecf/httpclientrepo
I tested this as requested in eclipse-dev and the upgrade of various Ganymede SR1 packages succeeded. Before installing this feature, it had gotten stuck multiple times (I think it was 4, but I stopped counting).
We might also investigate the use of the Jetty HTTP client. I've hear good things about it. See http://docs.codehaus.org/display/JETTY/Jetty+HTTP+Client. Might be smaller than the apache client. Heck, we should really support both in ECF and then just be able to use whatever we want in p2...
I also tested this following http://dev.eclipse.org/mhonarc/lists/eclipse-dev/msg08356.html in reply to Bug 248568, c3. I wasn’t able to install: An error occurred while collecting items to be installed No repository found containing: org.apache.commons.codec/osgi.bundle/1.2.0.200809231229 No repository found containing: org.apache.commons.httpclient/osgi.bundle/3.0.1.200809231229 No repository found containing: org.apache.commons.logging/osgi.bundle/1.0.4.200809231229 No repository found containing: org.eclipse.ecf.provider.filetransfer.httpclient/osgi.bundle/2.1.0.200809231229 No repository found containing: org.eclipse.ecf.provider.filetransfer.httpclient.ssl/osgi.bundle/1.0.0.200809231229 No repository found containing: org.eclipse.ecf.provider.filetransfer.httpclient_feature/org.eclipse.update.feature/2.1.0.200809231229 Could this be because I have a language pack installed? These often give installation problems, though I don’t think it’s the cause this time.
forked off the consideration of the Jetty HTTP client (see comment 15) into bug 248866
Hendrick, this problem is different. It is caused by the fact that the artifact repository does not seem to be found. Try removing and adding again the repository.
I sent this to the mailing list, but maybe it should go here: <snip> While I realize that a positive result means very little when testing.... here is at least one data point. I was trying to upgrade from 3.4 to the 3.5 Milestones. I added the milestone update site and selected the Eclipse SDK. Each time I did this it hung (once after 11% complete, once after 4% complete and once after 15% complete). I then used the apache http client (simply installed that), and I was able to upgrade. It could have been coincidence, but at least it's one data point we can use. </snip>
(In reply to comment #19) <stuff deleted> Thanks Ian for the report. I do think that we should use this bug for reporting tests. Stefan...in comment #2 you mention a problematic proxy situation for a version of httpclient. Could you try to reproduce the problem with the current httpclient provider and report the details of any problems (e.g. proxy env, authentication, etc)?
Yes, Scott. It is on my task list.
This has been released in I 20081014.
My apologies for being so late! Scott, I was following the instructions on http://wiki.eclipse.org/ECF_Apache_Httpclient-Based_Provider and I made the test on a WinXP box behind a corporate proxy/firewall. Step by step: - Eclipse 3.4 - Installed ECF-Apache-Http-Client (local repo) - Changed the network settings to use proxy settings (did not enable proxy authentication) - Tried to install regex-util: http://regex-util.sourceforge.net/update/ - Got the error message: No repository found at http://regex-util.sourceforge.net/update/ Error reading update site http://regex-util.sourceforge.net/update/. Proxy Authentication Required - Uninstalled ECF-Apache-Http-Client - Tried again to install regex-util - Works! - Uninstalled regex-util - Installed ECF-Apache-Http-Client (local repo) again - Enabled proxy authentication/using proxy (network setings) - Installed regex-util - Failed again with the same error message as above with stacktrace: javax.security.auth.login.LoginException: Proxy Authentication Required at org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientRetrieveFileTransfer.openStreams(HttpClientRetrieveFileTransfer.java:346) at org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.sendRetrieveRequest(AbstractRetrieveFileTransfer.java:639) at org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.sendRetrieveRequest(AbstractRetrieveFileTransfer.java:381) at org.eclipse.ecf.provider.filetransfer.retrieve.MultiProtocolRetrieveAdapter.sendRetrieveRequest(MultiProtocolRetrieveAdapter.java:95) at org.eclipse.equinox.internal.p2.updatesite.ECFTransport.transfer(ECFTransport.java:284) at org.eclipse.equinox.internal.p2.updatesite.ECFTransport.performDownload(ECFTransport.java:193) at org.eclipse.equinox.internal.p2.updatesite.ECFTransport.download(ECFTransport.java:167) at org.eclipse.equinox.internal.p2.updatesite.UpdateSite.loadSiteFile(UpdateSite.java:156) at org.eclipse.equinox.internal.p2.updatesite.UpdateSite.load(UpdateSite.java:105) at org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepository.<init>(UpdateSiteMetadataRepository.java:52) at org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepositoryFactory.load(UpdateSiteMetadataRepositoryFactory.java:48) at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:485) at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:452) at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:435) at org.eclipse.equinox.internal.provisional.p2.ui.operations.ProvisioningUtil.loadMetadataRepository(ProvisioningUtil.java:71) at org.eclipse.equinox.internal.provisional.p2.ui.dialogs.AvailableIUGroup$6.run(AvailableIUGroup.java:404) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) - I retried several times with slightly different user ids (i.e. domain\user, domain/user, user) but the effect was always the same. If there is anything I can do for you let me know.
Hi Stefan, (In reply to comment #23) > My apologies for being so late! > > Scott, I was following the instructions on > http://wiki.eclipse.org/ECF_Apache_Httpclient-Based_Provider and I made the > test on a WinXP box behind a corporate proxy/firewall. Could you describe this proxy configuration? > Step by step: > - Eclipse 3.4 > - Installed ECF-Apache-Http-Client (local repo) > - Changed the network settings to use proxy settings (did not enable proxy > authentication) Does this proxy expect/require authentication (username/password)? I assume no, since you didn't enable proxy authentication. > - Tried to install regex-util: http://regex-util.sourceforge.net/update/ > - Got the error message: > No repository found at http://regex-util.sourceforge.net/update/ > Error reading update site http://regex-util.sourceforge.net/update/. > Proxy Authentication Required > - Uninstalled ECF-Apache-Http-Client > - Tried again to install regex-util > - Works! You didn't make any change to the authentication info in the Network preferences at this point? > - Uninstalled regex-util > - Installed ECF-Apache-Http-Client (local repo) again > - Enabled proxy authentication/using proxy (network setings) So it sounds like you did add proxy authentication info via network settings at this point. Did you try the following sequence after specifying the proxy *without* the authentication info (as you apparently did above)? > - Installed regex-util > - Failed again with the same error message as above with stacktrace: > javax.security.auth.login.LoginException: Proxy Authentication Required > at > org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientRetrieveFileTransfer.openStreams(HttpClientRetrieveFileTransfer.java:346) > at > org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.sendRetrieveRequest(AbstractRetrieveFileTransfer.java:639) > at > org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.sendRetrieveRequest(AbstractRetrieveFileTransfer.java:381) > at > org.eclipse.ecf.provider.filetransfer.retrieve.MultiProtocolRetrieveAdapter.sendRetrieveRequest(MultiProtocolRetrieveAdapter.java:95) > at > org.eclipse.equinox.internal.p2.updatesite.ECFTransport.transfer(ECFTransport.java:284) > at > org.eclipse.equinox.internal.p2.updatesite.ECFTransport.performDownload(ECFTransport.java:193) > at > org.eclipse.equinox.internal.p2.updatesite.ECFTransport.download(ECFTransport.java:167) > at > org.eclipse.equinox.internal.p2.updatesite.UpdateSite.loadSiteFile(UpdateSite.java:156) > at > org.eclipse.equinox.internal.p2.updatesite.UpdateSite.load(UpdateSite.java:105) > at > org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepository.<init>(UpdateSiteMetadataRepository.java:52) > at > org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepositoryFactory.load(UpdateSiteMetadataRepositoryFactory.java:48) > at > org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:485) > at > org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:452) > at > org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:435) > at > org.eclipse.equinox.internal.provisional.p2.ui.operations.ProvisioningUtil.loadMetadataRepository(ProvisioningUtil.java:71) > at > org.eclipse.equinox.internal.provisional.p2.ui.dialogs.AvailableIUGroup$6.run(AvailableIUGroup.java:404) > at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) > - I retried several times with slightly different user ids (i.e. domain\user, > domain/user, user) but the effect was always the same. So the thing that's puzzeling to me is that it sounds like when you used the urlconnection provider (i.e. not the httpclient provider) that the authentication happened even though you did not specify proxy authentication info in the network settings preferences. Is that what happened? If so, did you try the httpclient provider with a similar approach (i.e. with the authentication info *not* explicitly given in network settings (i.e. like it sounds like what you did above)? If so did it have the same exception as above? > > If there is anything I can do for you let me know. > Well, actually there is. If you could get the httpclient provider and run it from the debugger I would like to verify that the code in HttpClientRetrieveFileTransfer.setupProxy is running correctly...i.e: see >>>>>> below protected void setupProxy(Proxy proxy) { if (proxy.getType().equals(Proxy.Type.HTTP)) { final ProxyAddress address = proxy.getAddress(); httpClient.getHostConfiguration().setProxy(getHostFromURL(address.getHostName()), address.getPort()); final String proxyUsername = proxy.getUsername(); final String proxyPassword = proxy.getPassword(); if (proxyUsername != null) { final Credentials credentials = new UsernamePasswordCredentials(proxyUsername, proxyPassword); final AuthScope proxyAuthScope = new AuthScope(address.getHostName(), address.getPort(), AuthScope.ANY_REALM); >>>>> XXX I would like to make sure that the following setProxyCredentials >>>>> call is being made in your environment with the correct >>>>>> proxyAuthScope and correct credentials information httpClient.getState().setProxyCredentials(proxyAuthScope, credentials); } } else if (proxy.getType().equals(Proxy.Type.SOCKS)) { proxyHelper.setupProxy(proxy); } } The idea is that if the setProxyCredentials method is getting called, and with the proper parameters, then this is likely some issue with httpclient 3.0.1 itself. If it's not getting called for whatever reason, then it's something else that we can fix.
Hi Scott, (In reply to comment #24) > Hi Stefan, > > (In reply to comment #23) > > My apologies for being so late! > > > > Scott, I was following the instructions on > > http://wiki.eclipse.org/ECF_Apache_Httpclient-Based_Provider and I made the > > test on a WinXP box behind a corporate proxy/firewall. > > > Could you describe this proxy configuration? The proxy is a squid 3 which supports NTLM. > > > > Step by step: > > - Eclipse 3.4 > > - Installed ECF-Apache-Http-Client (local repo) > > - Changed the network settings to use proxy settings (did not enable proxy > > authentication) > > Does this proxy expect/require authentication (username/password)? I assume > no, since you didn't enable proxy authentication. The proxy requires authentication. I did not enable proxy authentication here (but later). I wanted to check whether the HttpClient can handle proxy authentication like Sun´s HttpUrlConnection. > > > - Tried to install regex-util: http://regex-util.sourceforge.net/update/ > > - Got the error message: > > No repository found at http://regex-util.sourceforge.net/update/ > > Error reading update site http://regex-util.sourceforge.net/update/. > > Proxy Authentication Required > > - Uninstalled ECF-Apache-Http-Client > > - Tried again to install regex-util > > - Works! > > > You didn't make any change to the authentication info in the Network > preferences at this point? Correct. Sun´s HttpUrlConnection supports NTLM (single sign on). That´s why it was possible to authenticate with the proxy. > > > > - Uninstalled regex-util > > - Installed ECF-Apache-Http-Client (local repo) again > > - Enabled proxy authentication/using proxy (network setings) > > So it sounds like you did add proxy authentication info via network settings at > this point. Did you try the following sequence after specifying the proxy Correct. > *without* the authentication info (as you apparently did above)? I tried to install the regex-util with HttpClient once without authentication info (sequence above) and than with authentication info (the following sequence). > > > - Installed regex-util > > - Failed again with the same error message as above with stacktrace: > > javax.security.auth.login.LoginException: Proxy Authentication Required > > at > > org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientRetrieveFileTransfer.openStreams(HttpClientRetrieveFileTransfer.java:346) > > at > > org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.sendRetrieveRequest(AbstractRetrieveFileTransfer.java:639) > > at > > org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.sendRetrieveRequest(AbstractRetrieveFileTransfer.java:381) > > at > > org.eclipse.ecf.provider.filetransfer.retrieve.MultiProtocolRetrieveAdapter.sendRetrieveRequest(MultiProtocolRetrieveAdapter.java:95) > > at > > org.eclipse.equinox.internal.p2.updatesite.ECFTransport.transfer(ECFTransport.java:284) > > at > > org.eclipse.equinox.internal.p2.updatesite.ECFTransport.performDownload(ECFTransport.java:193) > > at > > org.eclipse.equinox.internal.p2.updatesite.ECFTransport.download(ECFTransport.java:167) > > at > > org.eclipse.equinox.internal.p2.updatesite.UpdateSite.loadSiteFile(UpdateSite.java:156) > > at > > org.eclipse.equinox.internal.p2.updatesite.UpdateSite.load(UpdateSite.java:105) > > at > > org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepository.<init>(UpdateSiteMetadataRepository.java:52) > > at > > org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepositoryFactory.load(UpdateSiteMetadataRepositoryFactory.java:48) > > at > > org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:485) > > at > > org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:452) > > at > > org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:435) > > at > > org.eclipse.equinox.internal.provisional.p2.ui.operations.ProvisioningUtil.loadMetadataRepository(ProvisioningUtil.java:71) > > at > > org.eclipse.equinox.internal.provisional.p2.ui.dialogs.AvailableIUGroup$6.run(AvailableIUGroup.java:404) > > at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) > > - I retried several times with slightly different user ids (i.e. domain\user, > > domain/user, user) but the effect was always the same. > > > So the thing that's puzzeling to me is that it sounds like when you used the > urlconnection provider (i.e. not the httpclient provider) that the > authentication happened even though you did not specify proxy authentication > info in the network settings preferences. Is that what happened? Well, the stack trace above shows that the class HttpClientRetrieveFileTransfer is involved. My assumption is that it uses the apache HttpClient, does it? > > If so, did you try the httpclient provider with a similar approach (i.e. with > the authentication info *not* explicitly given in network settings (i.e. like That´s what I did above. > it sounds like what you did above)? If so did it have the same exception as > above? Yes. > > > > > If there is anything I can do for you let me know. > > > > Well, actually there is. If you could get the httpclient provider and run it > from the debugger I would like to verify that the code in > HttpClientRetrieveFileTransfer.setupProxy is running correctly...i.e: > see >>>>>> below > > > protected void setupProxy(Proxy proxy) { > if (proxy.getType().equals(Proxy.Type.HTTP)) { > final ProxyAddress address = proxy.getAddress(); > > httpClient.getHostConfiguration().setProxy(getHostFromURL(address.getHostName()), > address.getPort()); > final String proxyUsername = proxy.getUsername(); > final String proxyPassword = proxy.getPassword(); > if (proxyUsername != null) { > final Credentials credentials = new > UsernamePasswordCredentials(proxyUsername, proxyPassword); > final AuthScope proxyAuthScope = new AuthScope(address.getHostName(), > address.getPort(), AuthScope.ANY_REALM); > > >>>>> XXX I would like to make sure that the following setProxyCredentials > >>>>> call is being made in your environment with the correct > >>>>>> proxyAuthScope and correct credentials information > httpClient.getState().setProxyCredentials(proxyAuthScope, credentials); > } > } else if (proxy.getType().equals(Proxy.Type.SOCKS)) { > proxyHelper.setupProxy(proxy); > } > } > > The idea is that if the setProxyCredentials method is getting called, and with > the proper parameters, then this is likely some issue with httpclient 3.0.1 > itself. If it's not getting called for whatever reason, then it's something > else that we can fix. > Everything looks good. setupProxy() gets called, setProxyCredentials() gets called with parameters that look good. However, the request returns with error (message box): No repository found at http://regex-util.sourceforge.net/update/ This message is somehow misleading. I also debugged into openStreams() and there is a LoginException thrown with Messages.HttpClientRetrieveFileTransfer_Proxy_Auth_Required which than gets wrapped by a IncomingFileTransferException. We can continue Über-Remote-Debugging!
<stuff deleted> > > Could you describe this proxy configuration? > > The proxy is a squid 3 which supports NTLM. > <stuff deleted> > Well, the stack trace above shows that the class HttpClientRetrieveFileTransfer > is involved. My assumption is that it uses the apache HttpClient, does it? Yes. <stuff deleted> > > The idea is that if the setProxyCredentials method is getting called, and with > > the proper parameters, then this is likely some issue with httpclient 3.0.1 > > itself. If it's not getting called for whatever reason, then it's something > > else that we can fix. > > > > Everything looks good. setupProxy() gets called, setProxyCredentials() gets > called with parameters that look good. Hmmm. Unfortunately this suggests that httpclient 3.0.1 is having the problem with NTLM and auth rather than ECF code. Do others have experience with an issue like this for httpclient 3.0.1 and proxy authentication (with NTLM I guess)? > However, the request returns with error (message box): > No repository found at http://regex-util.sourceforge.net/update/ > This message is somehow misleading. I also debugged into openStreams() and > there is a LoginException thrown with > Messages.HttpClientRetrieveFileTransfer_Proxy_Auth_Required which than gets > wrapped by a IncomingFileTransferException. Where is the original exception thrown? Is it this block? (line 500 in HEAD): } else if (code == HttpURLConnection.HTTP_PROXY_AUTH) { getMethod.releaseConnection(); throw new LoginException(Messages.HttpClientRetrieveFileTransfer_Proxy_Auth_Required); > > We can continue Über-Remote-Debugging! > OK great. Not as fast a local debugging, but much better than nothing. Thanks much.
RE: Comment 26: > Hmmm. Unfortunately this suggests that httpclient 3.0.1 is having the problem > with NTLM and auth rather than ECF code. > > Do others have experience with an issue like this for httpclient 3.0.1 and > proxy authentication (with NTLM I guess)? http://oaklandsoftware.com/product_http/compare.html seems to indicate that Apache's HTTP client still only has partial support for NTLM and doesn't have any support for NTLM v2. I'm not sure if this is the issue you are running in to but seems like it might be related.
(In reply to comment #27) > RE: Comment 26: > > > Hmmm. Unfortunately this suggests that httpclient 3.0.1 is having the problem > > with NTLM and auth rather than ECF code. > > > > Do others have experience with an issue like this for httpclient 3.0.1 and > > proxy authentication (with NTLM I guess)? > > http://oaklandsoftware.com/product_http/compare.html seems to indicate that > Apache's HTTP client still only has partial support for NTLM and doesn't have > any support for NTLM v2. I'm not sure if this is the issue you are running in > to but seems like it might be related. > Thanks Tim. According to the oaklandsoftware doc the NTLM support is: NTLM - The JRE 1.4.2 supports NTLM only on Windows. The Oakland Software and Apache HTTP clients support NTLM on any platform. However only the Oakland Software NTLM support correctly supports both Unicode and OEM encoding for NTLM and supports NTLM V2. The Sun implementation in JRE 6 and 1.5_08 and higher support NTLM V2 on all platforms, though on Windows they use the current user's credentials by default. Does this (no support for both Unicode and OEM encoding and no support for NTLM V2) square with your experience Stefan (i.e. is this NTLM v2 that you are using)?
Another question: In the Oakland software review, I don't see any specification of what version of Apache httpclient is being discussed. Does anyone know which one it is? (i.e. 3.0.1...which is what we are currently using)? Further...does anyone know if the NTLM v2 support exists in later versions of httpclient (e.g. 3.1...which I *think* is the last 3.X release). Stefan could you try those (httpclient > 3.0.1). For people's information, the httpcomponents (httpclient v4.0beta3) with new io support was recently added to Orbit (bug #251137). ECF has started an effort to create an ECF filetransfer provider from the nio-based httpcomponents v4.0 for ECF 3.0/Europa. See bug #251740.
The most direct information on the web I can find about the Apache client is that it does not support NTLMv2 which may be what the issue is here though I'm certainly interested in hearing the feedback too. If reading all of this correctly, it seems that if you want NTLMv2 to work, you need to use the JRE connection services with a 1.6 JRE, and that if you want reliable connections with timeouts, you need to use the Apache client. Hopefully there is a better way than having a fallback of no Apache for NTLMv2 users. From http://hc.apache.org/httpclient-3.x/authentication.html#Known_limitations_and_problems Cannot authenticate with Microsoft IIS using NTLM authentication scheme NT Lan Manager (NTLM) authentication is a proprietary, closed challenge/response authentication protocol for Microsoft Windows. Only some details about NTLM protocol are available through reverse engineering. HttpClient provides limited support for what is known as NTLMv1, the early version of the NTLM protocol. HttpClient does not support NTLMv2 at all. Workaround: Disable NTLMv2.
Thanks Much Tim. I would like to ask Stefan to verify that we are talking about NTLMv2 here for his failure case. If this is a problem limited to NTLMv2 and apache httpclient 3.x there are a couple of things we can potentially do: 1) Allow/give user interface for people to disable the httpclient-based (i.e. and downshift to the JRE urlconnection) 2) Implement a new provider (e.g. httpcomponents-based one). Obviously this isn't guaranteed to deal with this issue (especially since httpcomponents is the apache follow on to httpclient), but there are other reasons to do this as well (use of nio). 3) Give people the option of using/not using httpclient-based provider at install time. 4) Do nothing (i.e. NTLMv2 proxy-with-auth users are out of luck). Others I haven't yet thought of...please add. Of the above, I have the most positive response to 1...which is doable but will take some UI work (which will need some support from p2 UI folks). We (ECF) are likely going to do 2 for other reasons so it may provide a solution for this, but I won't bet on it.
http://hc.apache.org/httpcomponents-client/ntlm.html Looks like HttpClient 4.0 does allow NTLM via a way you can leverage JCIFS. http://jcifs.samba.org/FAQ.html#ntlmv2 And JCIFS has partial support for NTLMv2 known as LMv2. I tend to lean towards option 1 as well, providing the option to the user to fall back to the JRE for connectivity, though in looking at the chart, this may only be applicable for certain JREs and is certainly only supported in particular Sun JREs or at least as documentation reveals. Is there a way to detect that we might be hitting NTLMv2 and attempt seamlessly a connection using the JRE, and only if that succeeds somehow present the option to the user of using the fallback connectivity? It seems less than ideal to have a user know if they are hitting one of the proxy servers that this might be an issue for.
Hi Scott, (in reply to comment 31) I guess it is NTLMv2! At least http://go.microsoft.com/fwlink/?linkid=14840&clcid=0x409 (Windows XP Security Guide) does not mention NTLMv3! HTH Tschüß, Stefan
(In reply to comment #32) > http://hc.apache.org/httpcomponents-client/ntlm.html > > Looks like HttpClient 4.0 does allow NTLM via a way you can leverage JCIFS. > > http://jcifs.samba.org/FAQ.html#ntlmv2 > > And JCIFS has partial support for NTLMv2 known as LMv2. > > I tend to lean towards option 1 as well, providing the option to the user to > fall back to the JRE for connectivity, though in looking at the chart, this may > only be applicable for certain JREs and is certainly only supported in > particular Sun JREs or at least as documentation reveals. > > Is there a way to detect that we might be hitting NTLMv2 and attempt seamlessly > a connection using the JRE, and only if that succeeds somehow present the > option to the user of using the fallback connectivity? It seems less than > ideal to have a user know if they are hitting one of the proxy servers that > this might be an issue for. I agree that it's less than ideal, but I don't know how/if NTLMv2 proxy with authentication can even be detected from within Eclipse (and when would it do the detecting?). If someone knows how this can be done and is willing to contribute work in this area then I would certainly be willing to include it, but I'm not willing to commit to it myself (NTLMv2 proxy auth detection...I'm willing to do the downshifting work...at least the stuff that I can do).
(In reply to comment #31) > 1) Allow/give user interface for people to disable the httpclient-based (i.e. > and downshift to the JRE urlconnection) > <stuff deleted> > Of the above, I have the most positive response to 1...which is doable but will > take some UI work (which will need some support from p2 UI folks). We (ECF) > are likely going to do 2 for other reasons so it may provide a solution for > this, but I won't bet on it. If we are talking about some kind of pref page that lets the user select from the different providers, then I would expect this to appear in the General->Network Connections pref page for Eclipse, and if that was not possible due to dependency issues, then I would expect to see a sub-page underneath that one. Ideally that would be an ECF-managed preference, I don't see why p2 would even have to know about it. I agree with Scott and others that the ideal thing is to downshift automatically (if anyone knows how to detect the problem case from within Eclipse), but even so, there will always be some case we haven't seen and therefore don't detect yet. Having a generic way to switch providers in the preferences seems like a good way for the user to help diagnose these things.
(In reply to comment #35) <stuff deleted> > If we are talking about some kind of pref page that lets the user select from > the different providers, then I would expect this to appear in the > General->Network Connections pref page for Eclipse, and if that was not > possible due to dependency issues, then I would expect to see a sub-page > underneath that one. Ideally that would be an ECF-managed preference, I don't > see why p2 would even have to know about it. So I think this could be done one of two ways: 1) We (ECF) add a IRetrieveFileTransferManager (or some such) API/service to the org.eclipse.ecf.filetransfer API. This manager service would be analogous to the org.eclipse.core.net.proxy.IProxyService, and allow a user interface to manage the use/selection of providers. Access to it could/would go on whatever preference page was deemed appropriate. 2) We (ECF) access a simple preference at bundle start time (for org.eclipse.ecf.filetransfer API bundle) that allows the users to defeat the ECF filetransfer provider prioritization scheme and force the use of JRE HttpURLConnection. Such a preference selection would likely have to be followed by an eclipse restart to take effect, as currently we're basing this on the registry (unless we fuss with the registry at runtime). 2 is easier, 1 is more general. I don't suspect that 1 is going to be needed other than for this specific case (need to turn off apache httpclient for NTLMv2 networks), although there may be other use cases (anyone?). > > I agree with Scott and others that the ideal thing is to downshift > automatically (if anyone knows how to detect the problem case from within > Eclipse), but even so, there will always be some case we haven't seen and > therefore don't detect yet. Having a generic way to switch providers in the > preferences seems like a good way for the user to help diagnose these things. Yes...but I'm not sure that 1 (above) is really needed...and of course more user interface that's not really needed (i.e. a full 'management' preferences page) is IMHO undesirable. So how do people feel about this? I'm inclined to go for something like 2 (e.g. just a preference that allows people to defeat any provider and force the usage of the JRE provider). And if someone know how/can provide code for detecting NTLMv2 with authentication then we can flip that switch automatically.
Hi Scott, I was very curious to see what happens within HttpClient when ECF tries to perform a request with it. So, I debugged the path down and it seems HttpClient does not like the UsernamePasswordCredentials within my environment (i.e. WinXp Client and Proxy requiring NTLM authentication). Using this credential causes an error. Than I simply replaced it with a NTCredentials in the HttpClientRetrieveFileTransfer.setupProxy(Proxy proxy) and the requests that formerly failed succeeded. However, there must be some other requests later in the process that do not go this way (had a breakpoint set) because while installing the regex-util I got the error: An error occurred while collecting items to be installed No repository found containing: org.apache.commons.httpclient/osgi.bundle/3.1.0.v20080605-1935 Any idea about that? May be ECF could have a look at the response header "Proxy-Authenticate" and check for an NTLM challenge (http://www.ietf.org/rfc/rfc2616.txt, Section 14.33 Proxy-Authenticate, Page 136) if that is possible and use NTCredentials when running on Win boxes. Another thing - when debugging a saw some code that selects a credential from a list of credentials via some "best match". This happens in HttpState.matchCredentials() which in turn uses AuthScope.match(). It seems to me that it is possible to set multiple credentials for a request. May be that is something that we could try. Tschüß, Stefan
(In reply to comment #37) > Hi Scott, > > I was very curious to see what happens within HttpClient when ECF tries to > perform a request with it. So, I debugged the path down and it seems HttpClient > does not like the UsernamePasswordCredentials within my environment (i.e. WinXp > Client and Proxy requiring NTLM authentication). Using this credential causes > an error. > Than I simply replaced it with a NTCredentials in the > HttpClientRetrieveFileTransfer.setupProxy(Proxy proxy) and the requests that > formerly failed succeeded. > However, there must be some other requests later in the process that do not go > this way (had a breakpoint set) because while installing the regex-util I got > the error: > An error occurred while collecting items to be installed > No repository found containing: > org.apache.commons.httpclient/osgi.bundle/3.1.0.v20080605-1935 > Any idea about that? No, unfortunately not. I have no idea why this would be happening. > > May be ECF could have a look at the response header "Proxy-Authenticate" and > check for an NTLM challenge (http://www.ietf.org/rfc/rfc2616.txt, Section 14.33 > Proxy-Authenticate, Page 136) if that is possible and use NTCredentials when > running on Win boxes. We certainly can/could look at the Proxy-Authenticate response header and call HttpState.setProxyCredentials with NTCredentials if this will fix the situation. > Another thing - when debugging a saw some code that selects a credential from a > list of credentials via some "best match". This happens in > HttpState.matchCredentials() which in turn uses AuthScope.match(). It seems to > me that it is possible to set multiple credentials for a request. May be that > is something that we could try. Yes, I suppose even before the request we could simply create both UsernamePasswordCredentials and NTCredentials and set them both. I don't know if this would have any negative effects on other proxies...any idea about that? If you want to add the following lines to HttpClientRetrieveFileTransfer line 550 we could try this out: NTCredentials ntCredentials = new NTCredentials(username,password,address.getHostName(),domain); httpClient.getState().setProxyCredentials(proxyAuthScope, ntCredentials); I don't know where you get the NT domain name from though. Is it somehow included in the proxy info? Scott
> > Hi Scott, > > > > I was very curious to see what happens within HttpClient when ECF tries to > > perform a request with it. So, I debugged the path down and it seems HttpClient > > does not like the UsernamePasswordCredentials within my environment (i.e. WinXp > > Client and Proxy requiring NTLM authentication). Using this credential causes > > an error. > > Than I simply replaced it with a NTCredentials in the > > HttpClientRetrieveFileTransfer.setupProxy(Proxy proxy) and the requests that > > formerly failed succeeded. > > However, there must be some other requests later in the process that do not go > > this way (had a breakpoint set) because while installing the regex-util I got > > the error: > > An error occurred while collecting items to be installed > > No repository found containing: > > org.apache.commons.httpclient/osgi.bundle/3.1.0.v20080605-1935 > > Any idea about that? > > > No, unfortunately not. I have no idea why this would be happening. > > > > > > May be ECF could have a look at the response header "Proxy-Authenticate" and > > check for an NTLM challenge (http://www.ietf.org/rfc/rfc2616.txt, Section 14.33 > > Proxy-Authenticate, Page 136) if that is possible and use NTCredentials when > > running on Win boxes. > > > We certainly can/could look at the Proxy-Authenticate response header and call > HttpState.setProxyCredentials with NTCredentials if this will fix the > situation. > > > Another thing - when debugging a saw some code that selects a credential from a > > list of credentials via some "best match". This happens in > > HttpState.matchCredentials() which in turn uses AuthScope.match(). It seems to > > me that it is possible to set multiple credentials for a request. May be that > > is something that we could try. > > Yes, I suppose even before the request we could simply create both > UsernamePasswordCredentials and NTCredentials and set them both. I don't know > if this would have any negative effects on other proxies...any idea about that? Unfortunatly, not! We need more help here. > > If you want to add the following lines to HttpClientRetrieveFileTransfer line > 550 we could try this out: > > NTCredentials ntCredentials = new > NTCredentials(username,password,address.getHostName(),domain); > httpClient.getState().setProxyCredentials(proxyAuthScope, ntCredentials); Yes, I created NTCredentials with String literals as parameters! > > I don't know where you get the NT domain name from though. Is it somehow > included in the proxy info? You can get it from the username. It is usually written as "domain\userid" if there is not a separate field in a dialog for the domain. To become more help I think it would be really great if you could create new repo contaning the changes above, i.e. create an additionally NTCredentials in case that NTLM is required for authentication and the client is a Win Box. What do you think about that? > > Scott Tschüß, Stefan
(In reply to comment #39) <stuff deleted> > You can get it from the username. It is usually written as "domain\userid" if > there is not a separate field in a dialog for the domain. > > To become more help I think it would be really great if you could create new > repo contaning the changes above, i.e. create an additionally NTCredentials in > case that NTLM is required for authentication and the client is a Win Box. > What do you think about that? Yes, I've created a branch called 'ntlm_support' for the org.eclipse.ecf.provider.filetransfer.httpclient project. Further, I've added the code to create NTCredentials and add to the set of credentials provided to the proxy upon get access. Would you please switch to this branch, and try out this new code and report results here? Thanks much.
(In reply to comment 40) I think we got it! The problem (comment 37) with the reg-util was a missing artifact. I tried other repos and I could update/install without a problem. I had a few breakpoints set to ensure that it was using the HttpClient. *grin* I would propose to additionally add the NTCredentials only when a proxy requests NTLM authentication. I am not sure if it is to restrictive to check also whether the client is a Win box. Are there any other operating system using NTLM authentication? To get more feedback from the community it would be good to have a repo as you described in comment 12 and on http://wiki.eclipse.org/ECF_Apache_Httpclient-Based_Provider This could than be announced on the p2 mailing list/news group and on the new enterprise-users newsgroup http://www.eclipse.org/newsportal/thread.php?group=eclipse.enterprise-users Tschüß, Stefan
Guys, can you open a new bug for 'http client doesn't work with NTLM authentication' and move the discussion for people who care to that bug? This bug is marked as 'ship httpclient' and it's fixed. It doesn't need a lengthy debate about what does/doesn't work in this bug.
(In reply to comment #42) > Guys, can you open a new bug for 'http client doesn't work with NTLM > authentication' and move the discussion for people who care to that bug? This > bug is marked as 'ship httpclient' and it's fixed. It doesn't need a lengthy > debate about what does/doesn't work in this bug. > OK Alex. I've created bug #252002 to track this further and added myself, Stephan, Tim, and Susan to the bug. Others that wish to follow the further discussion please add yourselves to that bug.
