Community
Participate
Working Groups
As per the discussion starting with comment 23 on bug 220240 https://bugs.eclipse.org/bugs/show_bug.cgi?id=220240#c23 The httpclient 3.0.1-based provider has some difficulties with NTLMv2 provider (see discussion on 220240). This bug is to track further work on NTLMv2 support.
In response to https://bugs.eclipse.org/bugs/show_bug.cgi?id=220240#c41 >(In reply to comment 40) > >I think we got it! >The problem (comment 37) with the reg-util was a missing artifact. I tried >other repos and I could update/install without a problem. I had a few >breakpoints set to ensure that it was using the HttpClient. *grin* >I would propose to additionally add the NTCredentials only when a proxy >requests NTLM authentication. Would you like to make this change to the code on the ntlm_support branch and try it out? I believe all that would be necessary is to move the 6 lines of code currently starting on line 550 to the appropriate point *after* the initial get request (i.e. so that the response can be checked for Proxy-Authentication of appropriate type). >I am not sure if it is to restrictive to check >also whether the client is a Win box. Are there any other operating system >using NTLM authentication? I don't know. I'm not at all certain that we should have operating-system specific code to do this...actually, I'm hoping that we can try things with these additional credentials and it won't negatively effect other usage...that way we would simply always tack on the NT credentials along with the usernamepassword credentials for proxy. >To get more feedback from the community it would be good to have a repo as you >described in comment 12 and on >http://wiki.eclipse.org/ECF_Apache_Httpclient-Based_Provider >This could than be announced on the p2 mailing list/news group and on the new >enterprise-users newsgroup >http://www.eclipse.org/newsportal/thread.php?group=eclipse.enterprise-users I'm willing to do this, but perhaps it would be better to just include things with next week's platform integration build and ask people to test using that...as getting many people to download and install an add on just to test the filetransfer provider hasn't worked very well IMHO.
(In reply to comment #1) > In response to https://bugs.eclipse.org/bugs/show_bug.cgi?id=220240#c41 > > >(In reply to comment 40) > > > >I think we got it! > >The problem (comment 37) with the reg-util was a missing artifact. I tried > >other repos and I could update/install without a problem. I had a few > >breakpoints set to ensure that it was using the HttpClient. *grin* > >I would propose to additionally add the NTCredentials only when a proxy > >requests NTLM authentication. > > > Would you like to make this change to the code on the ntlm_support branch and > try it out? Stefan, would you be able to make this change and test it on your NTLM proxy? (only adding NTCredentials when a proxy requests NTLM authentication). I'm frankly not even sure how such a request is made, and I have absolutely no means myself to test it with an NTLM proxy. So I need to depend upon you/yours. BTW, just for people's awareness. The httpclient-based provider is now in Eclipse 3.5M3, so for those that can try out various proxy scenarios against the new httpclient provider I would kindly request that you do so. Obviously we don't have the NTLM support fully implemented yet, so the M3 code does not have the solution we are working on here.
Stephan...ping. Would you be able to make the changes and test as described on comment #2? And then we can/could commit to ECF head in time for equinox M4 (Dec 12)? Note there is already a branch for this work in the org.eclipse.ecf.provider.filetransfer.httpclient project in /cvsroot/rt/org.eclipse.ecf/providers/bundles thanks
Hi Scott, I created patch (against HEAD) that creates the NTLM credentials only when needed. I did not find a solution to do that in org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientRetrieveFileTransfer.setupProxy(Proxy) However while debugging through HttpClient following what happens in request I found the CredentialsProvider class. I wrote a simple one and used it. With that it is now possible to pass the proxy that required NTLM authentication. See also my comments in the provided patch. Tschüß, Stefan
Created attachment 119380 [details] Add NTLM proxy authentication
Thanks Stephan. Passed all local tests. Committed to HEAD (ECF 3.0), 2.1, and 2.0 streams. Stephan please try out the latest from HEAD in your environment and verify that NTLMv2 still works out ok. If others could try this out in their local environments (with various proxies, etc) to check for regression it would be greatly appreciated. Marked contribution for IP log. Thanks again Stephan. Marking this bug is fixed. If others find issues, please reopen.
oops forgot resolve.
Scott, I think I have been to rash. While my testing I never went through a full install. I was satisfied with p2 getting the meta data. While debugging this, I could see that the NTCredentials were used and that was enough for me. However, now I tried to do a full install. This failed on my machine behind the proxy but succeeded in my machine with direct connection (They both use the same Eclipse/Workspace/.. - its on a memory stick). When installing (update site: http://download.eclipse.org/eclipse/updates/3.5-I-builds) it fails with: An error occurred while collecting items to be installed No repository found containing: ch.ethz.iks.r_osgi.remote/osgi.bundle/1.0.0.RC2_v20080623-0046 No repository found containing: ch.ethz.iks.slp/osgi.bundle/1.0.0.RC4_v200806091030 No repository found containing: javax.xml/osgi.bundle/1.3.4.v200806030440 No repository found containing: org.apache.ant.source/osgi.bundle/1.7.0.v200803061910 .. and about 320 more While the install phase neither openStreams() nor openStreamsForResume get called! Any ideas?
(In reply to comment #8) > Scott, I think I have been to rash. While my testing I never went through a > full install. I was satisfied with p2 getting the meta data. While debugging > this, I could see that the NTCredentials were used and that was enough for me. > > However, now I tried to do a full install. This failed on my machine behind the > proxy but succeeded in my machine with direct connection (They both use the > same Eclipse/Workspace/.. - its on a memory stick). > When installing (update site: > http://download.eclipse.org/eclipse/updates/3.5-I-builds) it fails with: > An error occurred while collecting items to be installed > No repository found containing: > ch.ethz.iks.r_osgi.remote/osgi.bundle/1.0.0.RC2_v20080623-0046 > No repository found containing: > ch.ethz.iks.slp/osgi.bundle/1.0.0.RC4_v200806091030 > No repository found containing: javax.xml/osgi.bundle/1.3.4.v200806030440 > No repository found containing: > org.apache.ant.source/osgi.bundle/1.7.0.v200803061910 > .. and about 320 more > > While the install phase neither openStreams() nor openStreamsForResume get > called! > Any ideas? It's possible that the Eclipse update is using the 'browse' (i.e. to find the artifact repository), and that is not succeeding because it's not using the httpclient provider (rather it's using the URLConnection). Could you try to find the place in the ECF and p2 code where this failure is initiated? It would/will be easy enough to add an httpclient-based browse handler...assuming this is what's wrong...but I want to verify that this is the case in your situation before I go down that road.
Unfortunately we are not going to make M4 with this work, but Stephan if you can help me figure out answers to questions in comment #9 (i.e. if/where P2 is using the ECF filetransfer 'browse' capability...which I think is what is failing as described in comment #8...because the httpclient doesn't implement a separate 'browse' handler (but will in order to support NTLMv2). If this is true/verified then what I will do is add an implementation of the file transfer 'browse' capability based upon the httpclient code, and do the same thing for NTLMv2 support that we are doing with the retrieve capability. So, when you can, please advise if this is the failure (i.e. p2 is doing a filetransfer browse operation with the org.eclipse.ecf.provider rather than the httpclient provider), and I will plan to add an httpclient-based browse capability. Thanks.
As far as I have seen is that the ´browsing´ is done with the URLConnection (as you said), but this does not fail! I have to investigate further what the cause for the failure is. Scott, I am very sorry that my available time is so limited.
(In reply to comment #11) > As far as I have seen is that the ´browsing´ is done with the URLConnection (as > you said), but this does not fail! > I have to investigate further what the cause for the failure is. Scott, I am > very sorry that my available time is so limited. > np Stephan...just trying to get this bug finished. Please just LMK here on this bug when you have some more time to investigate the NTLM failure.
(In reply to comment #12) > (In reply to comment #11) > > As far as I have seen is that the ´browsing´ is done with the URLConnection (as > > you said), but this does not fail! > > I have to investigate further what the cause for the failure is. Scott, I am > > very sorry that my available time is so limited. > > > > np Stephan...just trying to get this bug finished. Please just LMK here on > this bug when you have some more time to investigate the NTLM failure. > Stephan: as a side effect of bug #256684, I've added using the httpclient 3.0.1 provider for using the ECF file browsing. Previously, the httpclient provider was being used for only for retrieve, and not browsing (which was actually being provided by the URLConnection based provider). The upshot for this bug is that it's now quite possible that a p2-based install will work with your NTLMv2 proxy, as now the same proxy access code added in comment #6 (for retrieve) was now also added for browse...and p2 does use both the ECF browse and retrieve code (I suspect that was what was not working for comment #8). The 1.21.2009 platform integration build will contain this new code. If you have last week's platform I-build you can install the update directly by going to this ECF update site (you will need some of EMF though to install): http://www.eclipse.org/external/rt/ecf/integration/v20090119-0800/update Stephan I would appreciate it if you would test this against your NTLMv2 proxy when you can. Thanks.
Scott, I took the eclipse-SDK-I20090114-1322-win32 and tried to update ECF from http://www.eclipse.org/external/rt/ecf/integration/v20090119-0800/update I got the following error while updating: An error occurred while collecting items to be installed session context was:(profile=SDKProfile, phase=, operand=, action=). No repository found containing: osgi.bundle,org.eclipse.emf.common,2.4.0.v200808251517 No repository found containing: osgi.bundle,org.eclipse.emf.common.ui,2.4.0.v200808251517 No repository found containing: osgi.bundle,org.eclipse.emf.ecore,2.4.1.v200808251517 No repository found containing: osgi.bundle,org.eclipse.emf.ecore.change,2.4.0.v200808251517 No repository found containing: osgi.bundle,org.eclipse.emf.edit,2.4.1.v200808251517 No repository found containing: osgi.bundle,org.eclipse.emf.edit.ui,2.4.1.v200808251517 Retrying the install (same installation) caused this: java.lang.IllegalStateException: The profile is currently in use. at org.eclipse.equinox.internal.p2.engine.SimpleProfileRegistry.lockProfile(SimpleProfileRegistry.java:655) at org.eclipse.equinox.internal.provisional.p2.engine.Engine.perform(Engine.java:51) at org.eclipse.equinox.internal.provisional.p2.ui.operations.ProvisioningUtil.performProvisioningPlan(ProvisioningUtil.java:274) at org.eclipse.equinox.internal.provisional.p2.ui.operations.ProfileModificationOperation.doExecute(ProfileModificationOperation.java:59) at org.eclipse.equinox.internal.provisional.p2.ui.operations.ProvisioningOperation.execute(ProvisioningOperation.java:37) at org.eclipse.equinox.internal.provisional.p2.ui.ProvisioningOperationRunner$1.run(ProvisioningOperationRunner.java:80) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) Any idea what I did wrong?
(In reply to comment #14) > Scott, > > I took the eclipse-SDK-I20090114-1322-win32 and tried to update ECF from > http://www.eclipse.org/external/rt/ecf/integration/v20090119-0800/update > > I got the following error while updating: > An error occurred while collecting items to be installed > session context was:(profile=SDKProfile, phase=, operand=, action=). > No repository found containing: > osgi.bundle,org.eclipse.emf.common,2.4.0.v200808251517 > No repository found containing: > osgi.bundle,org.eclipse.emf.common.ui,2.4.0.v200808251517 > No repository found containing: > osgi.bundle,org.eclipse.emf.ecore,2.4.1.v200808251517 > No repository found containing: > osgi.bundle,org.eclipse.emf.ecore.change,2.4.0.v200808251517 > No repository found containing: > osgi.bundle,org.eclipse.emf.edit,2.4.1.v200808251517 > No repository found containing: > osgi.bundle,org.eclipse.emf.edit.ui,2.4.1.v200808251517 > > Retrying the install (same installation) caused this: > java.lang.IllegalStateException: The profile is currently in use. > at > org.eclipse.equinox.internal.p2.engine.SimpleProfileRegistry.lockProfile(SimpleProfileRegistry.java:655) > at > org.eclipse.equinox.internal.provisional.p2.engine.Engine.perform(Engine.java:51) > at > org.eclipse.equinox.internal.provisional.p2.ui.operations.ProvisioningUtil.performProvisioningPlan(ProvisioningUtil.java:274) > at > org.eclipse.equinox.internal.provisional.p2.ui.operations.ProfileModificationOperation.doExecute(ProfileModificationOperation.java:59) > at > org.eclipse.equinox.internal.provisional.p2.ui.operations.ProvisioningOperation.execute(ProvisioningOperation.java:37) > at > org.eclipse.equinox.internal.provisional.p2.ui.ProvisioningOperationRunner$1.run(ProvisioningOperationRunner.java:80) > at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) > > Any idea what I did wrong? > The newer discovery view requires a bunch of EMF bundles to be installed. But I guess there's either something wrong with our features or p2 screwed up. Anyway, does it help if you install the EMF SDK manually?
When installing EMF manually I get the ECF update when connected directly to the internet. This is just to prepare my test installation of Eclipse. With this installation I tried to install: http://regex-util.sourceforge.net/update/ with the internet proxy. And it failed (it did not without the proxy). It takes a very long time (several minutes > 5) until it gives up. Unfortunately there was no error message, I could just cancel the install dialog. Without the proxy I see the category and have to accept license, .. Any hints how to proceed?
(In reply to comment #14) Hi Stefan, The first problem is indeed due to a dependency on EMF...so if you could install EMF prior it will go away. The second problem seems to be p2 bug #261573.
Hi Scott, Yes, Markus gave me the hint to install EMF manually. With that I could get the Eclipse installation with the ECF update that you mentioned in comment 13. When I use this Eclipse installation behind the proxy I am not able to connect to update sites. It takes a lot of time (>3-5 min.) until it gives up silently - no error message, I just can hit cancel in the Install Software dialog.
(In reply to comment #18) > Hi Scott, > > Yes, Markus gave me the hint to install EMF manually. With that I could get the > Eclipse installation with the ECF update that you mentioned in comment 13. > > When I use this Eclipse installation behind the proxy I am not able to connect > to update sites. It takes a lot of time (>3-5 min.) until it gives up silently > - no error message, I just can hit cancel in the Install Software dialog. > That's very strange. Anything in the log? You can now also use tracing code when in debugger (or starting Eclipse in debug mode). So it might be a good idea to turn tracing on for the org.eclipse.ecf.provider.filetransfer.httpclient bundle. Turn on all the options in the org.eclipse.ecf.provider.filetransfer.httpclient/.options bundle and see what this produces as output when you do an install behind the NTLM proxy. Here is the .options file for org.eclipse.ecf.provider.filetransfer.httpclient if you want to run the existing Eclipse instance in debug mode: # Debugging options for the org.eclipse.ecf.provider.filetransfer.httpclient plug-in # Turn on general debugging for the org.eclipse.ecf.provider.filetransfer.httpclient plug-in org.eclipse.ecf.provider.filetransfer.httpclient/debug=true org.eclipse.ecf.provider.filetransfer.httpclient/debug/filter = * org.eclipse.ecf.provider.filetransfer.httpclient/debug/flag = true # Trace when exceptions are caught org.eclipse.ecf.provider.filetransfer.httpclient/debug/exceptions/catching=true # Trace when exceptions are thrown org.eclipse.ecf.provider.filetransfer.httpclient/debug/exceptions/throwing=true # Trace when methods are entered org.eclipse.ecf.provider.filetransfer.httpclient/debug/methods/entering=true # Trace when methods are exited org.eclipse.ecf.provider.filetransfer.httpclient/debug/methods/exiting=true
Hi Scott, I was using Eclipse build I20090125-2000 containing the required ECF bundles. I was debugging the download/install of my pet bundle http://regex-util.sourceforge.net/update. And it did not work as expected (because of my previous attempts). What I found out is that the HttpClientFileSystemBrowser.setupProxy(Proxy) does not set the required NTCredentials as the HttpClientRetrieveFileTransfer.createNTLMCredentials(Proxy). After adding something similar it works. At least the download. After applying the downloaded bundles I see the following stacktrace on the console: java.lang.NullPointerException at org.eclipse.core.internal.resources.CharsetDeltaJob.shutdown(CharsetDeltaJob.java:182) at org.eclipse.core.internal.resources.CharsetManager.shutdown(CharsetManager.java:294) at org.eclipse.core.internal.resources.Workspace.shutdown(Workspace.java:1896) at org.eclipse.core.internal.resources.Workspace.close(Workspace.java:431) at org.eclipse.core.resources.ResourcesPlugin.stop(ResourcesPlugin.java:354) at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:845) at java.security.AccessController.doPrivileged(Native Method) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:841) at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:474) at org.eclipse.osgi.framework.internal.core.AbstractBundle.uninstallWorker(AbstractBundle.java:870) at org.eclipse.osgi.framework.internal.core.AbstractBundle.uninstall(AbstractBundle.java:850) at org.eclipse.equinox.internal.simpleconfigurator.ConfigApplier.installBundles(ConfigApplier.java:200) at org.eclipse.equinox.internal.simpleconfigurator.ConfigApplier.install(ConfigApplier.java:74) at org.eclipse.equinox.internal.simpleconfigurator.SimpleConfiguratorImpl.applyConfiguration(SimpleConfiguratorImpl.java:118) at org.eclipse.equinox.internal.simpleconfigurator.SimpleConfiguratorImpl.applyConfiguration(SimpleConfiguratorImpl.java:132) at org.eclipse.equinox.internal.provisional.p2.ui.ProvisioningOperationRunner$3.run(ProvisioningOperationRunner.java:151) at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35) at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:133) at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3852) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3473) at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2388) at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2352) at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2204) at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:499) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:333) at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:492) at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149) at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:113) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:194) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:368) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:550) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:505) at org.eclipse.equinox.launcher.Main.run(Main.java:1264) at org.eclipse.equinox.launcher.Main.main(Main.java:1240) java.lang.NullPointerException at org.eclipse.core.internal.resources.ContentDescriptionManager.shutdown(ContentDescriptionManager.java:474) at org.eclipse.core.internal.resources.Workspace.shutdown(Workspace.java:1896) at org.eclipse.core.internal.resources.Workspace.close(Workspace.java:431) at org.eclipse.core.resources.ResourcesPlugin.stop(ResourcesPlugin.java:354) at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:845) at java.security.AccessController.doPrivileged(Native Method) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:841) at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:474) at org.eclipse.osgi.framework.internal.core.AbstractBundle.uninstallWorker(AbstractBundle.java:870) at org.eclipse.osgi.framework.internal.core.AbstractBundle.uninstall(AbstractBundle.java:850) at org.eclipse.equinox.internal.simpleconfigurator.ConfigApplier.installBundles(ConfigApplier.java:200) at org.eclipse.equinox.internal.simpleconfigurator.ConfigApplier.install(ConfigApplier.java:74) at org.eclipse.equinox.internal.simpleconfigurator.SimpleConfiguratorImpl.applyConfiguration(SimpleConfiguratorImpl.java:118) at org.eclipse.equinox.internal.simpleconfigurator.SimpleConfiguratorImpl.applyConfiguration(SimpleConfiguratorImpl.java:132) at org.eclipse.equinox.internal.provisional.p2.ui.ProvisioningOperationRunner$3.run(ProvisioningOperationRunner.java:151) at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35) at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:133) at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3852) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3473) at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2388) at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2352) at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2204) at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:499) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:333) at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:492) at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149) at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:113) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:194) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:368) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:550) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:505) at org.eclipse.equinox.launcher.Main.run(Main.java:1264) at org.eclipse.equinox.launcher.Main.main(Main.java:1240) If there is anything I can do for you, let me know.
Hi Stephan, So I understand...it looks like the equivalent of HttpClientRetrieveFileTransfer.createNTLMCredentials(Proxy) has to be added to the HttpClientFileSystemBrowser.setupProxy(Proxy)...and if this is done then the download works for you on your NTLMv2 network. Is that correct? As for the stack trace...I don't immediately understand it's relationship to this problem...but perhaps it's some side effect. But is my interpretation above correct? Thanks. (In reply to comment #20) > Hi Scott, > > I was using Eclipse build I20090125-2000 containing the required ECF bundles. I > was debugging the download/install of my pet bundle > http://regex-util.sourceforge.net/update. And it did not work as expected > (because of my previous attempts). > What I found out is that the HttpClientFileSystemBrowser.setupProxy(Proxy) does > not set the required NTCredentials as the > HttpClientRetrieveFileTransfer.createNTLMCredentials(Proxy). > After adding something similar it works. At least the download. After applying > the downloaded bundles I see the following stacktrace on the console: > > java.lang.NullPointerException > at > org.eclipse.core.internal.resources.CharsetDeltaJob.shutdown(CharsetDeltaJob.java:182) > at > org.eclipse.core.internal.resources.CharsetManager.shutdown(CharsetManager.java:294) > at > org.eclipse.core.internal.resources.Workspace.shutdown(Workspace.java:1896) > at > org.eclipse.core.internal.resources.Workspace.close(Workspace.java:431) > at > org.eclipse.core.resources.ResourcesPlugin.stop(ResourcesPlugin.java:354) > at > org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:845) > at java.security.AccessController.doPrivileged(Native Method) > at > org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:841) > at > org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:474) > at > org.eclipse.osgi.framework.internal.core.AbstractBundle.uninstallWorker(AbstractBundle.java:870) > at > org.eclipse.osgi.framework.internal.core.AbstractBundle.uninstall(AbstractBundle.java:850) > at > org.eclipse.equinox.internal.simpleconfigurator.ConfigApplier.installBundles(ConfigApplier.java:200) > at > org.eclipse.equinox.internal.simpleconfigurator.ConfigApplier.install(ConfigApplier.java:74) > at > org.eclipse.equinox.internal.simpleconfigurator.SimpleConfiguratorImpl.applyConfiguration(SimpleConfiguratorImpl.java:118) > at > org.eclipse.equinox.internal.simpleconfigurator.SimpleConfiguratorImpl.applyConfiguration(SimpleConfiguratorImpl.java:132) > at > org.eclipse.equinox.internal.provisional.p2.ui.ProvisioningOperationRunner$3.run(ProvisioningOperationRunner.java:151) > at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35) > at > org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:133) > at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3852) > at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3473) > at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2388) > at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2352) > at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2204) > at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:499) > at > org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:333) > at > org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:492) > at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149) > at > org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:113) > at > org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:194) > at > org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) > at > org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) > at > org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:368) > at > org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:550) > at org.eclipse.equinox.launcher.Main.basicRun(Main.java:505) > at org.eclipse.equinox.launcher.Main.run(Main.java:1264) > at org.eclipse.equinox.launcher.Main.main(Main.java:1240) > java.lang.NullPointerException > at > org.eclipse.core.internal.resources.ContentDescriptionManager.shutdown(ContentDescriptionManager.java:474) > at > org.eclipse.core.internal.resources.Workspace.shutdown(Workspace.java:1896) > at > org.eclipse.core.internal.resources.Workspace.close(Workspace.java:431) > at > org.eclipse.core.resources.ResourcesPlugin.stop(ResourcesPlugin.java:354) > at > org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:845) > at java.security.AccessController.doPrivileged(Native Method) > at > org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:841) > at > org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:474) > at > org.eclipse.osgi.framework.internal.core.AbstractBundle.uninstallWorker(AbstractBundle.java:870) > at > org.eclipse.osgi.framework.internal.core.AbstractBundle.uninstall(AbstractBundle.java:850) > at > org.eclipse.equinox.internal.simpleconfigurator.ConfigApplier.installBundles(ConfigApplier.java:200) > at > org.eclipse.equinox.internal.simpleconfigurator.ConfigApplier.install(ConfigApplier.java:74) > at > org.eclipse.equinox.internal.simpleconfigurator.SimpleConfiguratorImpl.applyConfiguration(SimpleConfiguratorImpl.java:118) > at > org.eclipse.equinox.internal.simpleconfigurator.SimpleConfiguratorImpl.applyConfiguration(SimpleConfiguratorImpl.java:132) > at > org.eclipse.equinox.internal.provisional.p2.ui.ProvisioningOperationRunner$3.run(ProvisioningOperationRunner.java:151) > at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35) > at > org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:133) > at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3852) > at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3473) > at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2388) > at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2352) > at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2204) > at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:499) > at > org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:333) > at > org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:492) > at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149) > at > org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:113) > at > org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:194) > at > org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) > at > org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) > at > org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:368) > at > org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:550) > at org.eclipse.equinox.launcher.Main.basicRun(Main.java:505) > at org.eclipse.equinox.launcher.Main.run(Main.java:1264) > at org.eclipse.equinox.launcher.Main.main(Main.java:1240) > > If there is anything I can do for you, let me know. >
Scott, Yes, that is right. If you have a new version, I can test it. Stefan
(In reply to comment #22) > Scott, > > Yes, that is right. > If you have a new version, I can test it. > > Stefan > Hi Stefan...I've committed a change to HEAD (in httpclient provider) that I believe should let the HttpClientFileSystemBrowser operate correctly in the NTLM environment...that is, it now does the same thing that HttpClientRetrieveFileTransfer does in terms of proxy setup and credentials. If you would please test and let me know here or via direct Skype connection I would appreciate it. AFAICT from code inspection it should be doing the same thing for browse (in terms of proxy setup/credentials handling) as it is doing with the retrieve. Unfortunately I can't verify this myself, so have to depend upon you/your network to test. Thanks.
Re comment #20: In this situation you are dynamically installing a bundle that is not dynamic-enabled (org.eclipse.core.resources). You're welcome to enter a bug but this is a known limitation.
Hi Scott, For some reason the version v20090119-0800 did not work when I tested it (comment 20). However, I rerun the tests again with Eclipse build I20090125-2000 and I could pass the proxy!! What was different? When I rerun the tests I was lazy and entered only proxy information (proxy host, credentials, ..) for http (Eclipse provider). With this I can pass the proxy. If add the proxy information for https, it still works, but if I add proxy information for SOCKS it doesn´t!! What I found out is that the ECFCredentialsProvider (defined in HttpClientFileSystemBrowser.runRequest()) is not called from HttpClient when a SOCKS proxy is defined - it is called if no SOCKS proxy is defined. In any case the proxy that should be used (http proxy) is set correctly by AbstractFileSystemBrowser.setupProxies(). To summarize: It works unless SOCKS is defined!
(In reply to comment #25) > Hi Scott, > For some reason the version v20090119-0800 did not work when I tested it > (comment 20). > However, I rerun the tests again with Eclipse build I20090125-2000 and I could > pass the proxy!! > What was different? Well, there was one test (about the proxyService enablement) that was not being done correctly for the browse part (i.e. line 189 in AbstractFileSystemBrowser). When I rerun the tests I was lazy and entered only proxy > information (proxy host, credentials, ..) for http (Eclipse provider). >With > this I can pass the proxy. If add the proxy information for https, it still > works, but if I add proxy information for SOCKS it doesn´t!! So you have two proxies on this network?...i.e. an http proxy and a socks proxy? > What I found out is that the ECFCredentialsProvider (defined in > HttpClientFileSystemBrowser.runRequest()) is not called from HttpClient when a > SOCKS proxy is defined - it is called if no SOCKS proxy is defined. This is interesting...the ECFCredentialsProvider is set with this line (86 in HttpClientFileSystemBrowser: getMethod.getParams().setParameter(CredentialsProvider.PROVIDER, new ECFCredentialsProvider()); So the apache httpclient code makes the decision about what credential providers to use when contacting the proxy. > In any case the proxy that should be used (http proxy) is set correctly by > AbstractFileSystemBrowser.setupProxies(). > > To summarize: It works unless SOCKS is defined! I suppose the explanation for this is as follows: The Socks proxy support is provided via the JREProxyHelper class...which uses the JRE-based socks proxy support...rather than rather than the apache httpclient call: httpClient.getState().setProxyCredentials(proxyAuthScope, credentials); i.e. see line 205 in HttpClientFileSystemBrowser (setupProxy). As you can see from setupProxy(), if the proxy type is HTTP/HTTPS, then the setProxyCredentials call is made, if socks it calls the proxyHelper.setupProxy(proxy) (using the JRE socks proxy method). The thing that puzzles me is why you are getting a conflict with the socks proxy set...as it should select the http proxy over the socks proxy (in AbstractFileSystemBrowser.selectProxyFromProxies (line 160). I'm naturally very glad to hear that it works (without the socks proxy), but just to be sure I would like to try to track down why the existence of the socks proxy seems to create issues for you...and once we get this addressed then we'll declare full victory over this issue :). Thanks.
Hi Scott, > So you have two proxies on this network?...i.e. an http proxy and a socks > proxy? Our proxy is a http/s proxy not a socks proxy. > I suppose the explanation for this is as follows: The Socks proxy support is > provided via the JREProxyHelper class...which uses the JRE-based socks proxy > support...rather than rather than the apache httpclient call: > > httpClient.getState().setProxyCredentials(proxyAuthScope, credentials); > > i.e. see line 205 in HttpClientFileSystemBrowser (setupProxy). > > As you can see from setupProxy(), if the proxy type is HTTP/HTTPS, then the > setProxyCredentials call is made, if socks it calls the > proxyHelper.setupProxy(proxy) (using the JRE socks proxy method). > > The thing that puzzles me is why you are getting a conflict with the socks > proxy set...as it should select the http proxy over the socks proxy (in > AbstractFileSystemBrowser.selectProxyFromProxies (line 160). As said in comment 20 in both cases (network settings: eclipse provider with http only/http plus socks) the proxy is correctly set, i.e. http proxy. However, when socks is set the connection attempt hanged for 2 days until I closed it (no timeout)!! I also tried to debug down into the guts (javax.net.*) but I have no sources.
(In reply to comment #27) > Hi Scott, > > > So you have two proxies on this network?...i.e. an http proxy and a socks > > proxy? > > Our proxy is a http/s proxy not a socks proxy. > > > I suppose the explanation for this is as follows: The Socks proxy support is > > provided via the JREProxyHelper class...which uses the JRE-based socks proxy > > support...rather than rather than the apache httpclient call: > > > > httpClient.getState().setProxyCredentials(proxyAuthScope, credentials); > > > > i.e. see line 205 in HttpClientFileSystemBrowser (setupProxy). > > > > As you can see from setupProxy(), if the proxy type is HTTP/HTTPS, then the > > setProxyCredentials call is made, if socks it calls the > > proxyHelper.setupProxy(proxy) (using the JRE socks proxy method). > > > > The thing that puzzles me is why you are getting a conflict with the socks > > proxy set...as it should select the http proxy over the socks proxy (in > > AbstractFileSystemBrowser.selectProxyFromProxies (line 160). > > As said in comment 20 in both cases (network settings: eclipse provider with > http only/http plus socks) the proxy is correctly set, i.e. http proxy. > > However, when socks is set the connection attempt hanged for 2 days until I > closed it (no timeout)!! That's bizarre. > > I also tried to debug down into the guts (javax.net.*) but I have no sources. > Hi Stefan, Why don't you try just debugging into the proxy selection code in AbstractFileSystemBrowser...e.g. with test code...to make sure the proxy setting is happening correctly. If it is (i.e. set to http proxy) then my thesis would be some strangeness in proxy credentials handling with httpclient. If you have the source for httpclient (i.e. from Orbit)...which I believe you already do...then you could debug into that if you wish. Also cc'ing Henrich in case he's encountered something like this in working with httpclient.
Hi Scott, Proxy selection is ok. In any case http proxy is selected. I debugged into ECF/HttpClient (not only once!!). In case of HTTP and SOCKS proxies defined (Eclipse provider) I found out, that the first request (without authentication) ends up in SocketInputStream.socketRead0() never coming back although the timeout value is 30000. If SOCKS proxy is not defined it also tries first without authentication, fails and than retries with authentication first using UsernamePasswordCredentials (fails too) and than NTCredentials which succeeds. This is very strange. Could it be that setting the proxies (preferences) has any side effects!?
(In reply to comment #29) > Hi Scott, > > Proxy selection is ok. In any case http proxy is selected. > I debugged into ECF/HttpClient (not only once!!). > In case of HTTP and SOCKS proxies defined (Eclipse provider) I found out, that > the first request (without authentication) ends up in > SocketInputStream.socketRead0() never coming back although the timeout value is > 30000. > If SOCKS proxy is not defined it also tries first without authentication, fails > and than retries with authentication first using UsernamePasswordCredentials > (fails too) and than NTCredentials which succeeds. > > This is very strange. Could it be that setting the proxies (preferences) has > any side effects!? > Stephan, could you retry your tests with the recent/latest builds of platform and ECF filetransfer? I would like to address this before Galileo if at all possible, but I can't reproduce an NTLMv2 environment for testing myself.
Hi Scott, Sorry for being so late but next week my vacation (California) starts and we also had a lot of things to do. However, I took eclipse-SDK-I20090429-1300 and tried to "Check for Updates ...". With manual proxy settings and credentials set I was able to perform the check. With native proxy settings checking I had to stop Eclipse because checking was hanging for quite a long time (several minutes). I was never asked for entering credentials. With native proxy settings and provided credentials (double clicking the line for the specific protocol - but the edit button is disabled!) Eclipse was also not returning from checking. Unfortunately I had no time for debugging into the guts.
(In reply to comment #31) > Hi Scott, > > Sorry for being so late but next week my vacation (California) starts and we > also had a lot of things to do. > However, I took eclipse-SDK-I20090429-1300 and tried to "Check for Updates > ...". > With manual proxy settings and credentials set I was able to perform the check. > With native proxy settings checking I had to stop Eclipse because checking was > hanging for quite a long time (several minutes). I was never asked for entering > credentials. > With native proxy settings and provided credentials (double clicking the line > for the specific protocol - but the edit button is disabled!) Eclipse was also > not returning from checking. > Unfortunately I had no time for debugging into the guts. > Hi Stefan. Thanks for the testing/input, but short of further work from you on what's going on on NTLMv2 networks this bug can't really be addressed by me or the resources I have access to...so I'm assigning to you.
Stefan, we would really need your help on this. Could you please set aside some time for this?
Sorry, I jump in for Stefan (for the reply at least). Stefan has 4 weeks holidays (well deserved) until end of may. Actually he noted that in his last entry didnt he ? He didnt mention the duration through.
Christian, would anybody else from your team help on this issue? You seem to be the only guys around that have access to those funny NTLM2 proxies.
Hi Pascal (ok....if you fix my pde-build :-) just kidding) ok post some steps that we should do to test whatever you want to test and we set some time aside on monday. Post them here or mail me. christian
(In reply to comment #36) > Hi Pascal (ok....if you fix my pde-build :-) just kidding) > > ok post some steps that we should do to test whatever you want to test and we > set some time aside on monday. Post them here or mail me. > > christian > Hi Christian. The main thing to try: use of a recent build of the filetransfer httpclient provider (which is used by p2) to access http/https resources via your NTLMv2 proxy...with username and password. You can see from Stefan's postings earlier on this bug that he actually was able to get through an NTLMv2 proxy with username/password at some point (e.g. comment #25), but it wasn't clear to me what/why subsequent tests failed. A complicating factor in this is the use of the proxy API, which has changed some since this bug was created...and as per comment #31 it didn't seem to be working as expected under some conditions (e.g. Native vs. Direct). But this may very well have changed since Stefan's last tests/posting in comment #31 so will need to be tested again. To summarize testing procedure: 1) Use the latest of both Eclipse and ECF filetransfer (probably the 20090515 build should be used). 2) Get the following plugins (at least) in your workspace so that if necessary you can debug into them (under /cvsroot/rt): org.eclipse.ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient org.eclipse.ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient.ssl org.eclipse.ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer The following test bundles will be helpful also org.eclipse.ecf/tests/bundles/org.eclipse.ecf.tests.filetransfer org.eclipse.ecf/tests/bundles/org.eclipse.ecf.tests.apache.httpclient.server org.eclipse.ecf/tests/bundles/org.eclipse.ecf.tests.filetransfer.httpserver If you use the ECF test cases the one to use is: org.eclipse.ecf.tests.filetransfer.URLRetrieveTest 3) Use some machine (windows?) that is behind your NTLMv2 proxy, where the proxy has been configured to use a client-provided username/password for proxy authentication. 4) Setup your Eclipse Connection preferences to use the NTLMv2 proxy...with Direct, Native, and/or Manual. I don't know what the appropriate proxy settings are for NTLMv2 proxies but this should be clear to you/users on that network. Note that if you run the ECF tests under the debugger, you will need to do set these preferences in the workspace that the debug process is using...and make sure the workspace/preferences are not cleared when you rerun things. 5) Set the folllowing system properties to produce httpclient debug output -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog -Dorg.apache.commons.logging.simplelog.showdatetime=true -Dorg.apache.commons.logging.simplelog.log.httpclient.wire.header=debug -Dorg.apache.commons.logging.simplelog.log.org.apache.commons.httpclient=debug -Dorg.apache.commons.logging.simplelog.log.httpclient.wire=debug 6) Run URLRetrieveTest and report what happens here or to me directly....both in terms of the httpclient debug output, and what happens with the http test methods. If you wish to set breakpoints in the ECF filetransfer code that sets up/uses the proxy and debug into this code see: org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.setupProxies() org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientRetrieveFileTransfer.setupProxy(Proxy) also see org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientRetrieveFileTransfer.ECFCredentialsProvider.getCredentials(AuthScheme, String, int, boolean) 7) If you would like to arrange to work on this together (e.g. via Skype/phone, etc) let me know and I will be happy to do so. If you wish/are able, perhaps we could even do a session with GoToMeeting with remote control...again let me know. Thanksinadvance Christian. Also, just for your/everyone's reference there is this on the lack of NTLMv2 support in httpclient: http://wiki.apache.org/jakarta-httpclient/FrequentlyAskedNTLMQuestions which unfortunately isn't very encouraging, but let's see what we can do, as Stefan's earlier tests did indicate that things seemed to be working so I think it's worth trying things out. Also, for everyone's info, there is a way to explicitly disable the httpclient-based provider and use the JDK-supplied URLConnection/Socket instead. This may provide a workaround for people on NTLMv2 networks...and so I believe it's worth trying for those on NTLMv2 networks. It's not desirable as a general solution as the URLConnection/Socket-based provider has other reliability issues which led us to use httpclient in the first place. To prevent the use of the httpclient-based provider, run Eclipse with the following system property set: -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient This will tell ECF filetransfer to exclude the httpclient provider, and so will use the JDK-provided http support instead.
(In reply to comment #37) I gave this workaround a try. I had one success when I added the following to the vmargs in my eclipse.ini: -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient -Dhttp.proxyPort=8080 -Dhttp.proxyHost=myProxy -Dhttp.proxyUser=MYDOMAIN\myUsername -Dhttp.proxyPassword=myPassword -Dhttp.nonProxyHosts=localhost|127.0.0.1 and using the Native method. I didn't have any success without the http.* properties. I had no success with the httpclient with any combo of connection method or properties. > Also, for everyone's info, there is a way to explicitly disable the > httpclient-based provider and use the JDK-supplied URLConnection/Socket > instead. > This may provide a workaround for people on NTLMv2 networks...and so I believe > it's worth trying for those on NTLMv2 networks. It's not desirable as a > general solution as the URLConnection/Socket-based provider has other > reliability issues which led us to use httpclient in the first place. > > To prevent the use of the httpclient-based provider, run Eclipse with the > following system property set: > > -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient > > This will tell ECF filetransfer to exclude the httpclient provider, and so will > use the JDK-provided http support instead. >
Thanks much for the trial Jonathan. I suggest we put this NTLMv2 workaround information somewhere more visible (e.g. on wiki). We can/will do this on ECF project wiki, but that may not be visible enough...Pascal (or others) do you have a preference? (In reply to comment #38) > (In reply to comment #37) > > I gave this workaround a try. I had one success when I added the following to > the vmargs in my eclipse.ini: > > -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient > -Dhttp.proxyPort=8080 > -Dhttp.proxyHost=myProxy > -Dhttp.proxyUser=MYDOMAIN\myUsername > -Dhttp.proxyPassword=myPassword > -Dhttp.nonProxyHosts=localhost|127.0.0.1 > > and using the Native method. I didn't have any success without the http.* > properties. > > I had no success with the httpclient with any combo of connection method or > properties. > > > Also, for everyone's info, there is a way to explicitly disable the > > httpclient-based provider and use the JDK-supplied URLConnection/Socket > > instead. > > This may provide a workaround for people on NTLMv2 networks...and so I believe > > it's worth trying for those on NTLMv2 networks. It's not desirable as a > > general solution as the URLConnection/Socket-based provider has other > > reliability issues which led us to use httpclient in the first place. > > > > To prevent the use of the httpclient-based provider, run Eclipse with the > > following system property set: > > > > -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient > > > > This will tell ECF filetransfer to exclude the httpclient provider, and so will > > use the JDK-provided http support instead. > > >
hi scott everyone else I finally came around to run some URLRetrieveTest (after setting up the environment that should be easier in the future) Tests are based on RC1 At least for me it made no difference whether I used Native or Manual proxy connection or whether I specified the http props on the commend line. The thing that made the difference was switching from httpclient to URL connection. With httpclient I had like 11 tests that failed. With URL Connection (specifying -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient ) I still had 3 tests blue (AssertionFailed) - testFTPReceiveFile, testHttpsReceiveFile, testFailedReceive i will attach a zip with a number of logfiles. If there is more tests that I can run or I should change the setup let me know. Stefan is returning on the 3rd June. christian
Created attachment 136811 [details] proxy test logging
(In reply to comment #41) > Created an attachment (id=136811) [details] > proxy test logging > Thanks Christian very much for the testing. I owe you a beer (we can exchange beers and providers next time we meet :). Based upon looking at your log files, I suggest that for 3.5 we focus on documenting/using the URLConnection provider (rather than the httpclient provider) as a workaround for people on NTLMv2 networks. Further, I think in order to create less confusion that we focus on using the http system properties rather than the platform native or manual proxy settings. Since people have to set the ECF system property anyway to disable the httpclient, it's probably going to be easier just to paste in the other system properties...i.e. -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient -Dhttp.proxyPort=8080 -Dhttp.proxyHost=myProxy -Dhttp.proxyUser=MYDOMAIN\myUsername -Dhttp.proxyPassword=myPassword -Dhttp.nonProxyHosts=localhost|127.0.0.1 Further, I would like to add on the following networking properties, to test the ftp support: -Dftp.proxyHost=myProxy -Dftp.proxyPort=8080 -Dftp.nonProxyHosts=localhost|127.0.0.1 If you (Christian) could rerun the tests with these ftp system properties to see if ftp can be supported this way I would appreciated it (my hope is that it will make the testFTPReceiveFile complete successfully...but I don't know whether it will because...where will it get the ntlm credentials from? i.e. will it use the http.proxyUser/proxyPassword values?. In any event, I think this needs some additional testing. BTW, for everyone's reference, here are the java networking properties: http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html Notice there is an NTLM-specific property: http.auth.ntlm.domain Which allows you to specify the ntlm domain. It can be specified either by the system property or by the http.proxyUser=MYDOMAIN\myUsername. So for consistency's/simplicity sake I propose that we just give people the instructions for using MYDOMAIN\myUsername (not using this property). I'm currently puzzled about the testHttpsReceiveFile failure. I'll try local tests with the URLConnection-based provider to see if I can reproduce (i.e. perhaps it's some other https issue...i.e. not the proxy). Thanks Christian.
Hi Scott, I did some tests with the settings you proposed: -Dftp.proxyHost=myProxy -Dftp.proxyPort=8080 -Dftp.nonProxyHosts=localhost|127.0.0.1 However to no success. I also found that all the http.proxy settings did not make a difference at all. If if left out it would work. The important setting was to use the URLConnection -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient Again the proxy settings in the Eclipse IDE -> Windows -> Preferences whether direct, manual or native didnt change anything. I also verified that my IE could in fact access a FTP site. So downloading through download.eclipse.org worked although I believe that the content is in our NTLM2 firewall scanned for viruses. Maybe that makes a difference. I attach the logfiles. christian
Created attachment 137134 [details] log file with ftp proxy settings
Thanks Christian, One clarifying question below. (In reply to comment #43) > Hi Scott, > > I did some tests with the settings you proposed: > > -Dftp.proxyHost=myProxy > -Dftp.proxyPort=8080 > -Dftp.nonProxyHosts=localhost|127.0.0.1 > > > However to no success. I also found that all the http.proxy settings did not > make a difference at all. If if left out it would work. Were there any conditions where ftp worked on NTLMv2?...or did the ftp test always fail (no matter what you did)?
I did not had one occurrence where the ftp tests worked. I also tried -Dftp.proxyUser -Dftp.proxyPassword and that did not help either. I tried FTP from IE to verify that FTP works in that environment and it connected and started to download, it felt relativly slow. So it might be that the proxy software that is used there only allows a few bytes while it already downloads everything, scans for virus and only forwards it when it is clean. A potential cause of the problem. Not sure. if you want me to try anything, or know a FTP site with a small download where you can get access to the log, then I start a download with IE to verify that FTP is not blocked altogether.... christian
OK, thanks Christian. I've created the following wiki page to explain how people that are using NTLMv2 proxies can work around the problems with Apache httpclient 3.X for Galileo/ECF 3.0. http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies If I've got anything wrong on this page please correct it. Christian RE: ftp...unfortunately, I don't think it's going to work given that your tests with the ftp proxy system properties set did not get things to work properly...my guess/expectation is that the ftp implementation in URLConnection (i.e. FTPURLConnection) simply doesn't have support for NTLMv2 proxies as the HttpURLConnection does. Unfortunate.
*** Bug 281472 has been marked as a duplicate of this bug. ***
Upgrading severity based on duplicates such as bug 281472. For people behind NTLMv2 proxies, not being able to use the Apache HTTP client and having to employ workarounds is a major problem.
*** Bug 282915 has been marked as a duplicate of this bug. ***
http://oaklandsoftware.com/papers/ntlm.html has a nice overview of NTLM support found in Java HTTP client libraries. Among other things mentioned is that "Sun JRE 6 or 5 (only 1.5_08 or higher) support now NTLMv2 on ALL platforms. Formally this was only provided on Windows. I have not found a definitive source on the sun site. Perhaps the information is based on http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4626557.
*** Bug 285532 has been marked as a duplicate of this bug. ***
Stefan, can we expect a resolution of this problem or at least a workaround (consisting in reverting to JVM impl of http) to be done for 3.5.1. This is hindering a great number of users.
(In reply to comment #53) > Stefan, can we expect a resolution of this problem or at least a workaround > (consisting in reverting to JVM impl of http) to be done for 3.5.1. If we need to fall back to reverting to the JVM impl, most/much of what needs to be done is Eclipse UI-related...and outside my personal reach for implementation. That is, we (ECF) can programmatically 'turn off' the httpclient provider, but the main question is how to put this under control of users (i.e. other than/in addition to the system property we have now). But I would hope that Stefan and/or others with access to an NTLMv2 proxy, along with my help/support could find some ways to use the httpclient provider...as from some of the previous tests on this bug (e.g. comment 25), it seems like it could be made to work.
I am not saying that I fully understand all the details of this bug. The outside view is - that we know a solution using the JVM http client provider rather than the http client from apache - a lot of people are bothered by the bug (see duplicate bugs) - its possible but rather uncertain (considering time, resources, access to NTLM proxy) when we understand under which conditions an apache httpclient works - this bug report was entered one year ago Here is my +1 to first choose the simple solution (use JVM http client) and take the pressure off this matter from our shoulders and then investigate the alternatives like apache httpclient. if you guys think otherwise go ahead. just my 2 cent.
Not sure if this is what is meant in the last comments but permanently reverting to the JVM http client is not a possibility for p2. We have gone through one release and most of the 3.5 dev cycle using apache http client to get confidence in this new code and reverting now could be more dramatic than this bug. Now, what I'm proposing here is for the ECF Apache http file transfer to discover when it is contacting an NTLM proxy and switch to the ECF JVM http file transfer for the duration of this request. Pawel, is there a way to detect when we are contacting an NTLM server?
Hi Christian, (In reply to comment #55) > I am not saying that I fully understand all the details of this bug. > > The outside view is > - that we know a solution using the JVM http client provider rather than the > http client from apache Although this will be better than the current situation, it's not particularly desirable for a couple of reasons: 1) It requires configuration by users and some UI to do so (the system property workaround given on: http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies is generally effective, but not very easy) 2) For those users (NTLMv2 proxy networks) they will be subject to all of the JVM http implementation problems that led us away from using it in the first place <stuff deleted> > - its possible but rather uncertain (considering time, resources, access to > NTLM proxy) when we understand under which conditions an apache httpclient > works I actually think that with the comments around comment 25 that we were getting pretty close, but I'm not certain as I don't have access to an NTLMv2 proxy myself. > - this bug report was entered one year ago true, but actually there hasn't been that much continuous work on this...i.e. we've had about 3 major spurts over the past year. > > Here is my > > +1 > > to first choose the simple solution (use JVM http client) and take the pressure > off this matter from our shoulders and then investigate the alternatives like > apache httpclient. As stated above, there are some disadvantages to using the JVM http client (user interface work needed that is not doable by ECF committers and JVM http bugs). But given some resources to do the necessary UI work, we can do this. I am of the opinion that some more effort should be put into httpclient workarounds with those with access to NTLMv2 proxies, but am unable to effect this myself.
(In reply to comment #56) > Pawel, is there a way to detect when we are contacting an NTLM server? There is. The NTLM scheme is picked as a result of negotiation between client and the server. These schemas are prioritized with NTLM having the highest priority (then Digest and Basic). So, technically speaking Http Client could connect to some proxy servers if it didn't send NTLM as one of supported ones. However it wouldn't work with proxy servers that are configured to use NTLM only with no fall back schemas.
I think we have several options to deal with this situation. 1.) Have a preference to switch the http client implementation. 2.) ECF switches (if possible) to JVM http client when NTLM is detected (comment #58) 3.) As I understood Scott there is global switch to tell ECF what htpp client implementation to use. I fully understand that this workaround is not worth striving for. Unfortunately my knowledge of ECF is very low. However, is it possible that users (p2, ..) of ECF could specify individually which http client implementation to use? 4.) I believe that finding the root cause of the problem can only be done effectively by someone who is very familiar with ECF and the different htpp client implementations. For the options 1.-3. Christian and I could help with testing. For the option 4. I think we can not (available time, knowledge, ..), but we can offer Scott to stay here and work on that problem. This means he may come to us (here in Frankfurt) and we would give him access to our NTLM environment. So, in case Scott comes to the ESE he may visit us before or after the summit. I think this would be the most effective solution. Tschüß, Stefan
Scott, Pawel is setting up an NTLM proxy on the IBM internal network to help us figure out what is going on. So hopefully in the next few days we should be able to debug that issue some more or at least try out easily any new solution. Pawel, how far along are you on setting up that proxy?
(In reply to comment #60) > Pawel, how far along are you on setting up that proxy? I got a domain set and I have to wait for an upgrade of my MSDN license to connect to a proxy server. The license transfer will take a few days.
(In reply to comment #59) <stuff deleted> > 4.) I believe that finding the root cause of the problem can only be done > effectively by someone who is very familiar with ECF and the different htpp > client implementations. I'm quite certain that the root cause is described/pointed to here: http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies (it's the fact that httpclient 3.1 has/did not license NTLMv2 support from Microsoft) If there is a workaround (i.e. as suggested by comment 25 than thereabouts) for httpclient, then it's only going to be tested for effectiveness on some existing NTLMv2 proxy. > > For the options 1.-3. Christian and I could help with testing. > For the option 4. I think we can not (available time, knowledge, ..), but we > can offer Scott to stay here and work on that problem. This means he may come > to us (here in Frankfurt) and we would give him access to our NTLM environment. > So, in case Scott comes to the ESE he may visit us before or after the summit. Well, thanks for the offer of use of your network, but I'm not certain yet whether I'll be attending ESE...and obviously that's a little ways off (past 3.5.1). I'm willing to work with other people's NTLMv2 networks (if made available to me), but I think Stefan you overstate the necessity for ECF expertise to do testing with httpclient. Basically, what's needed is access to an NTLMv2 proxy network and some time with that network/proxy to figure out what httpclient provider is doing/can do given the limitations described in http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies ...and whether a workaround exists. There's also the entirely different path of dropping back to using the JRE-based http client (which has licensed NTLMv2 support from Microsoft). The only questions here are: 1) Can we make the switch automatic (i.e. detect the NTLMv2 proxy auth info as per comment 56) 2) If not, then what sort of UI (e.g. preferences, etc) should we have for allowing the user to enter in NTLMv2 proxy info (e.g. auth info). Pawel is now investigating 1 (thanks Pawel, Pascal, IBM, etc). If 1 doesn't work out, I'm not sure if people would want 2, but if so it has to be done by someone with access to Eclipse UI contributions (which I don't have). But I was hoping that someone with access to NTLMv2 proxy would expend a little energy on trying out the httpclient workarounds we already started (i.e. comment 25), because if those could be made to work they would be preferable.
I think there is no disagreement that a "automagic", transparent solution either by switching on NTLM detection to the JVM httpclient or by making apache httpclient work would be the preferred one. However I sense a good uncertainty when we have that thing working. I would highly recommend that we start investigating in parallel a fallback strategy in case that our preferred solution does not work. (especially since 3.5.1. is around the corner I believe) I think a checkbox named "enabled NTLM proxy support" somewhere in the p2 UI is ugly (maybe very ugly) but for a) if I as a user were affected I would know that I am behind a NTLM proxy and b) I would be happy to click a checkbox if it makes the difference between a working solution and a failure. Maybe thats a over trivial look at things. another 2 cent from me
(In reply to comment #63) > I think there is no disagreement that a "automagic", transparent solution > either by switching on NTLM detection to the JVM httpclient or by making apache > httpclient work would be the preferred one. > > However I sense a good uncertainty when we have that thing working. I would > highly recommend that we start investigating in parallel a fallback strategy in > case that our preferred solution does not work. (especially since 3.5.1. is > around the corner I believe) Here's the way I see the order of desirability for the possible solutions 1) If we can find a workaround for the httpclient 3.1's published inability to deal with NTLMv2 proxies then we should do so...because then everything would just work using httpclient. That's why I think it would be useful to try some more investigation/testing into httpclient workarounds. Comment 25 raised my hope that such a workaround might be possible. I cannot do any such testing of workarounds without access to a NTLMv2 proxy for testing (although I'm willing to implement any workarounds). 2) As per comment 61, Pawel is looking into detecting NTLM proxy information (e.g. auth info), so that ideally we could detect NTLMv2 proxies, get the authentication information, and disable the httpclient provider programmatically (i.e. use the JRE http code) without requiring that the user do anything in the UI. This is less desirable than 1, because NTLMv2 users are forced to use the (buggy) JRE http client code...but it's a lot more desirable than using the workaround described on http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies. 3) If 2 cannot be done, then I would suggest that we introduce some UI to disable the httpclient provider (e.g. preference) so that at least users won't have to specify system properties as per the http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies. So, my proposal is that we persue both 1 and 2 (Stefan/I/Christian/others on 1, Pawel and I on 2). If 1 comes through (fairly unlikely perhaps, but still worth checking), all will be well for NTLMv2 users (and there will be much rejoicing). If 1 doesn't come through, but if 2 works out, Pawel (proxy API) will introduce a way to auto-detect NTLMv2 proxies, and I will enhance ECF to use this information to automatically/transparently fall back to using the JRE's http code for file transfer. Finally, if 2 turns out to be impossible or prohibitively expensive (e.g. licensing something from MS), then I would propose that a preference be introduced in UI to allow user to a) disable httpclient provider; b) specify NTLMv2 auth information to make it easier than the system properties method specified here http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies. Admitted, it's not a great option to introduce UI, but what we have now is system properties for workaround...which does work...but it's clearly a pain. So what I'm looking for is help with 1 and 2...since I do not have any regular access to NTLMv2 proxy, and so can't persue any of these solutions by myself/with only ECF resources. If Stefan/Christian/others/I work on 1, and Pawel and I work on 2, then I think there's a reasonable chance we won't have to do 3.
thx for establishing this concrete plan Scott.
I know I am coming late to this party, but there is a question hanging in the air that you Scott might be able to explain. I asked Stefan the other day what all this references to "buggy JRE http client" are about and why you prefer the apache httpclient. I have used both in the past and have no particular preference myself. What surprised me was that he couldnt tell me. So please shed some light on the inside knowledge you have. I am sure that helps us to better understand the whole problem. I mean we are running a productive customer site that uses JRE http client (the URLConnection) which takes 1.5 Mio Hits (Webservice calls) every day from a large user base (with and without NTLM proxy) that runs for a couple of years now and I would really like to understand the problems in that implementation. Is it also related to JRE versions like its bad with 1.4 but better with 1.5 or 1.6 ? thanks
(In reply to comment #66) > I know I am coming late to this party, but there is a question hanging in the > air that you Scott might be able to explain. > > I asked Stefan the other day what all this references to "buggy JRE http > client" are about and why you prefer the apache httpclient. I have used both in > the past and have no particular preference myself. What surprised me was that > he couldnt tell me. > > So please shed some light on the inside knowledge you have. I am sure that > helps us to better understand the whole problem. The main one was https://bugs.eclipse.org/bugs/show_bug.cgi?id=234916 > I mean we are running a productive customer site that uses JRE http client (the > URLConnection) which takes 1.5 Mio Hits (Webservice calls) every day from a > large user base (with and without NTLM proxy) that runs for a couple of years > now and I would really like to understand the problems in that implementation. > Is it also related to JRE versions like its bad with 1.4 but better with 1.5 or > 1.6 ? I don't know.
I can't understand few things. 1. This is annoying bug which was reported prety long time ago. And it seems that everybody know how to fix or at least work around it. How long it will take until it is fixed? 2. Prior to last release proxy in eclipse worked just fine for years. Update to proxy code only caused problems with UI (issue with enabled for edit entries which should be ead onlt), confusion for users (related to described UI problems) and problems with functionality (NTML 2 support). And at the same time it gave no real advantages. So why the hell someone decided to make such "enhancement"? And why the hell youcan't just revert proxy elated changes back and makeeveryobe happy?
(In reply to comment #68) > Prior to last release proxy in eclipse worked just fine for years. Update to > proxy code only caused problems with UI (issue with enabled for edit entries > which should be ead onlt), confusion for users (related to described UI > problems). There was a bug 228739 opened to show what discovered proxy settings are and how they were detected. This is because in some circumstances native library providing OS settings fails to load. In Eclipse 3.4 users only knew that Eclipse tries to use native settings (if available). The library could fail to load even if it loaded before (provided some configuration change was done) causing silent lost of functionality. In Eclipse 3.5 entries for native settings can be enable by double clicking them, "Edit" button is disabled for them. For sure this causes confusion and it's been fixed for 3.5.1, see bug 281628.
(In reply to comment #68) > I can't understand few things. > > 1. This is annoying bug which was reported prety long time ago. And it seems > that everybody know how to fix or at least work around it. That is not correct. Some relevant history: The issue is based upon the fact that ECF was asked to move to use apache httpclient during 3.5 development cycle...because of bug 234916 (among some lesser issues). Apache httpclient does not officially support NTLMv2 proxies, and this is the entire source of the issue. For details about why, please see http://wiki.apache.org/jakarta-httpclient/FrequentlyAskedNTLMQuestions#head-d318851e40156de248e72fbb15bbce52a5e18f2c >How long it will > take until it is fixed? The workaround described here works now: http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies It may be impossible to 'fix' the apache httpclient provider to work with NTLMv2 providers...but I think it's still an open question whether workarounds exist..see '1' in comment 64. As you point out, another way to workaround the httpclient limitation is to simply disable httpclient, and use the JRE-based provider (which doesn't suffer from the NTLMv2 proxy issue because Sun licensed the NTLMv2 proxy support from Microsoft). Disabling the httpclient provider and using the JRE impl is what is accomplished by the instructions on http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies. If you haven't, please give this a try. If you have and it doesn't work, I/we certainly want to know about it. The options 2 and 3 as described in comment 64 simply improve this disabling, so that users don't have to specify system properties. > 2. Prior to last release proxy in eclipse worked just fine for years. Update to > proxy code only caused problems with UI (issue with enabled for edit entries > which should be ead onlt), confusion for users (related to described UI > problems) and problems with functionality (NTML 2 support). And at the same > time it gave no real advantages. So why the hell someone decided to make such > "enhancement"? >And why the hell youcan't just revert proxy elated changes back > and makeeveryobe happy? > Before I respond to your question I'll say I object to your tone and language. If you want to get support for NTLMv2 proxies (without using system properties...which is already there), then accusing and insulting the people who will provide it to you is not a very effective way to get it. But in answer to your question: the decision to prioritize/use apache httpclient (instead of jre) was made at least partially because of bug 234916...which affected a seemingly large number of people using 3.4, and was/is not addressable by us (it's apparently some problem with the JRE socket impl). ECF has a multi provider architecture, and so *both* the httpclient provider *and* the jre provider are there and usable by p2 in Eclipse 3.5. This makes possible: http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies and I would ask that if you haven't then please *try this workaround*. I/we/all understand it's not appealing to use system properties for this workaround, and so simplification/elimination of the use of System properties is the path described for '2' and '3' in comment 64. Finally, it should be made clear (again), that work on this issue has/is/will be slowed and complicated by the following 1) I (Scott)/ECF team do *not* have access to an NTLMv2 proxy. This makes it impossible for me/ECF team to try out workarounds or even reproduce the issues people are having. We have no means/resources to setup such an environment. 2) As far as I know, the p2/Equinox/platform teams don't have access to an NTLMv2 proxy either. 3) We therefore have to depend on people/orgs that *do* have access to NTLMv2 proxies to attempt to find workarounds for httpclient's limitations (1 in comment 64) and/or find ways to auto-detect/get auth info for NTLMv2 proxies (2 in comment 64). That means *the community*...i.e. you. If you or others that really want NTLMv2 proxy support (without system properties), then you could donate/contribute hw, sw, or time to ECF (or p2 or Equinox) so that we can fully/more quickly deal with this issue. Or you could work with us to create/test workarounds on your NTLMv2 proxy network...that would be great also. But at least from my chair, accusing and/or insulting the people that are doing the work *for you* is not a good way to go about it.
I'm sorry if I insulted anyone. I am really sorry about that. But the problem is that the bug about proxy auth problem was opened the next day after eclipse 3.5 release. It was about 6 weeks ago. The root cause of this problem was established pretty quickly. Just look at this issue and at 281472. This one already has more that 70 comments and 281472 has more that 60 comments. There might be some other issues marked as duplicate for this one. I've read httplcient's faqs about ntlmv2 proxy. And from their remark about some potential legal problems it very unlikely that they will support ntlmv2 proxy. I understand that there was a real problem described in issue 234916. But from comments #32 and #34 it is not clear if httpclient provider really helped because there were no confirmation from reporter. And if you look at comment #4 for 220240 (it is dated 2008-02-27) it was already known problem that ntlmv2 proxy will not work with httpclient based provider. That is why from my point of view httpclient based provider does not solve any problem. Because even if it resolves the problem described in 234916 it introduces the new problem. Perhaps there were some other reasons to move to httpclient. But here is a real story. I work in offshore software company that has about 1000 employees. Most of them are java developers and wast majority of our java developers use eclipse IDE. Now imagine their confusion when after update to Eclipse 3.5 they were not able to use mylyn and automatic updates (software install from web repositories). Before update it worked fine. Yes, I understand the reason why ntlmv2 proxy is not working. But the problem is that it was know that ntlmv2 proxy is not working and why is not working week ago and two weeks ago and even earlier. I just don't understand what's next. As I wrote earlier it is very unlikely that httpclient will support ntlmv2 proxy. And seems everybody know this. So what is "official" statement about this.
Hi Serhiy, I can only support you in what you are writing here. Its also unclear to me why we switch to apache http client for bug 234916 which is a) maybe caused by a slow mirror or server b) was hard to reproduce at the end c) we have no confirmation that it is fixed now (just good faith) What we are doing is trading one bug for another. I understand that we want to address both bugs (that what Scott referred to when he names the comment 25) but I think we should think of a backup strategy in parallel to investigate the cause. Something along the line: - we MUST fix this (I mean THIS bug) for 3.5.1. - if all fails we have to revert to jre http client or - at least supply a checkbox that you can use to enable jre http client on demand at your choice. Maybe it helps to raise the priority of this bug. again 2 cent from me.
(In reply to comment #71) > I'm sorry if I insulted anyone. I am really sorry about that. Accepted. > > But the problem is that the bug about proxy auth problem was opened the next > day after eclipse 3.5 release. It was about 6 weeks ago. The root cause of this > problem was established pretty quickly. Right. That's because I was the one who established the root case...ala the apache httpclient provider not supporting NTLMv2 proxies...and provided the workaround on the web page: http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies >Just look at this issue and at 281472. > This one already has more that 70 comments and 281472 has more that 60 > comments. Right. Many of the comments on this bug are/were around testing various workarounds attempted/incompletely tested over the past year (because testing had to depend upon others), and many of those on bug 281472 were me pointing to the workaround and/or me repeatedly explaining the situation. >There might be some other issues marked as duplicate for this one. Sure...there have been a number of duplicates that have already been marked as such. > > I've read httplcient's faqs about ntlmv2 proxy. And from their remark about > some potential legal problems it very unlikely that they will support ntlmv2 > proxy. > > I understand that there was a real problem described in issue 234916. But from > comments #32 and #34 it is not clear if httpclient provider really helped > because there were no confirmation from reporter. And if you look at comment #4 > for 220240 (it is dated 2008-02-27) it was already known problem that ntlmv2 > proxy will not work with httpclient based provider. > > That is why from my point of view httpclient based provider does not solve any > problem. Because even if it resolves the problem described in 234916 it > introduces the new problem. I see that from your point of view the httpclient-based provider does not solve any problem, but from other points of view it does (e.g. the deadlock issue and others). > > Perhaps there were some other reasons to move to httpclient. FWIW, yes, there are. > > But here is a real story. I work in offshore software company that has about > 1000 employees. Most of them are java developers and wast majority of our java > developers use eclipse IDE. Now imagine their confusion when after update to > Eclipse 3.5 they were not able to use mylyn and automatic updates (software > install from web repositories). Before update it worked fine. So...why can't/don't they use the workaround on http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies This is proven to work for others using NTLMv2 proxies. If this can't be used, why not? As I've stated, this is not a desirable workaround (using system properties), and so that's the source of what Pawel and I will do (comment 64 items 2 and 3) to make the httpclient disabling more automatic. > > Yes, I understand the reason why ntlmv2 proxy is not working. But the problem > is that it was know that ntlmv2 proxy is not working and why is not working > week ago and two weeks ago and even earlier. > > I just don't understand what's next. As I wrote earlier it is very unlikely > that httpclient will support ntlmv2 proxy. And seems everybody know this. > So what is "official" statement about this. Here is the official statement about what I and ECF will do: As described in comment 64, we will be persue two threads: 1) see if a workaround can be created that will allow httpclient provider to work with NTLMv2 proxies. This will have to be done with the cooperation of *someone* who has a NTLMv2 proxy network. 2,3) Reduce as much as possible the user work necessary to disable the httpclient provider and use the jre provider...by (2) seeing if the proxy API can detect an NTLMv2 proxy config on local system, and if that doesn't work then (3) introducing some Eclipse UI to allow the disabling to occur in some other way than by System properties. If others have ideas for dealing with this that don't require us to rewind to Eclipse 3.4 jre-induced problems and limitations then I would love to hear them (and would appreciate even more people offering to actually do something with the ideas). (In reply to comment #72) > Hi Serhiy, > > I can only support you in what you are writing here. Its also unclear to me why > we switch to apache http client for bug 234916 which is a) maybe caused by a > slow mirror or server b) was hard to reproduce at the end c) we have no > confirmation that it is fixed now (just good faith) Christian, you and/or anyone else can feel free to test this with the jre provider, and if you can satisfy the relevant people that bug 234916 has been addressed in some/all necessary versions of the jre then I would guess the decision could be revisited. But there are other things about the httpclient provider that are advantageous (e.g. cancelability bug 263613). A fair amount of work was done during this past year on the httpclient provider on such things, and those features are not in the jre provider at this point. > > What we are doing is trading one bug for another. I understand that we want to > address both bugs (that what Scott referred to when he names the comment 25) > but I think we should think of a backup strategy in parallel to investigate the > cause. Please feel free Christian. You and Stefan have access to an NTLMv2 proxy network, and I would welcome (in fact I would beg for) your commitment to using it to investigate the cause further, since I cannot do that myself. That's actually why this bug is assigned to Stefan (because he/you can actually do this investigation...since he has access to an NTLMv2 proxy...while I cannot). > > Something along the line: > > - we MUST fix this (I mean THIS bug) for 3.5.1. That's nice to say, but given that there may not be a complete workaround possible for the httpclient provider, it seems sort of reckless to say that this can be done within the next few weeks (3.5.1 timeframe). But if you are able and willing to commit to that, then fine. But just to be clear: I/ECF are not able to commit to that. > - if all fails we have to revert to jre http client or > - at least supply a checkbox that you can use to enable jre http client on > demand at your choice. I'm not sure why this is so confusing, but it apparently is so I'll say it again: The options 2 and 3 in comment 64 describe the approach that we can take and are taking for 3.5.1 to disable the httpclient provider. The idea is to disable the httpclient provider in a user invisible way, rather than use of system properties. The first choice on this thread (2) is to make it completely invisible, by having the proxy API detect the NTLMv2 proxy and auth info, and then at connect time using that info to have ECF use the jre provider rather than the httpclient provider. The second choice (if 2 can't be made to work), is to introduce some ui (e.g. in prefs) to disable the httpclient provider and use the jre one. > > Maybe it helps to raise the priority of this bug. If you wish to do this then please go ahead. But the assignment of this bug to Stefan will not be changed (i.e. it's Stefan's/yours to do this)...since no matter what the bug priority, I am not able to investigate the cause or create/test workarounds for the resource reasons stated previously.
The switch from JRE http client to Apache http client was motivated by reliability issues. After having discussed with ppl in the community and in IBM, it came clear that some of the p2 related issues were caused by the flakiness of the JRE implementation, and as such as a group p2 decided to move to the Apache impl early in the 3.5 cycle. As I said in comment #56, for 3.5.x, switching back the SDK defaults to the JRE impl seems even more risky and I fully support Scott's plan described in comment #64. From now on, please let's stay focus on finding a technical solution.
I feel I should comment on this. A bit of background: my company (Oakland Software) has sold a Java HTTP client that supports NTLMv2 for about 7 years now, so I know a little about what's possible and what's not. I recommend: 1) Stop pursuing support for NTLMv2 in the Apache HTTP client. They never have supported it and they apparently never will (either in the 3.1 line or the newer HTTP Components); they are afraid of the legal issues. They also don't have a mechanism to test it. 2) Use the JRE support if it's available. On 1.5 and higher I'm pretty sure that it just works on all platforms. It's there in 1.4, but only works on Windows. 3) You can test it by setting up a windows machine with ISA and configuring that as a proxy. ISA can be obtained cheaply as part of the Windows development software (like as obtained from the Microsoft Action Pack). Of course in larger companies you probably already have licenses for it. And you need to find someone to help you configure this as a proxy for your testing (it's not that easy). It's worth hiring an expert to do this (I'm not one). 4) Dynamic detection can be done by trying normal NTLM support and having it fail. There is however no indication in the protocol that the server supports NTLM or NTLMv2 (there is a flag called NTLMv2 but it's not used for that purpose). So the only way to really check is to try NTLMv2 to see if it works after NTLM has failed.
(In reply to comment #75) > 3) You can test it by setting up a windows machine with ISA and configuring > that as a proxy. ISA can be obtained cheaply as part of the Windows > development software (like as obtained from the Microsoft Action Pack). Of > course in larger companies you probably already have licenses for it. And you > need to find someone to help you configure this as a proxy for your testing > (it's not that easy). It's worth hiring an expert to do this (I'm not one). Francis, thanks for the information. I've been working to setup this kind of environment in IBM internal network. I've already got ISA Server 2006 installed and a domain controller. Still working on configuring the ISA Server though. > 4) Dynamic detection can be done by trying normal NTLM support and having it > fail. There is however no indication in the protocol that the server supports > NTLM or NTLMv2 (there is a flag called NTLMv2 but it's not used for that > purpose). So the only way to really check is to try NTLMv2 to see if it works > after NTLM has failed. > Scott, please provide more light on that. Do you want to fall back to JRE client only when NTLMv2 is used? This could be done in case of NTLM in general since even if NTLM(v1) is used we still require users to provide login credentials in proxy settings. JRE client provides operates in a sign-on fashion in such cases. If the fall back is planned when only NTLMv2 authentication is used then Francis is absolutely right.
(In reply to comment #76) Thanks also to Francis for the info. <stuff deleted> > > 4) Dynamic detection can be done by trying normal NTLM support and having it > > fail. There is however no indication in the protocol that the server supports > > NTLM or NTLMv2 (there is a flag called NTLMv2 but it's not used for that > > purpose). So the only way to really check is to try NTLMv2 to see if it works > > after NTLM has failed. > > > > Scott, please provide more light on that. Do you want to fall back to JRE > client only when NTLMv2 is used? >This could be done in case of NTLM in general > since even if NTLM(v1) is used we still require users to provide login > credentials in proxy settings. JRE client provides operates in a sign-on > fashion in such cases. I'm inclined to fall back to the JRE provider in the case of either NTLMv1 or v2. The reason is that Henrich Kraemer has recently provided some information that would suggest that NTLMv1 is perhaps not as well supported by the apache httpclient 3.1 as it suggests on the apache wiki. So, let's assume falling back to JRE provider with either v1 or v2. Hopefully, this makes the proxy api detection work easier as well.
(In reply to comment #77) > I'm inclined to fall back to the JRE provider in the case of either NTLMv1 or > v2. The reason is that Henrich Kraemer has recently provided some information > that would suggest that NTLMv1 is perhaps not as well supported by the apache > httpclient 3.1 as it suggests on the apache wiki. > > So, let's assume falling back to JRE provider with either v1 or v2. Hopefully, > this makes the proxy api detection work easier as well. > +1 There are older configurations, like NT servers where the Apache stuff will not work correctly (even though it's not NTLMv2). Better to go with the Sun stuff which was done with the blessing of Microsoft for all NTLM.
I wonder if it would make sense to have some kind of native fragment that would support ntlm on windows?
(In reply to comment #79) > I wonder if it would make sense to have some kind of native fragment that would > support ntlm on windows? > As far as I'm/ECF is concerned, how the platform proxy API detects ntlm is up to it (i.e. Pawel)...and I believe that the proxy API already has native fragments...so Eugene's suggestion seems fine to me but is up to Pawel how implemented. Just to be clear, all ECF needs out of the proxy API in order to programmatically de-activate the apache httpclient provider, and use the JRE-one is: 1) An indication that it's an NTLM proxy type (via IProxyData.getType() returning a new "ntlm" type I assume). 2) Access to the proxy authentication info (domain/username, password) through the usual IProxyData.getUser() and IProxyData.getPassword() methods.
Maybe Eugene (comment #79) is thinking of a native http client provider using e.g. the win32 winhttp api. This would of course support NTLM with authentication with the credentials of the logged on user. ==> http://msdn.microsoft.com/en-us/library/aa383144(VS.85).aspx This would be similar to using the JRE HttpUrlConnection. The proxy API is also using winhttp to retrieve the proxy information. But I think that there is no direct way of retrieving whether NTLM is requested. It would be necessary to start an http request with winhttp and find out what authentication scheme is requested. This should already be possible with ECF/Apache HttpClient, or?
(In reply to comment #81) > Maybe Eugene (comment #79) is thinking of a native http client provider using > e.g. the win32 winhttp api. This would of course support NTLM with > authentication with the credentials of the logged on user. > ==> http://msdn.microsoft.com/en-us/library/aa383144(VS.85).aspx > This would be similar to using the JRE HttpUrlConnection. > > The proxy API is also using winhttp to retrieve the proxy information. But I > think that there is no direct way of retrieving whether NTLM is requested. It > would be necessary to start an http request with winhttp and find out what > authentication scheme is requested. > This should already be possible with ECF/Apache HttpClient, or? I'm confused by this question. Of course it's possible to start an httprequest with any/all of the following: JRE httpurlconnection Apache httpclient ECF filetransfer API (using either of the above to implement) winhttp (available via the windows fragment for proxy API I assume) But the ECF filetransfer API is the only one of the above that uses/looks to the platform proxy API to provide it with information about proxies...i.e. what type of proxies exist for the platform, what their type is, and what authentication information they require. As per comment 80, ECF needs the ntlm type from the platform proxy API in order to decide at runtime whether to use the jre-based provider (if NTLM proxy is present), or the apache httpclient provider (if NTLM proxy is not present). I don't understand how it would make sense for the proxy API/native provider to use ECF filetransfer to *determine* whether an NTLM proxy is present...because ECF needs that information to decide which provider to use.
While testing weather NTLM proxies do work I found the following issues: bug 286354 username, password of manual Proxy is not persisted. bug 286451 InvalidCredentialsException using NTLM authentication > However it does succeed after retry. bug 286450 ECF HttpClient transfer of URLs with a non default port do not work when using proxy However despite all this the NTLM proxy did work for me. This is an NTLM version 1 proxy (ISA server 2006). I believe my concerns that Scott mentioned in comment #77 are related to this issue in HttpClient 3.1 https://issues.apache.org/jira/browse/HTTPCLIENT-856
(In reply to comment #82) At first sorry for this lengthy comment and for annoying most of you. > I'm confused by this question. Of course it's possible to start an httprequest > with any/all of the following: I am even more confused. Maybe it is because I use the wrong vocabulary. > JRE httpurlconnection > Apache httpclient > ECF filetransfer API (using either of the above to implement) > winhttp (available via the windows fragment for proxy API I assume) AFAIK, the proxy api (org.eclipse.core.net) provides clients (such as ECF) with proxy data for a given url. The proxy data contains the proxy host and port, its type (http,https,..), maybe userid and password. But it does not provide the authentication scheme (basic, digest, NTLM, ..). This will be negotiated when doing an actual request. The proxy api (org.eclipse.core.net) can use proxy providers (sub-classes of AbstractProxyProvider) for system proxies, i.e. providing proxy data in a system dependent way (native code using winhttp for windows). From Eugene's comment #79 I was just guessing that he was thinking of an HttpClient provider that uses the win32 winhttp api because that would support NTLM (and single sign-on). But this does not exist. > > But the ECF filetransfer API is the only one of the above that uses/looks to > the platform proxy API to provide it with information about proxies...i.e. what > type of proxies exist for the platform, what their type is, and what > authentication information they require. As said above this information can only be retrieved by negotiation with the proxy. > > As per comment 80, ECF needs the ntlm type from the platform proxy API in order > to decide at runtime whether to use the jre-based provider (if NTLM proxy is > present), or the apache httpclient provider (if NTLM proxy is not present). Couldn´t ECF start with Apache HttpClient and fallback to JRE HttpUrlConnection when NTLM authentication with Apache HttpClient fails? (Although authentication may fail because of wrong userid/password!) > > I don't understand how it would make sense for the proxy API/native provider to > use ECF filetransfer to *determine* whether an NTLM proxy is present...because > ECF needs that information to decide which provider to use. I never thought about that. It was in response of ECF's "requirement" to know from the start whether NTLM authentication is requested or not so that it can decide which http client provider to use. If the proxy api (org.eclipse.core.net) should support that it would have to start a request to retrieve the requested authentication scheme. A native proxy provider would of course not use ECF but winhttp on windows. However, this seems for me a wrong solution.
(In reply to comment #84) <stuff deleted> > > I never thought about that. It was in response of ECF's "requirement" to know > from the start whether NTLM authentication is requested or not so that it can > decide which http client provider to use. If the proxy api > (org.eclipse.core.net) should support that it would have to start a request to > retrieve the requested authentication scheme. A native proxy provider would of > course not use ECF but winhttp on windows. > However, this seems for me a wrong solution. > As I understand it, the proxy api is intended to collect/have info about the existence and configuration of proxys (e.g. auth data) for use by clients that need to communicate via those proxies (e.g. ECF). It seems to me that the proxy API should collect/have the relevant data about NTLM proxies as well, and that given this information to distinguish an NTLM proxy, ECF can use the jre provider rather than the httpclient provider. Problem solved. It seems strange to me that ECF would be expected to figure out the proxy information/configuration on it's own for NTLM proxies...but be expected/required to get that info from the proxy api for all non NTLM proxies. That would seem to violate the separation of concerns that we already have with the proxy api for proxy config, and the ECF client (or other clients) for doing the actual file transfer.
Scott, but isnt the proxy API collecting information about HTTP proxies and HTTPS proxies, so in the end about various protocol. However NTLM is not a protocol but an authentication scheme like Basic Authentication. The authentication scheme that a server wants cannot be captured by a proxy definition but can only be captured at connection time. Only when you connect and try to retrieve the data you find out wether NTLM is required or not. That in my view would be somewhere in ECF where you use apache http client or jre http client. A proxy to use can change depending on the URI. So http://host/abc could be told to use a different proxy than http://host/xyz. And different proxies could mean different authentication schemes. Or do you have a different view on this ?
(In reply to comment #85) > As I understand it, the proxy api is intended to collect/have info about the > existence and configuration of proxys (e.g. auth data) for use by clients that > need to communicate via those proxies (e.g. ECF). > > It seems to me that the proxy API should collect/have the relevant data about > NTLM proxies as well, and that given this information to distinguish an NTLM > proxy, ECF can use the jre provider rather than the httpclient provider. > Problem solved. Negotiation of authentication schemes (simple challenge-response authentication mechanism) is defined by the HTTP authentication protocol: http://www.ietf.org/rfc/rfc2617.txt > > It seems strange to me that ECF would be expected to figure out the proxy > information/configuration on it's own for NTLM proxies...but be > expected/required to get that info from the proxy api for all non NTLM proxies. > That would seem to violate the separation of concerns that we already have > with the proxy api for proxy config, and the ECF client (or other clients) for > doing the actual file transfer. As already said if the proxy api (org.eclipse.core.net) should retrieve the authentication scheme it had to make an http request. And this for every uri - this would double the number of requests. That can´t be a solution. I think there is no violation of SoC because authentication belongs to the http protocol.
(In reply to comment #87) > As already said if the proxy api (org.eclipse.core.net) should retrieve the > authentication scheme it had to make an http request. And this for every uri - > this would double the number of requests. That can´t be a solution. > I think there is no violation of SoC because authentication belongs to the http > protocol. > That's correct. But since Workspace Team sets an appropriate environment we we'll also try to provide a patch. It will probably end being located in ECF/Apache HTTP Client code. However the location isn't our main concert at the moment.
(In reply to comment #88) > (In reply to comment #87) > > As already said if the proxy api (org.eclipse.core.net) should retrieve the > > authentication scheme it had to make an http request. And this for every uri - > > this would double the number of requests. That can´t be a solution. > > I think there is no violation of SoC because authentication belongs to the http > > protocol. > > > > That's correct. But since Workspace Team sets an appropriate environment we > we'll also try to provide a patch. It will probably end being located in > ECF/Apache HTTP Client code. However the location isn't our main concert at the > moment. > ok. Frankly I don't care where in the code the patch goes. If proxy negotiation/configuration has to go in the ECF httpclient provider code for reasons that still seem bizarre to me, then fine. Since I/ECF don't have access to NTLM/Workspace Team I leave it to Pawel...and/or others that will/can implement and test...to decide. I'm going to limit my further participation on the discussion on this bug in order to work on effecting the necessary changes.
I managed to setup the environment that requires clients to authenticate using NTLM scheme. Since the information about the negotiation process is sent in HTTP headers it isn't hard to find a place where it can be accessed. For example: HttpMethodDirector.executeWithRetry(HttpMethod) line: 398 After execution passes this line it's possible to call: method.getResponseHeaders("Proxy-Authenticate"); And the output on my machine looks like this: [Proxy-Authenticate: Negotiate , Proxy-Authenticate: Kerberos , Proxy-Authenticate: NTLM ] Scott, Pascal, I'm not sure how the fall back should be handled in such case. Please decide whether to propagate such situation to p2 which (probably) should react by displaying a message and switching providers. Or should it be logged by ECF and handled silently from p2 point of view. Also, this fix could potentially require a change in Apache HTTP Client code. I'm not sure what the policy for modifying HTTP Client code is. Does it require an IP process? One more thing. Like it was said before, NTLM value in the header means that either NTLM or NTLMv2 is used. If any more help from Workspace Team is needed at this point please don't hesitate.
(In reply to comment #90) > I managed to setup the environment that requires clients to authenticate using > NTLM scheme. Since the information about the negotiation process is sent in > HTTP headers it isn't hard to find a place where it can be accessed. For > example: > > HttpMethodDirector.executeWithRetry(HttpMethod) line: 398 > > After execution passes this line it's possible to call: > > method.getResponseHeaders("Proxy-Authenticate"); > > And the output on my machine looks like this: > > [Proxy-Authenticate: Negotiate > , Proxy-Authenticate: Kerberos > , Proxy-Authenticate: NTLM > ] > > Scott, Pascal, I'm not sure how the fall back should be handled in such case. > Please decide whether to propagate such situation to p2 which (probably) should > react by displaying a message and switching providers. Or should it be logged > by ECF and handled silently from p2 point of view. I should get together with whomever from p2 team is going to work on this to decide this question (Henrik?). I think probably Pawel is correct above...that ECF httpclient should throw some sort of exception when this situation is detected (i.e. proxy authenticate), and p2 should then catch this exception and 1) log what's happening 2) disable the httpclient provider (via an API added to ECF filetransfer for disabling and/or adding providers at runtime) 3) retry (now using the JRE-based provider) > > Also, this fix could potentially require a change in Apache HTTP Client code. > I'm not sure what the policy for modifying HTTP Client code is. Does it require > an IP process? Yes, my expectation is that changes to httpclient itself would require an IP process. I don't think its necessary to modify the apache code itself, however, as even if everything is done under the covers it can probably be done strictly within the httpclient provider. > > One more thing. Like it was said before, NTLM value in the header means that > either NTLM or NTLMv2 is used. Is there any information about NTLM version available from the headers? If so, where? >If any more help from Workspace Team is needed at this point please don't >hesitate. We are going to need a test environment for testing things out. I presume that you (Pawel) are able to help out with that because I do not otherwise have access to any NTLM test environment.
(In reply to comment #91) That approach sounds good to me. I can also offer testing. Regarding the ntlm version - as far as I remember I think there is no distinction within the http header (I maybe wrong).
(In reply to comment #92) > (In reply to comment #91) > Regarding the ntlm version - as far as I remember I think there is no > distinction within the http header (I maybe wrong). You are correct, in fact there really is no "version" per-se, it's a combination of a number of things that helps you tell the difference.
(In reply to comment #93) > (In reply to comment #92) > > (In reply to comment #91) > > > Regarding the ntlm version - as far as I remember I think there is no > > distinction within the http header (I maybe wrong). > You are correct, in fact there really is no "version" per-se, it's a > combination of a number of things that helps you tell the difference. > Thanks. Even with comment 83 (which indicates that the apache httpclient provider does indeed work with NTLMv1), I'm inclined to have the httpclient disabled for *any* NTLM proxy (v1 or v2). This way all that we will have to do is detect an NTLM proxy, and not have to detect the version. It means that NTLMv1 users will not get the robustness benefits of apache httpclient provider, but it seems like a reasonable tradeoff to me. If there are any objections to this (disabling httpclient for v1 as well as v2 NTLM proxies), please say so.
(In reply to comment #94) > Thanks. Even with comment 83 (which indicates that the apache httpclient > provider does indeed work with NTLMv1), I'm inclined to have the httpclient > disabled for *any* NTLM proxy (v1 or v2). This way all that we will have to do > is detect an NTLM proxy, and not have to detect the version. +1 > > It means that NTLMv1 users will not get the robustness benefits of apache > httpclient provider, but it seems like a reasonable tradeoff to me. If there > are any objections to this (disabling httpclient for v1 as well as v2 NTLM > proxies), please say so. It's never been clear to me that the httpclient has been all that robust, especially compared to the recent JVM support for HTTP. What you are proposing seems right.
(In reply to comment #90) > I managed to setup the environment that requires clients to authenticate using > NTLM scheme. Since the information about the negotiation process is sent in > HTTP headers it isn't hard to find a place where it can be accessed. For > example: > > HttpMethodDirector.executeWithRetry(HttpMethod) line: 398 > > After execution passes this line it's possible to call: > > method.getResponseHeaders("Proxy-Authenticate"); > > And the output on my machine looks like this: > > [Proxy-Authenticate: Negotiate > , Proxy-Authenticate: Kerberos > , Proxy-Authenticate: NTLM > ] Pawel, what is the response code returned from GetMethod.execute when these response headers are present?
(In reply to comment #96) > > Pawel, what is the response code returned from GetMethod.execute when these > response headers are present? Sorry I'm not Pawel, but he is likely sleeping and I know the answer. It's either a 401 or a 407 (proxy), depending on whether it's from the proxy or not.
(In reply to comment #97) > (In reply to comment #96) > > > > Pawel, what is the response code returned from GetMethod.execute when these > > response headers are present? > > Sorry I'm not Pawel, but he is likely sleeping and I know the answer. np. I appreciate the contribution, since I'm still having to do testing without an actual system to test against. >It's > either a 401 or a 407 (proxy), depending on whether it's from the proxy or not. > I don't understand...won't Proxy-Authenticate response headers always be from an ntlm proxy? When would 401 ever be the response code (in this situation anyway)? I'm implementing the logic for detecting the situation when the httpclient provider should give up...and so need to detect any/all situations when it's an NTLM proxy out there (and httpclient provider should not handle the request).
(In reply to comment #98) > I don't understand...won't Proxy-Authenticate response headers always be from > an ntlm proxy? When would 401 ever be the response code (in this situation > anyway)? Sorry, my mistake, it will always be a 407 if it's a Proxy-Authenticate header. (I did not pay attention to the question closely enough - common problem with me). > > I'm implementing the logic for detecting the situation when the httpclient > provider should give up...and so need to detect any/all situations when it's an > NTLM proxy out there (and httpclient provider should not handle the request). > One approach would be to assume you can't handle it if you see things other than what the httpclient can handle. For example, if you don't see Basic or Digest, then you should give up (I think that's the only two they handle).
(In reply to comment #99) > (In reply to comment #98) .. > > I'm implementing the logic for detecting the situation when the httpclient > > provider should give up...and so need to detect any/all situations when it's > > an > > NTLM proxy out there (and httpclient provider should not handle the request). > > > One approach would be to assume you can't handle it if you see things other > than what the httpclient can handle. For example, if you don't see Basic or > Digest, then you should give up (I think that's the only two they handle). > I suppose one can configure an NTLM proxy server to also allow for BASIC or DIGEST authentication but presumably HttpClient would still select NTLM authentication in this case. The logic for this is in org.apache.commons.httpclient.auth.AuthChallengeProcessor.selectAuthScheme(Map) It appears that HttpClient picks them out of the box in the order NTLM, DIGEST, BASIC. I am not sure at the moment how and where to best put the code that NTLM was detected for a particular transfer...
Progress report on this issue. I've released to ECF HEAD the following additions/changes 1) An NTLMProxyDetector class to the httpclient provider, which currently looks for an NTLM proxy in the response headers (i.e. NTLM in Proxy-Authenticate as per comment 90). It currently *only* looks for NTLM in the headers (i.e. it doesn't look for the Negotiate or the Kerberos). We can easily change/add to this if through testing we find that NTLM proxies should be detected more easily. 2) Added code to the httpclient provider to use the NTLMProxyDetector to detect an NTLM proxy. With these changes, the detection of an NTLM (v1 or v2) proxy results in the httpclient provider throwing an exception (browse or incoming), and having a new httpclient-provider error code (477) that will indicate to p2 and/or other ECF filetransfer clients that the httpclient provider does not support using an NTLM proxy. I request that those with an NTLM v1 or v2 network get the code recently committed to ECF HEAD (org.eclipse.ecf.provider.filetransfer.httpclient project in /cvsroot/rt/org.eclipse.ecf/providers/bundles module) and try out the tests (in org.eclipse.ecf.tests.filetransfer in /cvsroot/rt/org.eclipse.ecf/tests/bundles). What *should* happen is that filetransfer requests (using the httpclient provider) should result in an exception (e.g. IncomingFileTransferException) with an error code of 477 when run on networks that are behind an NTLMv1 or v2 proxy. As I've said previously, I have no access to using an NTLMv1 or v2 proxy network, and so cannot myself properly test this NTLM proxy detection code. I need to depend upon those of you with such networks to inform me of whether it's doing this detection appropriately given your NTLM proxy configuration. As I said, if it's working properly, any use of the httpclient provider should result in an exception (delivered to IFileTransferListener), with an error code == 477. 3) I've also added a new OSGi service registered on start of the org.eclipse.ecf.provider.filetransfer bundle that exposes API to (at runtime) *disable* the httpclient provider. The service interface is /org.eclipse.ecf.provider.filetransfer/src/org/eclipse/ecf/provider/filetransfer/IFileTransferProtocolToFactoryMapper.java. This service interface will allow the httpclient provider to be disabled by p2/Eclipse when an NTLM proxy is detected at file transfer time by the httpclient provider (and thereby enable the future use of the JRE-based provider in environments...like Eclipse...that wish to have this be the behavior). I'm also in contact with Henrik Lindberg about the incorporation/usage of these additions within p2 to complete the circle (i.e. to catch/detect the exception from httpclient, disable the httpclient provider, and retry the request with the JRE-based provider). If there are others from p2 team that wish to be involved with implementing this integration, please let me know.
A question for Pawel: I'm assuming that *https* (ssl) requests that go through an NTLM proxy have exactly the same response headers as reported in comment 90. Would you (Pawel...or someone else that can definitively verify what NTLM proxies do) please verify that this is correct? thanks.
(In reply to comment #102) > A question for Pawel: I'm assuming that *https* (ssl) requests that go through > an NTLM proxy have exactly the same response headers as reported in comment 90. > Would you (Pawel...or someone else that can definitively verify what NTLM > proxies do) please verify that this is correct? The NTLM protocol is exactly the same for https or http.
Created attachment 145093 [details] Error dialog (In reply to comment #101) > 2) Added code to the httpclient provider to use the NTLMProxyDetector to detect > an NTLM proxy. With these changes, the detection of an NTLM (v1 or v2) proxy > results in the httpclient provider throwing an exception (browse or incoming), > and having a new httpclient-provider error code (477) that will indicate to p2 > and/or other ECF filetransfer clients that the httpclient provider does not > support using an NTLM proxy. I run ECF from HEAD with p2 and the result is in the attachement. I haven't run the tests because I'm off until end of the week.
(In reply to comment #104) > Created an attachment (id=145093) [details] > Error dialog > > (In reply to comment #101) > > 2) Added code to the httpclient provider to use the NTLMProxyDetector to detect > > an NTLM proxy. With these changes, the detection of an NTLM (v1 or v2) proxy > > results in the httpclient provider throwing an exception (browse or incoming), > > and having a new httpclient-provider error code (477) that will indicate to p2 > > and/or other ECF filetransfer clients that the httpclient provider does not > > support using an NTLM proxy. > > I run ECF from HEAD with p2 and the result is in the attachement. I haven't run > the tests because I'm off until end of the week. > Thanks. The result (477 error...as expected) looks good.
On p2 list, you asked "I would like to increment the minor version number segment (i.e. from 3.0 to 3.1) for the filetransfer bundles contributed to the platform build.". Is this for SR1 maintenance? Or Helios? (Sorry if obvious, I've just scanned bug report). If for SR1, why is the 'minor' increment required? That's a pretty big change. If for SR1, I suggest you post to cross-platform as well, once Platform responds to your question. If for Helios, then nevermind me ... you can do what you and the Platform work out. :)
(In reply to comment #106) > On p2 list, you asked "I would like to increment the minor version number > segment (i.e. from 3.0 to 3.1) for the filetransfer bundles contributed to the > platform build.". > > Is this for SR1 maintenance? Or Helios? (Sorry if obvious, I've just scanned > bug report). SR1 maintenance. I expect Helios to use some future release of ECF (not sure what number yet). > > If for SR1, why is the 'minor' increment required? That's a pretty big change. Because the new API as described in comment 101 (number 3) is a new API...and I believe that implies minor release. > > If for SR1, I suggest you post to cross-platform as well, once Platform > responds to your question. ok, but I would like not to slow things down too much if possible.
Created attachment 145538 [details] Zipped Logs of the ECF tests Here are the logs of running the ECF tests (not all) behind our customers proxy.
(In reply to comment #108) > Created an attachment (id=145538) [details] > Zipped Logs of the ECF tests > > Here are the logs of running the ECF tests (not all) behind our customers > proxy. > Thanks. Happily it looks like all requests via the httpclient provider are ending up with IncomingFileTransferException: HttpClient Provider is not configured to support NTLM proxy authentication. (or BrowseFileTransferException version). So for these ntlm proxies so far, this is correct...the httpclient's new ntlm detection (and resulting exception) seems to be working. Until I receive contrary information, I'm inclined to conclude that this ntlm proxy detector works for finding NTLM proxies...and we'll go about adding code in p2 to use this information to disable the httpclient provider and then retry (using other provider).
This is a HELP request to everyone CC'ed on this bug. There is now a binary patch that we believe addresses this problem for 3.5 and 3.5.1. The patch can be found there https://bugs.eclipse.org/bugs/attachment.cgi?id=145800 and the instructions are available at https://bugs.eclipse.org/bugs/show_bug.cgi?id=287558#c7. Please report success and failure in the bug #287558
(In reply to comment #107) > (In reply to comment #106) > > Because the new API as described in comment 101 (number 3) is a new API...and I > believe that implies minor release. > It does, especially if you know a client (outside your control) that _has_ to use the new API. But, it's also implied "there are no minor increases in maintenance release" [with out all the proper review, notifications, etc.]. So, something's gotta give. Just a question of which. > > ... I suggest you post to cross-platform as well, once Platform > > responds to your question. > > ok, but I would like not to slow things down too much if possible. I just meant to notify everyone, keep them informed that something out of the ordinary was coming. Was this related to a change in org.eclipse.core.net? That broke Riena? I've no idea ... I hear "net" I think "ECF". :)
Scott, I thought ECF was going to use provisional API for SR1 and make p2 an x-friend to avoid introducing new API in SR1.
(In reply to comment #112) > Scott, I thought ECF was going to use provisional API for SR1 and make p2 an > x-friend to avoid introducing new API in SR1. It is doing this (provisional api/x-internal with x-friends). Here's the relevant markup in org.eclipse.ecf.provider.filetransfer Export-Package: org.eclipse.ecf.provider.filetransfer;x-friends:="org.eclipse.equinox.p2.repository", (i.e. the x-friends apparently implies x-internal for all other consumers as adding both x-internal and x-friends results in warning). >I just meant to notify everyone, keep them informed that something out of the >ordinary was coming. Yes, understood. >Was this related to a change in org.eclipse.core.net? That >broke Riena? I believe there are some on this bug that have been working on org.eclipse.core.net...but it hasn't been me/us...so I'm quite certain that none of what we/ECF have done for this bug has resulted in any break. >I've no idea ... I hear "net" I think "ECF". :) Thank you for thinking that :) ...but ECF will need to have some more authority (and commit rights) before all things with 'net' in the package end up with us (and breaks become our fault/responsibility). :)
(In reply to comment #111) >Was this related to a change in org.eclipse.core.net? That broke Riena? David, could you please be more specific about that?
You don't need a minor release or increment of minor segment if only provisional (x-internal or x-friends) packages have changes.
(In reply to comment #114) > (In reply to comment #111) > >Was this related to a change in org.eclipse.core.net? That broke Riena? > > David, could you please be more specific about that? Pawel, Riena happen to deliver core.net as part of its riena.core.sdk feature because core.net is neither in equinox sdk or rcp.sdk. The version we delivered was the galileo one and I didnt do anything for SR1. Since core.net was changed for SR1 that created a conflict which I solved today. (by updating to the current core.net version) Does shed some light onto this ? Its not that Riena has to be changed, its just a version that we packaged with Riena in our delivery. christian
I'm going to give a short summary of what has transpired on this bug. 1) Initially, it was thought that the httpclient (v3.1) implementation could be made to work with NTLMv2 proxies. After investigation and attempts along this line, as well as input from a variety of sources, it has become obvious that httpclient doesn't/can't work with NTLMv2. 2) In order to provide a workaround easier on the user than the one provided in Galileo via http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies, ECF introduced a new API to allow runtime disabling of the apache httpclient provider...and the subsequent use of the UrlConnection-based provider...which does have NTLMv2 support. The testing done in response to the changes described in comment 101 shows that at least for the NTLMv2 network tested, this workaround does work. I am going to resolve this bug, with the understanding that the workaround described in 2 is working for NTLMv2 proxy users, and given that the httpclient implementation is not modifiable (as described in 1). Thanks to Francis Upton, and Henrich Kraemer for their consultation on the subtleties of NTMLv2 proxies, and to Henrik Lindberg for his assistance in implementing the workaround. Thanks to Pawel, Stefan, Henrich, and Christian for testing.
And thanks to you Scott for all the time you have spent investigating and coordinating the fix, even though you don't suffer from the problem yourself and it's really a limitation in the Apache client rather than ECF that we were dealing with.
(In reply to comment #117) > I'm going to give a short summary of what has transpired on this bug. > > 1) Initially, it was thought that the httpclient (v3.1) implementation could be > made to work with NTLMv2 proxies. After investigation and attempts along this > line, as well as input from a variety of sources, it has become obvious that > httpclient doesn't/can't work with NTLMv2. > > 2) In order to provide a workaround easier on the user than the one provided in > Galileo via > http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies, ECF > introduced a new API to allow runtime disabling of the apache httpclient > provider...and the subsequent use of the UrlConnection-based provider...which > does have NTLMv2 support. The testing done in response to the changes > described in comment 101 shows that at least for the NTLMv2 network tested, > this workaround does work. > > I am going to resolve this bug, with the understanding that the workaround > described in 2 is working for NTLMv2 proxy users, and given that the httpclient > implementation is not modifiable (as described in 1). > > Thanks to Francis Upton, and Henrich Kraemer for their consultation on the > subtleties of NTMLv2 proxies, and to Henrik Lindberg for his assistance in > implementing the workaround. Thanks to Pawel, Stefan, Henrich, and Christian > for testing. Given that Apache HTTPClient 4.x supports NTLMv2 through JCIFS can we now expet to see this reopened and fixed PROPERLY? more here: http://mail-archives.apache.org/mod_mbox/hc-dev/201002.mbox/%3C442075161.487771267003108001.JavaMail.jira@brutus.apache.org%3E
(In reply to comment #119) <stuff deleted> > > Given that Apache HTTPClient 4.x supports NTLMv2 through JCIFS can we now expet > to see this reopened and fixed PROPERLY? ECF has a multi-provider architecture, and I (Scott) have started an httpclient/httpcore-based provider (available via anonymous CVS at ecf1.osuosl.org module: plugins/org.eclipse.ecf.provider.filetransfer.httpcore). But even though it's not a lot of work, we/ECF don't have the resources to complete this provider in the near term, and so lacking contributions from either the community and/or interested member companies the answer must be no for Helios release cycle. If other people want it fixed PROPERLY (by their definition), then my suggestion would be to do some actual WORK...and contribute the PROPER fix YOURSELF.
(In reply to comment #120) > (In reply to comment #119) > <stuff deleted> > > > > > Given that Apache HTTPClient 4.x supports NTLMv2 through JCIFS can we now expet > > to see this reopened and fixed PROPERLY? > > > ECF has a multi-provider architecture, and I (Scott) have started an > httpclient/httpcore-based provider (available via anonymous CVS at > ecf1.osuosl.org module: > plugins/org.eclipse.ecf.provider.filetransfer.httpcore). > > But even though it's not a lot of work, we/ECF don't have the resources to > complete this provider in the near term, and so lacking contributions from > either the community and/or interested member companies the answer must be no > for Helios release cycle. > > If other people want it fixed PROPERLY (by their definition), then my > suggestion would be to do some actual WORK...and contribute the PROPER fix > YOURSELF. Hi Scott, thanks for prompt reply.. I guess the CAPS were a bit ungrateful :) Appreciate all the work so far, will try to tone down my questions in future and would love to contribute. Will see if I get time between contracts.. Kind Regards Ben
Using 3.6 Helios with P2, we suffer a bit from this, but with a twist: * for some people, P2, connect to sites out of box; * for others, it could not (until we found this info on the workaround) all behind the same SQUID proxy (with no password). Do you have any idea about why this could happen? With all the respect to Scott, Francis, Henrich, Pawel, Stefan, Christian, and everyone who devoted time to this - is this a <small>proper</small> fix, or a link workaround? If it's the latter, can we make it "REOPENED" so that it's not forgotten, and someone could fix it. Even if Apache doesn't fix it, would it make sense to fall back to the URLConnection provider upon failure, perhaps asking user as well (popup "Proxy connection failed, would you like to fall back to the basic connection provider?). I think that would be much user-friendlier than waiting for frustrated users to run into this bug, find that great wiki page, mess with the .ini files, etc..
(In reply to comment #122) > Using 3.6 Helios with P2, we suffer a bit from this, but with a twist: > * for some people, P2, connect to sites out of box; > * for others, it could not (until we found this info on the workaround) > all behind the same SQUID proxy (with no password). > Do you have any idea about why this could happen? I do not have any idea of why this would happen. > > With all the respect to Scott, Francis, Henrich, Pawel, Stefan, Christian, and > everyone who devoted time to this - is this a <small>proper</small> fix, or a > link workaround? It is a work around...essentially for Apache 3.1 httpclient's inability to handle NTLM v2 proxying (which, as I understand it, ultimately is a licensing issue with Microsoft, but I don't know/recall all the details about this). > If it's the latter, can we make it "REOPENED" so that it's not forgotten, and > someone could fix it. No, I don't think it's appropriate to reopen this bug, because as I understand it, this bug can't be fixed 'properly' with the Apache httpclient 3.1 provider...thus the workaround done for this bug of using the JRE provider/URLConnection fallback. I, for one, can't dedicate any more resources to investigating solutions having to do with the Apache httpclient 3.1 impl. Reopening this bug can't/won't, therefore, have any effect on my own efforts. If it can be used to marshal other/more resources from other community members then it's fine with me, but I don't frankly see how it would have that effect. Rather, it would be better (IMHO) to focus on a fix that involves implementing/completing a provider that does not depend upon Apache httpclient 3.1...e.g. Apache httpclient 4.0 (aka httpcomponents). It's my understanding that the httpcomponents *does* support NTLM v2 proxies directly...i.e. see comment 113 (although I haven't verified this myself). Work on an httpcomponents provider has been started (by me), and I believe much of the existing Apache 3.1 provider code could be reused for a 4.0-based provider...so the implementation job here is not huge for people with an understanding of the existing codebase. BUT, I can't currently commit to working on this provider. If others are willing to support it it could be done fairly readily in the ECF 3.4+ cycle, but lacking such support or direct code contributions from other community members I cannot currently commit to doing it myself. If others create contributions for the 4.0-based provider then I will, of course, assist in the testing, building, and integration. > > Even if Apache doesn't fix it, would it make sense to fall back to the > URLConnection provider upon failure, perhaps asking user as well (popup "Proxy > connection failed, would you like to fall back to the basic connection > provider?). Without the UI, this is essentially what happens now...i.e. the JRE URLConnection provider is used if the httpclients 3.1 provider encounters an NTLMv2 response. Although I understand the desire for a more aesthetic solution, this fallback strategy does work for many/most use cases as far as I can tell (I don't have access to a wide variety of proxy configurations, and there are apparently as many proxy configurations as there are people using NTLM v2 proxies). > I think that would be much user-friendlier than waiting for frustrated users to > run into this bug, find that great wiki page, mess with the .ini files, etc.. This is the workaround available given the resources. It's not optimal, but it does work. If more resources can be provided (feel free to contact me directly at slewis at composent.com to discuss this), then other alternatives can be explored. I would suggest continuation about resources and actual contributions/technical work be moved/focused on bug 251740.
(In reply to comment #122) > Do you have any idea about why this could happen? As Scott mentioned we always fall back to the URLConnection in case of NTLM detected. So this could be a result of a different Java class library used.
(In reply to comment #123) > It is a work around... Thanks for the clarification. > > If it's the latter, can we make it "REOPENED" so that it's not forgotten, and > it, this bug can't be fixed 'properly' with the Apache httpclient 3.1 > provider...thus the workaround done for this bug of using the JRE > provider/URLConnection fallback. I'd clarify that I agree a workaround is fine as long as is automatic and transparent (hidden enough from the user). To me it wasn't clear from the comments above whether this workaround was automatic & actually consisted of 0) running into the bug 1) find the link to the wiki page 2) fix up the eclipse.ini manually.. So, there should be a working fallback mechanism under the hood now in 3.6? If so, that does not work for us - we do need the manual tweak. I don't think we are hit by the NTLM issue (no passwords/authentication are needed for our SQUID proxy), maybe that's why the fallback mechanism is skipped. However, the manual tweak does work so the issue must be related to httpclient. If you do not intend to reopen this - should I open another bug then? > Although I understand the desire for a more aesthetic solution, While IMHO we *should* care about aesthetics, it's more important to avoid breaking user's flows silently (my colleague almost gave up to upgrade to 3.6 from 3.4) -- so any _ugly but useful_ solutions are welcome (the ugliest I can think of: a popup "Connection failed, please look a this wiki page: <url>" :) > this fallback strategy does work for many/most use cases as far as I can tell > does work. The automatic one does not - pls see above. > If more resources can be provided (feel free to contact me > directly at slewis at composent.com to discuss this), then other alternatives > can be explored. I would suggest continuation about resources and actual > contributions/technical work be moved/focused on bug 251740. Also, I really did not indend to give You tasks, nor wanted to deal with resourcing issues. I just simply don't agree with setting something "FIXED" until it is *fixed* - perhaps a less misleading state would be "REMIND". Now, again I just realised - that if my current problem is not specifically related to NTLM (although the symptoms and manual workaround is the same) - would you prefer a separate bug? Anyhow, thanks a million for your thorough clarification!
(In reply to comment #124) > (In reply to comment #122) > > Do you have any idea about why this could happen? > > As Scott mentioned we always fall back to the URLConnection in case of NTLM > detected. So this could be a result of a different Java class library used. We tried with a Sun JRE 1.5.0/16 and a fresh install of 1.6.0/20 - could that be a reason?
(In reply to comment #126) > We tried with a Sun JRE 1.5.0/16 and a fresh install of 1.6.0/20 - could that > be a reason? It should work with Sun Java 6 on all platforms. If you have it working on some machines and on other not then probably a separate bug should be opened (it suggests an environment issue).
Gergely, I've just read comment 125. Since it's not connected with NTLM you should definitely open a separate bug or maybe Scott can refer to an existing one.
(In reply to comment #125) <stuff deleted> > I'd clarify that I agree a workaround is fine as long as is automatic and > transparent (hidden enough from the user). > To me it wasn't clear from the comments above whether this workaround was > automatic & actually consisted of 0) running into the bug 1) find the link to > the wiki page 2) fix up the eclipse.ini manually.. > So, there should be a working fallback mechanism under the hood now in 3.6? What happens in 3.6 is: 1) When the NTLMv2 proxy asks the httpclient 3.1 provider code to authenticate (and cannot), this situation is detected (by the provider), and an exception is thrown from the ECF code. 2) The p2 download code catches this exception, and uses API that I introduced for this bug to change the provider *at runtime*. The API is this interface: org.eclipse.ecf.provider.filetransfer.IFileTransferProtocolToFactoryMapper 3) The p2 download code then retries...and the fallback/JRE URLConnection-based provider is used. Since this depends upon the JRE version and vendor, this may be the source of your problem, as per comment 124. The effect of this is that at runtime, if the default/httpclient 3.1 provider encounters an NTLMv2 proxy, it will automatically fallback and use the underlying JRE-based URLConnection provider. This is a work-around, but with more modern JREs, it effectively deals with the NTLMv2 proxy problem from httpclient 3.1. There are a lot of potential issues with proxies, however...e.g. configuration, local network topology, different/unknown types of proxies...so it's possible that this fallback strategy is getting confused in your specific case...for reasons having only to do with how your network and proxy are configured. > > If so, that does not work for us - we do need the manual tweak. I don't think > we are hit by the NTLM issue (no passwords/authentication are needed for our > SQUID proxy), maybe that's why the fallback mechanism is skipped. > However, the manual tweak does work so the issue must be related to httpclient. Could you explain what you mean by 'manual tweak'? (on a new bug probably). > > If you do not intend to reopen this - should I open another bug then? Feel free to open another bug if you think the issue you are having is actually a problem with Eclipse/ECF code, but just be aware that WRT NTLMv2 problems, the current solution (this fallback strategy as described above) is not going to change very much except through work on something like bug 251740. I would prefer that further work on NTLMv2 support/issues specifically be focused in that direction...and that's all that I'm personally going to focus on. > > > Although I understand the desire for a more aesthetic solution, > While IMHO we *should* care about aesthetics, it's more important to avoid > breaking user's flows silently (my colleague almost gave up to upgrade to 3.6 > from 3.4) -- so any _ugly but useful_ solutions are welcome (the ugliest I can > think of: a popup "Connection failed, please look a this wiki page: <url>" :) > > > this fallback strategy does work for many/most use cases as far as I can tell > > does work. > The automatic one does not - pls see above. Your explanation of what goes wrong on your network/your proxy config needs more details...e.g. the details of the proxy config and network, the specific vendor and version of the JRE that you are using, etc...because others are not seeing the issue you are seeing on NTLMv2 proxied networks...i.e. the fallback strategy *is* working for others. So it has to be something specific about your network, JRE, and/or proxy config AFAICT. Again, please use a new bug to convey these details. > > > If more resources can be provided (feel free to contact me > > directly at slewis at composent.com to discuss this), then other alternatives > > can be explored. I would suggest continuation about resources and actual > > contributions/technical work be moved/focused on bug 251740. > > Also, I really did not indend to give You tasks, nor wanted to deal with > resourcing issues. Well, I'd just like to point out to all readers of this bug that ultimately *someone* has to deal with the resource issues to deal with what appear to be edge proxy cases such as this...and if it's not the community (i.e. represented by you/Gergely in this case), then apparently it's no one. See larger 'tragedy of commons' discussion. > I just simply don't agree with setting something "FIXED" until it is *fixed* - > perhaps a less misleading state would be "REMIND". According to all the testing that has been done...as well as other NTLMv2 folks in the community...the problem expressed by this bug is fixed by the workaround. So there's nothing to remind about as far as I'm concerned. What is apparently not fixed is the problems you are experiencing...on your network, with your proxy config, in your JRE environment. > > Now, again I just realised - that if my current problem is not specifically > related to NTLM (although the symptoms and manual workaround is the same) - > would you prefer a separate bug? Yes, if it's not related to NTLMv2 proxy support I would prefer a new/separate bug...because it could be related to extant issues with the core proxy API (i.e. not ECF filetransfer), or something having to do with the JRE, or something else not having anything to do with this NTLMv2 issue...as this bug is not intended as and should not be used as a catch all for 'every problem with proxies'. Since I'm not clear on the relevant bugs, Pawel would you please direct Gergely to the extant core proxy API issues?...as I recall at least one of them are related to authentication...and I suppose it's possible that Gergely's issues are related.
Created attachment 175298 [details] Helios Proxy Preferences screenshot
Created attachment 175301 [details] Helios Proxy Issues Screenshots This zip file contains following screenshots: 1. helios_proxy_preferences.jpg - Shows the Helios proxy preferences 2. ie_proxy_preferences.JPG - Shows the IE proxy configuration 3. errordialog.JPG - Error encountered while trying to update Helios 4. errorlog.JPG - List of errors encountered while trying to update Helios 5. ecfidentityerror_stacktrace.JPG - Stacktrace of ECF Identity error encountered. 6. p2uierror_stacktrace.JPG - Stacktrace of P2 UI Error encountered.
Hi, I also feel that this issue has not been resolved. Also the workarounds are not reliably working. I am trying to update Helios through proxy, but facing proxy authentication issue. When I use 'Native' connection, the authentication indicates 'false' and username/password are blank. If I provide 'Manual' and do provide username/password, it still comes out as 407 / Proxy authentication. It doesn't seem to be a proxy blocking issue because all the update URLs can be accessed through browser. Also I can do updates to my RAD (Rational Application developer) through the update sites. I am not sure about what exact proxy the company has, but it seems to be using NTLM authentication through Microsoft ADAM. I don't have to provide username/password for proxy while browsing external websites, it is taken from ADAM transparently. I have tried all different combinations/permutations based on the earlier tickets opened on similar topics - like only Socks proxy, only HTTP/HTTPS proxy or both etc. I have also tried the solution provided at http://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies. But nothing has worked so far. I feel the primary reason for this is that username / password is not being picked up and is null when setting up proxy in 'UrlFileSystemBrowser' in the 'Native' provider. For 'Manual' provider, I am not sure why is this failing. I have attached some screenshots for the same here (Attachment #175301 [details]). Hopefully if you could suggest any solution/workaround for the same. Thanks, Sandesh.
(In reply to comment #132) <stuff deleted> > I feel the primary reason for this is that username / password is not being > picked up and is null when setting up proxy in 'UrlFileSystemBrowser' in the > 'Native' provider. For 'Manual' provider, I am not sure why is this failing. > > I have attached some screenshots for the same here (Attachment #175301 [details]). > Hopefully if you could suggest any solution/workaround for the same. Based upon an examination of your screenshots, I don't think your proxy issues are specifically associated with the behavior of the workaround created for this bug. What seems to be happening is that 1) the httpclient provider is detecting the NTLMv2 proxy and switching to the JRE-based provider 2) the JRE-based provider file system browser (class URLFileSystemBrowser in stack trace) is then failing to authenticate I believe this is not an issue with this workaround, therefore, and rather either an issue with the core proxy API (which ECF uses/consumes to get credentials) OR an interaction between the JRE Authenticator and the use of a proxy requiring authentication. On bug 295030 is a rather involved discussion of proxy handling, when multiple proxy types are involved...and authentication is required. It looks from your preferences that (as with bug 295030)...you also have a socks proxy (with authentication) in addition to an http proxy. Specifically, see Henrich's comment https://bugs.eclipse.org/bugs/show_bug.cgi?id=295030#c82 I suspect that what you are experiencing is a conflict with the java.net.Authenticator as described in Henrich's comment number 82 on 295030. This could be wrong, but it's what I suspect given your screen shots. Pawel has subsequently opened bug 312224 and 312228 against the core proxy API to deal with these issues moving forward as described by Henrich's comment. I think these bugs closer to being able to deal with your proxy issues than this bug.
