Bug 367960 - [transport] Can't download through authenticating NTLM proxy
Summary: [transport] Can't download through authenticating NTLM proxy
Status: CLOSED WONTFIX
Alias: None
Product: Equinox
Classification: Eclipse Project
Component: p2 (show other bugs)
Version: 3.7.1   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: P2 Inbox CLA
QA Contact:
URL:
Whiteboard: stalebug
Keywords:
: 386002 (view as bug list)
Depends on:
Blocks:
 
Reported: 2012-01-05 12:32 EST by Philippe Gil CLA
Modified: 2019-07-23 06:02 EDT (History)
7 users (show)

See Also:


Attachments
eclipse config (447.19 KB, text/plain)
2012-07-06 10:10 EDT, Jonathan Sartin CLA
no flags Details
eclipse errors (35.31 KB, text/plain)
2012-07-06 10:12 EDT, Jonathan Sartin CLA
no flags Details
failed update packet capture (237.66 KB, application/octet-stream)
2012-07-06 10:13 EDT, Jonathan Sartin CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Philippe Gil CLA 2012-01-05 12:32:46 EST
Build Identifier: Version: 3.7.1 Build id: M20110909-1335

When the following Eclipse preferences setting is used

General > Network Connections
..... Active Provider : Native

If the proxy requires an authentication, the following "Problem Occured" dialog is displayed, and the eclipse update process fails.

'Contacting Software Sites' has encountered a problem. Some sites could not be found. See the error log for more detail.

Details:
Some sites could not be found.  See the error log for more detail.
HTTP Proxy Authentication Required: http://download.eclipse.org/eclipse/updates/3.7/compositeContent.xml
Proxy Authentication Required
HTTP Proxy Authentication Required: http://download.eclipse.org/releases/indigo/compositeContent.xml
Proxy Authentication Required

Error log:
!SESSION 2012-01-05 18:27:20.558 -----------------------------------------------
eclipse.buildId=M20110909-1335
java.version=1.6.0_30
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=fr_FR
Command-line arguments:  -os win32 -ws win32 -arch x86_64

!ENTRY org.eclipse.equinox.p2.core 4 0 2012-01-05 18:27:35.319
!MESSAGE Provisioning exception
!STACK 1
org.eclipse.equinox.p2.core.ProvisionException: HTTP Proxy Authentication Required: http://download.eclipse.org/eclipse/updates/3.7/compositeContent.xml
	at org.eclipse.equinox.internal.p2.repository.CacheManager.createCache(CacheManager.java:189)
	at org.eclipse.equinox.internal.p2.metadata.repository.CompositeMetadataRepositoryFactory.getLocalFile(CompositeMetadataRepositoryFactory.java:73)
	at org.eclipse.equinox.internal.p2.metadata.repository.CompositeMetadataRepositoryFactory.load(CompositeMetadataRepositoryFactory.java:98)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.factoryLoad(MetadataRepositoryManager.java:57)
	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:749)
	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:651)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:96)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:92)
	at org.eclipse.equinox.p2.ui.LoadMetadataRepositoryJob.doLoad(LoadMetadataRepositoryJob.java:117)
	at org.eclipse.equinox.p2.ui.LoadMetadataRepositoryJob.runModal(LoadMetadataRepositoryJob.java:102)
	at org.eclipse.equinox.internal.p2.ui.sdk.PreloadingRepositoryHandler$2.runModal(PreloadingRepositoryHandler.java:82)
	at org.eclipse.equinox.p2.operations.ProvisioningJob.run(ProvisioningJob.java:177)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Caused by: org.eclipse.ecf.filetransfer.BrowseFileTransferException: Proxy Authentication Required
	at org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:247)
	at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69)
	... 1 more
!SUBENTRY 1 org.eclipse.equinox.p2.transport.ecf 4 1002 2012-01-05 18:27:35.320
!MESSAGE HTTP Proxy Authentication Required: http://download.eclipse.org/eclipse/updates/3.7/compositeContent.xml
!STACK 1
org.eclipse.ecf.filetransfer.BrowseFileTransferException: Proxy Authentication Required
	at org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:247)
	at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
!SUBENTRY 2 org.eclipse.ecf.identity 4 0 2012-01-05 18:27:35.320
!MESSAGE Proxy Authentication Required

!ENTRY org.eclipse.equinox.p2.core 4 0 2012-01-05 18:27:35.467
!MESSAGE Provisioning exception
!STACK 1
org.eclipse.equinox.p2.core.ProvisionException: HTTP Proxy Authentication Required: http://download.eclipse.org/releases/indigo/compositeContent.xml
	at org.eclipse.equinox.internal.p2.repository.CacheManager.createCache(CacheManager.java:189)
	at org.eclipse.equinox.internal.p2.metadata.repository.CompositeMetadataRepositoryFactory.getLocalFile(CompositeMetadataRepositoryFactory.java:73)
	at org.eclipse.equinox.internal.p2.metadata.repository.CompositeMetadataRepositoryFactory.load(CompositeMetadataRepositoryFactory.java:98)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.factoryLoad(MetadataRepositoryManager.java:57)
	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:749)
	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:651)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:96)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:92)
	at org.eclipse.equinox.p2.ui.LoadMetadataRepositoryJob.doLoad(LoadMetadataRepositoryJob.java:117)
	at org.eclipse.equinox.p2.ui.LoadMetadataRepositoryJob.runModal(LoadMetadataRepositoryJob.java:102)
	at org.eclipse.equinox.internal.p2.ui.sdk.PreloadingRepositoryHandler$2.runModal(PreloadingRepositoryHandler.java:82)
	at org.eclipse.equinox.p2.operations.ProvisioningJob.run(ProvisioningJob.java:177)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Caused by: org.eclipse.ecf.filetransfer.BrowseFileTransferException: Proxy Authentication Required
	at org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:247)
	at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69)
	... 1 more
!SUBENTRY 1 org.eclipse.equinox.p2.transport.ecf 4 1002 2012-01-05 18:27:35.467
!MESSAGE HTTP Proxy Authentication Required: http://download.eclipse.org/releases/indigo/compositeContent.xml
!STACK 1
org.eclipse.ecf.filetransfer.BrowseFileTransferException: Proxy Authentication Required
	at org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:247)
	at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
!SUBENTRY 2 org.eclipse.ecf.identity 4 0 2012-01-05 18:27:35.467
!MESSAGE Proxy Authentication Required

!ENTRY org.eclipse.equinox.p2.ui 4 1000 2012-01-05 18:27:36.044
!MESSAGE Some sites could not be found.  See the error log for more detail.
!SUBENTRY 1 org.eclipse.equinox.p2.transport.ecf 4 1002 2012-01-05 18:27:36.044
!MESSAGE HTTP Proxy Authentication Required: http://download.eclipse.org/eclipse/updates/3.7/compositeContent.xml
!STACK 1
org.eclipse.ecf.filetransfer.BrowseFileTransferException: Proxy Authentication Required
	at org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:247)
	at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
!SUBENTRY 2 org.eclipse.ecf.identity 4 0 2012-01-05 18:27:36.044
!MESSAGE Proxy Authentication Required
!SUBENTRY 1 org.eclipse.equinox.p2.transport.ecf 4 1002 2012-01-05 18:27:36.044
!MESSAGE HTTP Proxy Authentication Required: http://download.eclipse.org/releases/indigo/compositeContent.xml
!STACK 1
org.eclipse.ecf.filetransfer.BrowseFileTransferException: Proxy Authentication Required
	at org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:247)
	at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
!SUBENTRY 2 org.eclipse.ecf.identity 4 0 2012-01-05 18:27:36.044
!MESSAGE Proxy Authentication Required


Reproducible: Always

Steps to Reproduce:
1. Setup a proxy requiring authentication
2. Configure Internet Explorer to use it.
3. Configure eclipse to use native settings
4. Configure eclipse to use at least on update site
5. Ask for a "Check for updates"
Comment 1 Jonathan Sartin CLA 2012-07-06 10:10:16 EDT
Created attachment 218387 [details]
eclipse config
Comment 2 Jonathan Sartin CLA 2012-07-06 10:12:00 EDT
Created attachment 218389 [details]
eclipse errors
Comment 3 Jonathan Sartin CLA 2012-07-06 10:13:14 EDT
Created attachment 218390 [details]
failed update packet capture
Comment 4 Jonathan Sartin CLA 2012-07-06 10:14:26 EDT
Issues not just restricted to native provider:

Similar Errors seen with manual proxy configuration (Symantec Supplied NTLM authenticating squid proxy forwarding to symantec.clould web content filter). Problem Occurred dialogue reads:

"Some sites could not be found.  See the error log for more detail.
Unable to read repository at http://download.eclipse.org/webtools/repository/juno/compositeContent.jar.
Authentication failure
Unable to read repository at http://download.eclipse.org/eclipse/updates/4.2/compositeContent.jar.
Authentication failure
Unable to read repository at http://e-p-i-c.sf.net/updates/site.xml.
Authentication failure
Unable to read repository at http://download.eclipse.org/releases/juno/compositeContent.jar.
Authentication failure
Unable to read repository at http://download.eclipse.org/mylyn/releases/juno/compositeContent.jar.
Authentication failure"

Eclipse config, errors and wireshark packet capture attached.

Appears that the initial HTTP GET request sent by eclipse triggers an NTLM auth sequence between client (eclipse) and proxy. A challenge is issued by proxy to client as expected but eclipse never responds and instead issues a different HTTP HEAD request to which it receives an immediate 302 from the content filter.

Note that the content filtering service is wide open at this point, No eclipse update sites or file types are blocked.
Comment 5 Pascal Rapicault CLA 2012-11-28 15:35:03 EST
*** Bug 386002 has been marked as a duplicate of this bug. ***
Comment 6 Scott Lewis CLA 2012-11-28 16:23:33 EST
Since it seems that this bug is associated with NTLM support...

In Eclipse/p2 (3.8) the ECF filetransfer provider is using httpclient 3.1, which doesn't natively support NTLM proxies.  In order to support NTLM proxies we introduced a work around two (three?) years ago, but this work around apparently doesn't support NTLM's challenge protocol for authenticated access.

The good news is that ECF has already implemented a provider based upon httpclient 4, and the indication from the httpclient 4 docs is that NTLM is supported natively:

http://hc.apache.org/httpcomponents-client-ga/ntlm.html

If the httpclient 4 native NTLM support works fully, then it should eliminate this problem...so I think addressing this issue is best addressed simply by moving p2 to the httpclient 4-based provider.

The remaining issues are:

1) We/ECF need assistance from the community for testing this new provider prior to using it to replace the httpclient 3.1-based provider in Kepler...particularly against various proxy configurations (including but not limited to NTLM).  This need is described and discussed on this thread on p2-dev:

http://dev.eclipse.org/mhonarc/lists/p2-dev/msg04865.html

2) I don't know for certain that the httpclient 4 NTLM support actually handles this (authenticated access) use case...and have no way to test it myself...so it would be very useful if the people on this bug were to assist with testing this use case (and/or others in 1...if possible).
Comment 7 Jonathan Sartin CLA 2013-01-08 06:12:15 EST
Will assist where I can (have an NTLM auth proxy to test against). Not a regular user so not sure how to go about testing. Happy to work through this if I can have a little guidance.
Comment 8 Scott Lewis CLA 2013-01-08 18:07:17 EST
(In reply to comment #7)
> Will assist where I can (have an NTLM auth proxy to test against). Not a
> regular user so not sure how to go about testing. Happy to work through this
> if I can have a little guidance.

Hi Jonathan,

Thanks for your offer of help...it will be needed.  I've contacted the p2 project lead about coordinating testing efforts...so that we can be maximally effective in coordinating end-to-end testing.  I think we will be using bug 337449 for coordination, so would you (and anyone else willing to help testing) be willing to add yourself to that bug?

https://bugs.eclipse.org/bugs/show_bug.cgi?id=337449

Thanks.

Scott
Comment 9 Eclipse Genie CLA 2018-10-19 16:38:13 EDT
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.
Comment 10 Mat Booth CLA 2019-05-17 06:23:47 EDT
Hi all, consider trying the latest 4.12 I-builds of Eclipse from Thu, 16 May onwards:

https://download.eclipse.org/eclipse/downloads/index.html

It has a new ECF filetransfer provider implementation and we'd love to know if this improves the situation for you.

Thanks
Comment 11 Matthias Leitl CLA 2019-07-22 09:06:29 EDT
(In reply to Mat Booth from comment #10)
> Hi all, consider trying the latest 4.12 I-builds of Eclipse from Thu, 16 May
> onwards:
> 
> https://download.eclipse.org/eclipse/downloads/index.html
> 
> It has a new ECF filetransfer provider implementation and we'd love to know
> if this improves the situation for you.
> 
> Thanks

Hi.

I'm very sorry to inform you that this new implementation changed everything to the worst for me. Now even the workaround by adding

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient4

to my eclipse.ini doesn't work anymore. I had to revert back to 2019-03 (4.11) to get everything working again.

I also had a chat with the guy maintaining our proxy-server and he told me that the first connection try was working fine (downloading some setup.zip he said) but all other connection attempts (e.g. Check for updates) failed due to not providing the required crendentials (HTTP/407)

wkr
ADT
Comment 12 Matthias Leitl CLA 2019-07-23 06:02:59 EDT
(In reply to Matthias Leitl from comment #11)
> (In reply to Mat Booth from comment #10)
> > Hi all, consider trying the latest 4.12 I-builds of Eclipse from Thu, 16 May
> > onwards:
> > 
> > https://download.eclipse.org/eclipse/downloads/index.html
> > 
> > It has a new ECF filetransfer provider implementation and we'd love to know
> > if this improves the situation for you.
> > 
> > Thanks
> 
> Hi.
> 
> I'm very sorry to inform you that this new implementation changed everything
> to the worst for me. Now even the workaround by adding
> 
> -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.
> provider.filetransfer.httpclient4
> 
> to my eclipse.ini doesn't work anymore. I had to revert back to 2019-03
> (4.11) to get everything working again.
> 
> I also had a chat with the guy maintaining our proxy-server and he told me
> that the first connection try was working fine (downloading some setup.zip
> he said) but all other connection attempts (e.g. Check for updates) failed
> due to not providing the required crendentials (HTTP/407)
> 
> wkr
> ADT

Ok, nevermind.
I just found the new workaround in 
https://wiki.eclipse.org/Disabling_Apache_Httpclient45