Community
Participate
Working Groups
In 565796 we identified that there is a transitive dependency to older JNA from ECF (org.eclipse.ecf.provider.filetransfer.httpclient45.win32 -> org.apache.httpcomponents.httpclient.win -> com.sun.jna;version="[4.4,5)" Can the version range in httpcomponents simply be increased, the latest httpcomponents still uses JNA 4.5.2. https://search.maven.org/artifact/org.apache.httpcomponents/httpclient-win/4.5.12/jar - see parent for version info https://search.maven.org/artifact/org.apache.httpcomponents/httpcomponents-client/4.5.12/pom
New Gerrit change created: https://git.eclipse.org/r/c/orbit/orbit-recipes/+/167745
What's the status on this one? This bug still prevents JNA 5.x to be used throughout simrel. Current version is org.apache.httpcomponents.httpclient.win_4.5.13 which still has upper [4.4.0,5) as range.
(In reply to Sebastian Ratz from comment #2) > What's the status on this one? The change I submitted is waiting on a review and merge: https://git.eclipse.org/r/c/orbit/orbit-recipes/+/167745 Once that is approved a new orbit build with the increased range can be rolled out, and then ECF can target that version, and then Platform can consume the updated ECF, and then Platform won't need to contribute JNA 4.5.x anymore to simrel. At that point it may be that others are relying on 4.5.x, so we can address them then?
New Gerrit change created: https://git.eclipse.org/r/c/ecf/org.eclipse.ecf/+/179005
See also https://bugs.eclipse.org/bugs/show_bug.cgi?id=565796#c5
Gerrit change https://git.eclipse.org/r/c/ecf/org.eclipse.ecf/+/179005 was merged to [master]. Commit: http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/commit/?id=589731456f608a3e87e2e8152671637a33556ebc
ECF no longer has any direct references to com.sun.jna.platform.win32, thanks to https://git.eclipse.org/r/c/ecf/org.eclipse.ecf/+/179005 which has been merged. But we have been requested to use httpclient 4.5.13 from orbit here: https://download.eclipse.org/tools/orbit/downloads/drops/R20210223232630/ which...unfortunately...still has dep in manifest: com.sun.jna.platform.win32;version="[4.4,5)" So I believe the next step is to get httpclient 4.5.13 so that it can use com.sun.jna.platform.win32 to be able to use version 5 of com.sun.jna.*. By looking at httpclient 4.5.13 it seems that it allows usage of com.sun.jna version 5, so I hope this will simply be an expansion of the Orbit bundle manifest version range for httpclient.win32 Once this change is made to the Orbit bundles, and the fix is available via a new Orbit repo, we/ECF should be notified and we will consume that new version of httpclient, include it in our contribution to platform, and get it out in 2021-06 (hopefully).
Sorry for not reacting sooner. The change in https://git.eclipse.org/r/c/ecf/org.eclipse.ecf/+/179005 looks good. I have had JNA 5 working with the httpclient code before, but will explicitly test with the latest JNA 5 version. Unfortunately my NTLM test env didn't survive migration to my current computer, so I'm currently in the process of recreating it. I hope to be able to run a test by the weekend.
So I managed to get a test env up again. Unfortunately, the result isn't that great: httpclient-win 4.5.13 is NOT compatible with jna 5.x. There were some incompatible changes in the SSPI API that is used in httpclient, specifically how Sspi.SecBufferDesc works, see https://github.com/java-native-access/jna/blob/master/CHANGES.md: > com.sun.jna.platform.win32.Sspi.SecBufferDesc was incompatibly changed > to match the correct native semantics. SecBufferDesc describing more than > one buffer were broken. For most usecases > com.sun.jna.platform.win32.SspiUtil.ManagedSecBufferDesc is the best > alternative. We would have to either a) migrate away from httpclient 4 to httpclient 5, or b) backport the changes in WindowsNegotiateScheme to httpclient 4, but that looks like a total rewrite to me, or c) stay at JNA 4 (or d) remove the httpclient-win functionality, but that would mean breaking everyone in a Windows domain that uses authenticated proxied)
I had another look at WindowsNegotiateScheme in httpclient 5. It's not as dissimilar as I thought at first look. We could provide a possible patched-up version of a httpclient-win 4.5.x bundle that is compatible with JNA 5. But is that even a road that Orbit wants to go down?
(In reply to Carsten Reckord from comment #10) > But is that even a road that Orbit wants to go down? No. I don't think it is a road we should go down of maintaining a fork of this bundle.
(In reply to Carsten Reckord from comment #10) > I had another look at WindowsNegotiateScheme in httpclient 5. It's not as > dissimilar as I thought at first look. I was going to offer to put the httpclient 5 into Orbit - but it doesn't exist in Maven Central. Perhaps I am looking in the wrong place.
My comments: Good news: I did start the creation of an httpclient5-based provider some time ago. It's not pushed yet, but at least I started it. Bad News: I have not tried/run it yet...or even compiled against httpclient 5-as-bundle (Orbit) yet, so especially given it's a major revision I'm more than a little concerned that they might have deprecated (or removed) things the ECF code could still be using. Another problem is that whatever ECF/p2/Platform does will be forced on all the simrel projects...so that could cause hardship to other simrel projects. Obviously someone will need to coordinate that. Once in Orbit I will be able to give it a try (or before that if you point me at something close to the binaries in bundle form). *If* no breaking changes/code removals then I will still need support from Platform + Carsten and possibly others for testing prior to a real release. But at this point...without a real commitment from Platform and others...I can't commit to having this done before 5-28-21 (api and feature freeze release for Platform). I agree that we don't want to maintain a httpclient 4 in Orbit that's hacked to support JNA 5...but...for the 2021-06 release (only) this might be acceptable if the changes are not too large. I wouldn't advise betting everything on httpclient 5 + jna 5 until its in Orbit and I've confirmed that it doesn't break filetransfer too badly.
(In reply to Scott Lewis from comment #13) > My comments: > > Good news: I did start the creation of an httpclient5-based provider some > time ago. It's not pushed yet, but at least I started it. Oops...I did push it, it's here: https://git.eclipse.org/c/ecf/org.eclipse.ecf.git/tree/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5
Except for the httpclient-win bundle, it is already in Orbit. Just the naming has changed a bit: Maven Central Coordinates: org.apache.httpcomponents.core5:httpcore5 org.apache.httpcomponents.client5:httpclient5 org.apache.httpcomponents.client5:httpclient5-win Bundles: org.apache.httpcomponents.client5.httpclient5 org.apache.httpcomponents.core5.httpcore5 I can add the httpclient5-win bundle. Downstream bundles would need to migrate, since the API has changed in a bunch of places and all classes have been moved to new packages.
(In reply to Scott Lewis from comment #13) > Bad News: I have not tried/run it yet...or even compiled against httpclient > 5-as-bundle (Orbit) yet, so especially given it's a major revision I'm more > than a little concerned that they might have deprecated (or removed) things > the ECF code could still be using. > ... > I wouldn't advise betting everything on httpclient 5 + jna 5 until its in > Orbit and I've confirmed that it doesn't break filetransfer too badly. Anything I can help out with? > Another problem is that whatever > ECF/p2/Platform does will be forced on all the simrel projects...so that > could cause hardship to other simrel projects. Obviously someone will need > to coordinate that. I would be able to migrate MPC and might be able to help out with Oomph and Userstorage. But obviously that's just the tip of the iceberg > Once in Orbit I will be able to give it a try (or before that if you point > me at something close to the binaries in bundle form). *If* no breaking > changes/code removals then I will still need support from Platform + Carsten > and possibly others for testing prior to a real release. > I agree that we don't want to maintain a httpclient 4 in Orbit that's hacked > to support JNA 5...but...for the 2021-06 release (only) this might be > acceptable if the changes are not too large. I've spent the evening creating a version that works with both JNA 4 and 5. Changes are pretty minimal. I'll push it to my GitHub in case we do this.
The modified httpclient 4.x code is here: https://github.com/creckord/httpcomponents-client. Only the httpclient-win needs a new version.
(In reply to Carsten Reckord from comment #17) > The modified httpclient 4.x code is here: > https://github.com/creckord/httpcomponents-client. Only the httpclient-win > needs a new version. Ok. I'll work over the next few days to try out the ECF httpclient5 provider (with package rename, etc) on httpclient5 from Orbit. Then we should have an idea of whether 5 can be used for 2021-06 or we will have to use the modified 4.x code. >Anything I can help out with? Yes. I started out trying to use the httpclient5 bundles from this Orbit repo: https://download.eclipse.org/tools/orbit/downloads/drops/R20210223232630/ But I ran into problems trying to add this to my target platform. I added these three to target: org.apache.httpcomponents.client5.httpclient5 org.apache.httpcomponents.core5.httpcore5 org.apache.httpcomponents.core5.httpcore5-h2 Good news is: I've already done the package refactoring and the filetransfer provider compiles (which is a good sign for possible deprecation/removal of old code in httpclient 5). But when I go to run/test I get dependency errors for httpcore5-h2: javax.net.ssl org.conscrypt It seems that there is this bundle for conscrypt in orbit: org.conscrypt.conscrypt-openjdk-uber when I add that bundle to the target platform it has no effect...and I see by examination of the conscript manifest that it doesn't export any packages...so that explains the no effect. In any event, I would as that someone would create a target platform with Orbit httpclient5 and dependencies...with all necessary versions. Hopefully such a httpclient5-in-Orbit target exists and I just need to be pointed to it...that would be a help. Also, I'm confused by the dependency error for javax.net.ssl. That package is imported explicitly by the httpclient5 filetransfer provider, so I don't know why it wouldn't match up in httpcore5-h2, but maybe there's a version issue or some such. Any advice about that would be helpful also.
(In reply to Scott Lewis from comment #18) > But I ran into problems trying to add this to my target platform. > ... > In any event, I would as that someone would create a target platform with > Orbit httpclient5 and dependencies...with all necessary versions. > Hopefully such a httpclient5-in-Orbit target exists and I just need to be > pointed to it...that would be a help. I'll give it a shot
(In reply to Carsten Reckord from comment #19) > (In reply to Scott Lewis from comment #18) > > But I ran into problems trying to add this to my target platform. > > ... > > In any event, I would as that someone would create a target platform with > > Orbit httpclient5 and dependencies...with all necessary versions. > > Hopefully such a httpclient5-in-Orbit target exists and I just need to be > > pointed to it...that would be a help. > > I'll give it a shot Through looking at the httpclient5 jars from Orbit build: https://download.eclipse.org/tools/orbit/downloads/drops/R20210223232630/ I see this in the manifest.mf of org.apache.httpcomponents.core5.httpcore5-h2_5.0.2.v20201015-2258.jar: under import package: javax.net.ssl;resolution=optional,org.cons crypt;resolution=optional this is incorrect..these should be (with colons) javax.net.ssl;resolution:=optional,org.cons crypt;resolution:=optional I've created Orbit bug 573231. If anyone has Orbit committer permissions please push through this small but critical fix so that httpclient5 filetransfer testing can proceed. Would someone please submit the appropriate but for Orbit project
(In reply to Scott Lewis from comment #20) > Would someone please submit the appropriate but for Orbit project Done. org.apache.httpcomponents.core5.httpcore5-h2_5.0.2.v20210429-1809.jar in https://download.eclipse.org/tools/orbit/downloads/drops/I20210429200932/ has the corrected manifest.
(In reply to Jonah Graham from comment #21) > (In reply to Scott Lewis from comment #20) > > Would someone please submit the appropriate but for Orbit project > > Done. > > org.apache.httpcomponents.core5.httpcore5-h2_5.0.2.v20210429-1809.jar in > https://download.eclipse.org/tools/orbit/downloads/drops/I20210429200932/ > has the corrected manifest. Thanks. I have verified that this fixes the dependency problem...as these optional dependencies are now seen by the osgi resolver as optional as they should be. And I have additional good news. With the above fix, I've been able to successfully run the ECF filetransfer test suite from localhost using the httpclient5 provider located here: https://download.eclipse.org/tools/orbit/downloads/drops/I20210429200932/repository This is very good news as it means that httpclient5 didn't remove any deprecated classes wrt httpclient45 and so a simple package renaming (mostly) was all that was necessary for the httpclient5 provider to function. It also means that if we can get 1 an 2 below done in time that we should be able to have the httpclient5 provider for 2021-06. What remains to happen is: 1) If httpclient5 has an equivalent to httpclient45's org.apache.httpcomponents.httpclient.win fragment, then that bundle must be added to Orbit. 2) a new org.eclipse.ecf.provider.filetransfer.httpclient5.win32 fragment that uses/depends upon 1 must be created. I imagine this new fragment can be mostly based upon the httpclient45.win32 fragment Carsten is the expert on that. Since Carsten created/implemented/tested in appropriate win32 environment, then I think he will have to do the same for the httpclient5 provider. I'm happy to open bugs for 1 or 2 if desired, but won't be able to do that work myself. Carsten or other please let me know how they want to handle this. While Carsten and/or others are working on the above, I will put together the releng metadata (feature, maven build stuff, etc) for the new httpclient5 provider. Finally, someone should propose on cross projects mailing list that Simrel move to httpclient5 for 2021-06...since ECF/p2/Platform are doing so and to prevent difficulties there has only been one impl of httpclient in simrel repos at once. Other simrel projects that depend upon httpclient will therefore need to move to 5 and since 5 changes package names it won't be a drop-in replacement for them (i.e. refactoring and recompile needed).
Oops, the httpclient5 source code is here (in ECF repo): https://git.eclipse.org/c/ecf/org.eclipse.ecf.git/tree/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5
FWIW: I've created an ECF httpclient5 provider feature, modified the target platoform that's currently being used to build ECF (to include the apache httpclient5/core5 bundles from Orbit integration build referenced in comment 21 ), added this new feature to the ECF build, and produced a snapshot repo here: https://download.eclipse.org/rt/ecf/snapshot/site.p2/3.14.21.v20210502-0038/ the feature id is: org.eclipse.ecf.filetransfer.httpclient5 This feature does *not* yet include the win32 fragment (as the httpclient45 has) because first the two tasks in comment 22 must be completed (the apache win32 jar in Orbit...as a fragment)...and the o.e.e.provider.filetransfer.httpclient5.win32 has to be created and added to ecf git repo. Once that's ready I'll add these to the ECF httpclient5 feature.
(In reply to Scott Lewis from comment #24) > FWIW: I've created an ECF httpclient5 provider feature, modified the target > platoform that's currently being used to build ECF (to include the apache > httpclient5/core5 bundles from Orbit integration build referenced in comment > 21 ), added this new feature to the ECF build, and produced a snapshot repo > here: > > https://download.eclipse.org/rt/ecf/snapshot/site.p2/3.14.21.v20210502-0038/ > > the feature id is: org.eclipse.ecf.filetransfer.httpclient5 meant the feature id is: org.eclipse.ecf.filetransfer.httpclient5.feature
By my reading of the platform plan release milestones, 5/28 is RC1...api and feature freeze. Effectively...for 'in the wild' testing, that means httpclient5 needs to be part of the M3 release on 5/21. That means that ECF has to make the filetransfer contribution by EOD 5/17. To meet these deadlines I need to have everything ready to go for building, testing, and packaging by 5/13. That doesn't leave much time to get the httpclient win32 bundle into Orbit, get the new win32 fragment that depends on it into ECF, test these in proxy environment (win32 with NTLM proxy at least) and get added to feature, etc as described by comment 22. Also...with httpclient in simrel we need to make sure that *all simrel projects that use httpclient* are willing to use the new httpclient5. This may already be a problem for some projects, as httpclient5 requires source changes (package renaming, etc). We've had trouble with multiple versions of httpclient in simrel before, so this should be avoided. If all of the above can't be done in time, then ECF will continue to contribute httpclient45 for 2021-06 (with dep on 4.4 JNA) and not contribute a newer version of filetransfer -> httpclient45 -> JNA 4.
Since afaik nothing has happened wrt the two needed changes described in comment 22 I am going to build and release to Platform a new version of ECF filetransfer based upon httpcomponents 4.5 for inclusion into Platform 2021-06. Although the gerrit changes submitted (and applied/committed) update the version range to allow JNA 5, the analysis in comment 9 means that this build of ECF filetransfer (and therefore platform and therefore simrel) will still have a dependency on JNA 4.5...because...as described in comment 7 org.eclipse.ecf.provider.filetransfer.httpclient45.win32 -> org.apache.httpcomponents.httpclient.win 4.5 -> com.sun.jna 4.5 As per comment 22 org.apache.httpcomponents.httpclient.win 5 is not yet in Orbit, and ECF doesn't yet have a replacement org.eclipse.ecf.provider.filetransfer.httpclient5.win32 (httpclient5 requires source level changes due to package renames in httpclient/httpcore 5). This means it can't be contributed to platform in current state, without removing the win32-specific proxy (NTLM2) support. So, for the monday contribution, ECF will still use httpclient45.win32, apache httpclient 4.5 win, and JNA 4.5. FWIW, I have done the changes to create an httpclient5-based provider, and successfully tested using the ECF filetransfer test suite. See comment 22 . A snapshot of ECF 3.14.21 is available here https://download.eclipse.org/rt/ecf/snapshot/site.p2/3.14.21.v20210512-0614/ This repo has (and will have) *both* httpclient 45 provider and the (incomplete...without httpclient5.win32,jna 5 deps) httpclient5 provider.
Scott, would you please open the cq for httpclient5-win on behalf of ecf (only committer can do it)? Once approved I'll get it in orbit.
(In reply to Alexander Kurtakov from comment #28) > Scott, would you please open the cq for httpclient5-win on behalf of ecf > (only committer can do it)? Once approved I'll get it in orbit. I've created this ecf.filetransfer issue to take over/clarify steps: https://bugs.eclipse.org/bugs/show_bug.cgi?id=576564 Here's the CQ for httpclient5-win http://dev.eclipse.org/ipzilla/show_bug.cgi?id=23784
What is the status here? Is using JNA 5.8.0 in JGit a problem for Eclipse 2021-12 on Windows or not? Installing a JGit using JNA 5.8.0 into a downloaded Eclipse 2021-12-M1 EPP on Windows is definitely possible. I just get an installation with both JNA 4.5.1 and 5.8.0 present, the JGit stuff relying on JNA works, and I didn't notice any problems. But could it be a problem for the EPP builds? Or should I just not worry about it? (JGit plans to ship SSH agent integration, using JNA for the communication; for now on Windows shared memory communication with Pageant.)
(In reply to Thomas Wolf from comment #30) > What is the status here? Is using JNA 5.8.0 in JGit a problem for Eclipse > 2021-12 on Windows or not? The newer JNA is not a problem - CDT uses it too. The issue is only that we will be shipping two versions of JNA until the dependency from ECF -> httpcomponents 4.x -> JNA 4.x is resolved. > Installing a JGit using JNA 5.8.0 into a downloaded Eclipse 2021-12-M1 EPP > on Windows is definitely possible. I just get an installation with both JNA > 4.5.1 and 5.8.0 present, the JGit stuff relying on JNA works, and I didn't > notice any problems. > > But could it be a problem for the EPP builds? Or should I just not worry > about it? Not a problem at all - the C/C++ EPP package has had both for a while now with no reported ill effects. > > (JGit plans to ship SSH agent integration, using JNA for the communication; > for now on Windows shared memory communication with Pageant.) :-)
Thanks for the heads-up, Jonah! (In reply to Jonah Graham from comment #31) > The newer JNA is not a problem - CDT uses it too. The issue is only that we > will be shipping two versions of JNA until the dependency from ECF -> > httpcomponents 4.x -> JNA 4.x is resolved. Oh, that's all? Good to know. > > But could it be a problem for the EPP builds? Or should I just not worry > > about it? > > Not a problem at all - the C/C++ EPP package has had both for a while now > with no reported ill effects. Glad to hear that. So we don't have to worry about it.
> (In reply to Jonah Graham from comment #31) > > The newer JNA is not a problem - CDT uses it too. The issue is only that we > > will be shipping two versions of JNA until the dependency from ECF -> > > httpcomponents 4.x -> JNA 4.x is resolved. To sum it up: - The httpcomponents 4.x component can't use JNA 5.x, so JNA 4.x needs to stay around until everyone has migrated to httpcomponents 5.x - There are a whole bunch of other projects besides ECF depending on httpcomponents in SimRel currently, although other than ECF and MPC nobody seems to care about the Win32 bit that brings in the JNA dependency: com.google.http-client.google-http-client org.eclipse.epp.mpc org.eclipse.jgit org.eclipse.json org.eclipse.linuxtools.docker.core org.eclipse.mylyn org.eclipse.oomph org.eclipse.passage.lic.json org.eclipse.php org.eclipse.rcptt.testrail org.eclipse.rcptt.zephyr org.eclipse.userstorage org.eclipse.wst org.glassfish.jersey.apache.connector org.mandas.docker-client > > > But could it be a problem for the EPP builds? Or should I just not worry > > > about it? > > > > Not a problem at all - the C/C++ EPP package has had both for a while now > > with no reported ill effects. > > Glad to hear that. So we don't have to worry about it. I can confirm that having both JNA versions and both HttpClient versions in the platform should work without problem. The HttpClient 4/5 bundles and packages have distinct coordinates, so that much was expected. And all current JNA users have proper version constraints to select one or the other. The only exceptions to this that I could find, are both only in the Orbit, but not currently in Platform or SimRel, namely assertj 3.20.2 and an ancient Cassandra 0.7.0. Ultimately, I think it would be good to reach the point where we can ship just one JNA version to avoid wiring issues from carelessly defined dependencies, but at the moment, all appears to be in order.
The Eclipse Orbit project is now on GitHub at https://github.com/eclipse/orbit If this issue is still relevant, please create an issue (and PR :-) on GitHub. This notice is only going to the 17 bugzilla's that have been changed in the last ~18 months or so to avoid inundating everyone's inbox with long out of date issues. Please see https://bugs.eclipse.org/bugs/buglist.cgi?product=Orbit&query_format=advanced&resolution=--- for all the unresolved Orbit bugs
