Bug 544852 - Releases were & are built/executed/tested/released in the context of insecure/untrusted code
Summary: Releases were & are built/executed/tested/released in the context of insecure...
Status: CLOSED MOVED
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: Vulnerability Reports (show other bugs)
Version: unspecified   Edit
Hardware: All All
: P3 major (vote)
Target Milestone: ---   Edit
Assignee: Security vulnerabilitied reported against Eclipse projects CLA
QA Contact:
URL:
Whiteboard:
Keywords: security
Depends on:
Blocks:
 
Reported: 2019-02-26 17:41 EST by Jonathan Leitschuh CLA
Modified: 2021-12-23 06:44 EST (History)
12 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Leitschuh CLA 2019-02-26 17:41:01 EST
Eclipse Security Team,
 
This email is to responsibly disclose a widespread security vulnerability in several of the Eclipse Repositories.
 
CWE-829: Inclusion of Functionality from Untrusted Control Sphere
CWE-494: Download of Code Without Integrity Check
 
If this vulnerability has been exploited by anyone, it could have massive downstream impacts on the rest of the Eclipse ecosystem.

Related issue: https://bugs.eclipse.org/bugs/show_bug.cgi?id=444350


Impact Locations
----------------

I've been searching through GitHub repositories looking for places where build infrastructure (either Gradle, Maven, or other tools) are downloading code over HTTP and then executing it.


These are the locations that I found in your organization's builds:

XText-XTend:

- https://github.com/eclipse/xtext-xtend/blob/a59579814e881550e043f886fc62acc52d077616/gradle/bootstrap-setup.gradle#L4-L15


Brit:

- https://github.com/eclipse/birt/blob/638a21a72f1c0b3695070f1dd2128723db6f502e/pom.xml#L28-L39
- https://github.com/eclipse/birt/blob/638a21a72f1c0b3695070f1dd2128723db6f502e/pom.xml#L48-L59
- https://github.com/eclipse/birt/blob/638a21a72f1c0b3695070f1dd2128723db6f502e/pom.xml#L68-L79


Orion:

- https://github.com/eclipse/orion.client/blob/3b9390bfff0afb1f7efabc30396c2336eee393ed/pom.xml#L91-L95


Gef:

- https://github.com/eclipse/gef/blob/c28a7e9018aebec1627ad68f552c2ffd8bb842f0/org.eclipse.gef.releng/environment/Mars.p2f#L3-L25

Sw360:
- https://github.com/eclipse/sw360/blob/e06538b6461b0549002b45acec0b8b1333641063/rest/authorization-server/pom.xml#L137-L146
- https://github.com/eclipse/sw360/blob/e06538b6461b0549002b45acec0b8b1333641063/rest/resource-server/pom.xml#L261-L270

Hawkbit:
- https://github.com/eclipse/hawkbit/blob/7870f9f87f12c047a3ee818fb0ea5edeb95bc432/pom.xml#L122-L127

XText-Maven:
- https://github.com/eclipse/xtext-maven/blob/6d94f6cf58240779283b7e5983ea07d76b404435/org.eclipse.xtext.maven.parent/pom.xml#L287-L304

Vorto:
- https://github.com/eclipse/vorto/blob/a48e4c2bc9d3c75d14d2b0807a0ca158e5769466/pom.xml#L105-L114


This is not a complete list. This is all I was able to find using a few simple search queries using GitHub's fuzzy search. I'm also not very good at looking at ANT buildscripts and determining if a specific URL is being used to resolve dependencies or not.

This wildcard query may help your team find more of these than I could report here:
- https://github.com/search?l=Maven+POM&q=org%3Aeclipse+repositories&type=Code

WARNING! If any of the Eclipse builds are using a shared `~/.gradle` or `~/.m2` cache between builds and any of these downloads were maliciously compromised, the compromised jar may remain inside of cache directory and continue to be used in the future.


This isn't just theoretical
---------------------------

POC code has existed since 2014 to maliciously compromise a JAR file inflight.
See:
https://max.computer/blog/how-to-take-over-the-computer-of-any-java-or-clojure-or-scala-developer/
https://github.com/mveytsman/dilettante


MITM Attacks Increasingly Common
--------------------------------

See:
https://thenextweb.com/insights/2017/12/11/comcast-continues-to-inject-its-own-code-into-websites-you-visit/
https://serverfault.com/a/153065
https://security.stackexchange.com/a/12050


Fix & Disclosure
----------------

At a minimum, all of these code locations where artifacts are downloaded from an untrusted source needs to be fixed. I don't know what the Elastic team wants to do about versions that are already released.
 
Additionally, CVE numbers need to be issued for these vulnerabilities after they are responsibly fixed.


This responsible disclosure follows Google's 90-day vulnerability disclosure policy (I'm not an employee of Google, I just like their policy).
https://www.google.com/about/appsecurity/
Comment 1 Mikaël Barbero CLA 2019-02-27 14:07:50 EST
Thanks for your report.

(In reply to Jonathan Leitschuh from comment #0)
> Related issue: https://bugs.eclipse.org/bugs/show_bug.cgi?id=444350

This has been fixed and http:80 is now automatically redirected to https:443 so no content is served from download.eclipse.org without secured TLS connection.
 
> XText-XTend:
> 
> -
> https://github.com/eclipse/xtext-xtend/blob/
> a59579814e881550e043f886fc62acc52d077616/gradle/bootstrap-setup.gradle#L4-L15

This is an issue that should be discussed with the xtext project.

> Brit:
> 
> -
> https://github.com/eclipse/birt/blob/
> 638a21a72f1c0b3695070f1dd2128723db6f502e/pom.xml#L28-L39
> -
> https://github.com/eclipse/birt/blob/
> 638a21a72f1c0b3695070f1dd2128723db6f502e/pom.xml#L48-L59
> -
> https://github.com/eclipse/birt/blob/
> 638a21a72f1c0b3695070f1dd2128723db6f502e/pom.xml#L68-L79

Fixed by bug 444350

> Orion:
> 
> -
> https://github.com/eclipse/orion.client/blob/
> 3b9390bfff0afb1f7efabc30396c2336eee393ed/pom.xml#L91-L95

Fixed by bug 444350

> Gef:
> 
> -
> https://github.com/eclipse/gef/blob/c28a7e9018aebec1627ad68f552c2ffd8bb842f0/
> org.eclipse.gef.releng/environment/Mars.p2f#L3-L25

Fixed by bug 444350

> Sw360:
> -
> https://github.com/eclipse/sw360/blob/
> e06538b6461b0549002b45acec0b8b1333641063/rest/authorization-server/pom.
> xml#L137-L146
> -
> https://github.com/eclipse/sw360/blob/
> e06538b6461b0549002b45acec0b8b1333641063/rest/resource-server/pom.xml#L261-
> L270

http://repo.spring.io is always redirected (HTTP 302) to TLS secured connection.

> Hawkbit:
> -
> https://github.com/eclipse/hawkbit/blob/
> 7870f9f87f12c047a3ee818fb0ea5edeb95bc432/pom.xml#L122-L127

This is an issue that should be discussed with the Hawkbit project

> XText-Maven:
> -
> https://github.com/eclipse/xtext-maven/blob/
> 6d94f6cf58240779283b7e5983ea07d76b404435/org.eclipse.xtext.maven.parent/pom.
> xml#L287-L304

Same as above (Xtext project)

> Vorto:
> -
> https://github.com/eclipse/vorto/blob/
> a48e4c2bc9d3c75d14d2b0807a0ca158e5769466/pom.xml#L105-L114

Fixed by bug 444350

> WARNING! If any of the Eclipse builds are using a shared `~/.gradle` or
> `~/.m2` cache between builds and any of these downloads were maliciously
> compromised, the compromised jar may remain inside of cache directory and
> continue to be used in the future.

We don't do that.
 
> This isn't just theoretical
> ---------------------------
> 
> POC code has existed since 2014 to maliciously compromise a JAR file
> inflight.
> See:
> https://max.computer/blog/how-to-take-over-the-computer-of-any-java-or-
> clojure-or-scala-developer/
> https://github.com/mveytsman/dilettante

Well, this is about maven central serving content over plain, non TLS, connection. They fixed that since then (https://www.infoq.com/news/2014/08/Maven-SSL-Default). Same as we did with bug 444350. Note that we were less exposed to such issue because most jars we serve are actually digitally signed (https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html), greatly mitigating a MITM attack.

> MITM Attacks Increasingly Common
> --------------------------------
> 
> See:
> https://thenextweb.com/insights/2017/12/11/comcast-continues-to-inject-its-
> own-code-into-websites-you-visit/
> https://serverfault.com/a/153065
> https://security.stackexchange.com/a/12050
> 

> Fix & Disclosure
> ----------------
> 
> At a minimum, all of these code locations where artifacts are downloaded
> from an untrusted source needs to be fixed. I don't know what the Elastic
> team wants to do about versions that are already released.

Well, we're not the Elastic team... 

Each Eclipse project is responsible for its build scripts. There is nothing we can do to prevent that. What we did is to enable TLS on download.eclipse.org as you noted in bug 444350. I guess next step is to reach out to Xtext and Hawkbit.
Comment 2 Jonathan Leitschuh CLA 2019-02-28 17:08:54 EST
> This has been fixed and http:80 is now automatically redirected to https:443 so no content is served from download.eclipse.org without secured TLS connection.

If the initial connection is made over http:80 a malicious MITM can keep the connection downgraded even though the artifact server automatically redirects to https:443. The malicious attacker proxies the HTTP traffic via a MITM thus leaving any connection initially made over http:80 vulnerable.


Unless Gradle/Maven know how to understand the Strict-Transport-Security header or there is something else I'm not aware of, bug 444350 does not fix this for all of these projects.

Even if they do respect the Strict-Transport-Security header, the timeout on the header is 15552000 seconds (180 days) so if that long passes before the release server performs a build, that release is vulnerable to a malicious MITM.


> Note that we were less exposed to such issue because most jars we serve are actually digitally signed (https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html), greatly mitigating a MITM attack.


Unless I'm mistaken, I don't believe that Maven or Gradle checks digital signatures.
Evidence:
Gradle: https://discuss.gradle.org/t/pgp-verify-plugin-how-to-verify-dependency-signatures/17516
Maven: https://stackoverflow.com/a/25302527/3708426

I highly doubt that ANT does either, but I have no evidence either way. Also, if you have a MITM, why not just strip off the signature? I know that by default, Gradle and Maven accept unsigned JAR files without hesitation.

Also, even if they are signed and Maven has updated to check signatures, does it compare the signature with the coordinates declared? Are the POM files are MITMed, an additional dependency could be injected that was malicious.

> Well, we're not the Elastic team... 

My mistake, bad copy and paste from the email I sent them.

> I guess next step is to reach out to Xtext and Hawkbit.

Should I just open an issue on their Github projects?

> Each Eclipse project is responsible for its build scripts. There is nothing we can do to prevent that.

Regular automated audits could be used to prevent this. After the security teams that I'm currently in contact with have fixed their builds I'm going to publicly disclose this issue (it's indusry wide). I'm going to open proposals to Maven, Gradle and SBT to require users to explicitly opt-out of using HTTPS for artifact resolution. That should also help prevent this in the future.

Additional
----------
There is also a concern that maliciously downloaded code that is executed in the context of a release could be used to comprimise artifacts, thus creting a worm hidden in the bytecode of the released JAR file. The only way to detect this is to compare the hash of the JAR files published with a known trusted build created after these fixes are applied.

Thanks,
Jonathan Leitschuh
Comment 3 Mikaël Barbero CLA 2019-03-06 12:43:53 EST
(In reply to Jonathan Leitschuh from comment #2)
> > This has been fixed and http:80 is now automatically redirected to https:443 so no content is served from download.eclipse.org without secured TLS connection.
> 
> If the initial connection is made over http:80 a malicious MITM can keep the
> connection downgraded even though the artifact server automatically
> redirects to https:443. The malicious attacker proxies the HTTP traffic via
> a MITM thus leaving any connection initially made over http:80 vulnerable.

Correct. 

> > Note that we were less exposed to such issue because most jars we serve are actually digitally signed (https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html), greatly mitigating a MITM attack.
> 
> 
> Unless I'm mistaken, I don't believe that Maven or Gradle checks digital
> signatures.
> Evidence:
> Gradle:
> https://discuss.gradle.org/t/pgp-verify-plugin-how-to-verify-dependency-
> signatures/17516
> Maven: https://stackoverflow.com/a/25302527/3708426

I was talking about the usage inside Equinox, which checks the jar signature, not the pgp signature of maven artifacts. Signed jars cannot be MITM. Also, equinox warn if installing unsigned jar. But the issue remains for downstream user who don't use the projects jars in the context of Equinox.

> > I guess next step is to reach out to Xtext and Hawkbit.
> 
> Should I just open an issue on their Github projects?

I think you should do that for each project you are mentioning here. I would let Wayne confirm that to you before you actually do it. Thanks.


> I'm going to open
> proposals to Maven, Gradle and SBT to require users to explicitly opt-out of
> using HTTPS for artifact resolution. That should also help prevent this in
> the future.

That's great.
Comment 4 Jonathan Leitschuh CLA 2019-03-21 12:56:55 EDT
Still waiting on confirmation from Wayne? Should I move forward with the public report?
Comment 5 Wayne Beaton CLA 2019-03-21 14:16:57 EDT
(In reply to Mikaël Barbero from comment #3)
> I think you should do that for each project you are mentioning here. I would
> let Wayne confirm that to you before you actually do it. Thanks.

+1

(In reply to Jonathan Leitschuh from comment #4)
> Should I move forward with the
> public report?

+1
Comment 6 Christian Dietrich CLA 2019-03-26 13:55:05 EDT
@Wayne, @Michael:

how to prevent mirrors to be used with http:

[INFO] Fetching org.hamcrest.core_1.3.0.v201303031735.jar.pack.gz from http://ftp.osuosl.org/pub/eclipse/releases/oxygen/201804111000/plugins/ (19.33kB)
Comment 7 Christian Dietrich CLA 2019-03-26 13:56:05 EDT
and it would have been simple to mention us here to get it fixed before going public
Comment 8 Christian Dietrich CLA 2019-03-26 14:02:10 EDT
or should we disable p2 mirrors for builds?
Comment 9 Christian Dietrich CLA 2019-03-29 06:34:26 EDT
anybody here?
Comment 10 Sebastian Zarnekow CLA 2019-04-02 08:27:36 EDT
(In reply to Christian Dietrich from comment #6)
> @Wayne, @Michael:
> 
> how to prevent mirrors to be used with http:
> 
> [INFO] Fetching org.hamcrest.core_1.3.0.v201303031735.jar.pack.gz from
> http://ftp.osuosl.org/pub/eclipse/releases/oxygen/201804111000/plugins/
> (19.33kB)

Mikaël, Wayne,

any recommendation on this question? Should mirrors be disabled? Should the foundation remove all mirrors that are not using https?
Comment 11 Mikaël Barbero CLA 2019-04-02 08:57:45 EDT
cc'ing Denis.

(In reply to Sebastian Zarnekow from comment #10)
> any recommendation on this question? Should mirrors be disabled? Should the
> foundation remove all mirrors that are not using https?

I guess that disabling mirrors during your builds would mitigate the risk. 

Denis, what's your take on disabling mirrors with no https?
Comment 12 Denis Roy CLA 2019-04-02 09:25:16 EDT
(In reply to Mikaël Barbero from comment #1)
> This has been fixed and http:80 is now automatically redirected to https:443
> so no content is served from download.eclipse.org without secured TLS
> connection.

This is technically incorrect. We don't force https on download.e.o, we have an HSTS policy that requires the use of https for those clients that adhere to HSTS.

(In reply to Mikaël Barbero from comment #11)
> I guess that disabling mirrors during your builds would mitigate the risk. 

+1 for disabling mirrors at build time

 
> Denis, what's your take on disabling mirrors with no https?

I have mixed feelings about it. Digital Signing > https any day, as it is still possible to MITM https (every proxy server on the planet is a MITM). Even our mirrors pull their bits from us using unencrypted rsync.

Even then -- mirrors are maintained by third parties. Once the bits are on their server hard drives, who's to say what happens to them. The ideal solution would be to eliminate mirrors entirely, but since they serve about 90% of our downloads, that would cost us $$$$$.
Comment 13 Christian Dietrich CLA 2019-04-03 06:05:26 EDT
what can we do about

https://www.eclipse.org/downloads/download.php?protocol=https&file=/orion/drops/R-17.0-201801111044/built-codeEdit.zip&r=1

this one redirects to http on mirrors too
Comment 14 Denis Roy CLA 2019-04-03 09:03:25 EDT
I've never even tied the redirect with the "protocol" parameter.
Comment 15 Christian Dietrich CLA 2019-04-03 09:04:05 EDT
protocol=https does not help
Comment 16 Christian Dietrich CLA 2019-04-03 09:04:29 EDT
i tried to fix it that way.
Comment 17 Jonathan Leitschuh CLA 2019-04-03 10:22:34 EDT
> Should the foundation remove all mirrors that are not using https?

I believe this is an appropriate step.

I also advise that the eclipse foundation consider dropping support for resolving dependencies from the various Eclipse artifact servers over HTTP.

I've also advised Sonatype (Maven Central) and JFrog (JCenter) to consider rolling out a plan to drop HTTP support by 2020.

I believe that in order to fully protect users, connections on port 80 (HTTP) should be blocked, not redirected.

A redirect merely hides the problem. Blocking port 80 will expose this to users as the vulnerability that it is.
Comment 18 Jonathan Leitschuh CLA 2019-04-03 10:22:59 EDT
> Should the foundation remove all mirrors that are not using https?

I believe this is an appropriate step.

I also advise that the eclipse foundation consider dropping support for resolving dependencies from the various Eclipse artifact servers over HTTP.

I've also advised Sonatype (Maven Central) and JFrog (JCenter) to consider rolling out a plan to drop HTTP support by 2020.

I believe that in order to fully protect users, connections on port 80 (HTTP) should be blocked, not redirected.

A redirect merely hides the problem. Blocking port 80 will expose this to users as the vulnerability that it is.
Comment 19 Jonathan Leitschuh CLA 2019-05-06 10:11:14 EDT
Hi all,

Just checking in to see if there is interest in pursuing the block of port 80.

Maven Central has just announced that they will be blocking port 80 access on January 15th, 2020.

https://central.sonatype.org/articles/2019/Apr/30/http-access-to-repo1mavenorg-and-repomavenapacheorg-is-being-deprecated/

Thanks,
Jonathan Leitschuh
Comment 20 Jonathan Leitschuh CLA 2019-05-14 16:07:17 EDT
Hi all,

The following issues were filed publicly on May 6th about Orion and Brit. There hasn't been any followup by any of the maintainers.

https://bugs.eclipse.org/bugs/show_bug.cgi?id=547008
https://bugs.eclipse.org/bugs/show_bug.cgi?id=547007

Thanks,
Jonathan Leitschuh
Comment 21 Frederic Gurr CLA 2021-12-23 06:44:59 EST
This issue has been migrated to https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/417.