Re: [jgit-dev] Flaky tests in the master branch

[Antonin Delpeuch <antonin@xxxxxxxxxxx>]

Hi Matthias,

Thanks for your reply!
On 10/04/2024 23:51, Matthias Sohn wrote:
> Some of the tests (e.g. RacyGitTests) depend on timing which may
> depend on hardware and OS.
> Appreciate any improvement with flaky tests. AFAIK all JGit committers
> use either MacOS or Linux.

Would it make sense to disable or even remove those tests? I could
submit a patch for that. Of course it would be better to fix the tests
so that they keep testing the intended functionality but pass reliably.
While embarking on a journey to debug a race condition in tests which
cover functionality I am unfamiliar with sounds like an interesting
challenge, it's unlikely I can afford to take time for that unfortunately.

Given that this seems to be a hurdle for other new contributors (not
just for me), I would say that the detrimental effect of those flaky
tests on contributor experience may outweigh the potential benefit of
them catching a regression in the future. But it's not an easy call to
make for sure.

By the way, I took the time to write up a blog post about my overall
experience contributing to JGit, in case you are interested (it's
something I do for other projects too, I don't mean to single JGit out):

https://antonin.delpeuch.eu/posts/contribution-experience-report-jgit/

To summarize, the main hurdles I encountered were:

- difficulty to identify where to submit issues and patches

- Eclipse making changes to tracked editor config files

- unreliable test suite

I hope it provides interesting insights :) Thanks a lot for your work on
this library!

Best,

Antonin