Community
Participate
Working Groups
Some of the Mylyn builds are failing due to errors creating temp dirs: [WARNING] Error initializing: org.eclipse.tycho.p2.resolver.P2TargetPlatformResolver@4dce53ad java.lang.RuntimeException: java.io.IOException: Could not create temp dir /tmp/config2379889343146535315equinox at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.checkStarted(DefaultEquinoxEmbedder.java:301) at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.getService(DefaultEquinoxEmbedder.java:275) at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.getService(DefaultEquinoxEmbedder.java:270) at org.eclipse.tycho.p2.resolver.P2TargetPlatformResolver.initialize(P2TargetPlatformResolver.java:434) See for example https://hudson.eclipse.org/mylyn/job/gerrit-mylyn-commons/99/console
Too many files/dirs in /tmp. I've cleared some out but it seems Xtext is the culprit: hipp2:/tmp # ls -l | awk {'print $3'} | sort | uniq -c | sort -nr 25258 genie.modeling.tmf.xtext 966 genie.technology.tycho 204 genie.tools.cdt 193 genie.rt.equinox.p2 82 genie.modeling.sirius 7 genie.mylyn 1 root 1 These are objects less than one day old. Xtext project: please look into this - you're creating waaay too many files on /tmp.
We do that for a reason, as we test all kinds of different project setups, have many many compiler tests and so on. 25,000 doesn't sound like a huge number in 2014. Is the disc space the problem or is it some special tmp dir config?
Xtext HIPP was configured to write only to the separate /tmp/genie.modeling.tmf.xtext directory. So we can easily check what kind of tmp files we forgot to cleanup and wipe out the whole /tmp/genie.modeling.tmf.xtext folder if needed. Why is it not the case since Nov 29? There were a bug report about, but I can't find it any more. Adding Than Ha to CC, maybe he knows the right bug number.
(In reply to Dennis Huebner from comment #3) > Xtext HIPP was configured to write only to the separate > /tmp/genie.modeling.tmf.xtext directory. So we can easily check what kind of > tmp files we forgot to cleanup and wipe out the whole > /tmp/genie.modeling.tmf.xtext folder if needed. Why is it not the case since > Nov 29? > There were a bug report about, but I can't find it any more. https://bugs.eclipse.org/bugs/show_bug.cgi?id=421847#c22
(In reply to Sven Efftinge from comment #2) > We do that for a reason, as we test all kinds of different project setups, > have many many compiler tests and so on. Sure, but most of those files were several hours old. Are you relying on some kind of "garbage collection" to clean them up? > 25,000 doesn't sound like a huge number in 2014. We don't run Linux kernels and filestsystems that were written in 2014 :) There's s 32000 link per inode limit. Since you're not alone on that box, there are also other directories. Like Dennis said, simply create one directory under /tmp and go wild in there. But be aware of the 32000 limit. I'm seeing many files like this in /tmp: -rw-rw-r-- 1 genie.modeling.tmf.xtext modeling.tmf.xtext 6457 Feb 10 04:44 Package123761730058133186310.ecore And tons of directories like this in /tmp: drwxrwxr-x 3 genie.modeling.tmf.xtext modeling.tmf.xtext 4096 Feb 10 04:26 1392024418858-0 (both of those are now two days old)
Would you clean up tmp files owned by modeling.tmf.xtext? Can I login to the HIPP server over ssh?
Dennis, I've purged everything owned by xtext.
Could you please create /tmp/genie.modeling.tmf.xtext again with appropriate rights? FATAL: Unable to produce a script file hudson.util.IOException2: Failed to create a temp file on /home/hudson/genie.modeling.tmf.xtext/.hudson/jobs/xtend-xtext-playground/workspace at hudson.FilePath.createTextTempFile(FilePath.java:1015) at hudson.tasks.CommandInterpreter.createScriptFile(CommandInterpreter.java:120) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:62) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:54) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:34) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:646) at hudson.model.Build$RunnerImpl.build(Build.java:181) at hudson.model.Build$RunnerImpl.doRun(Build.java:136) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:434) at hudson.model.Run.run(Run.java:1390) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:40) at hudson.model.ResourceController.execute(ResourceController.java:82) at hudson.model.Executor.run(Executor.java:137) Caused by: hudson.util.IOException2: Failed to create a temporary directory in /tmp/genie.modeling.tmf.xtext at hudson.FilePath$12.invoke(FilePath.java:1001) at hudson.FilePath$12.invoke(FilePath.java:989) at hudson.FilePath.act(FilePath.java:791) at hudson.FilePath.act(FilePath.java:773) at hudson.FilePath.createTextTempFile(FilePath.java:989) ... 12 more Caused by: java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createTempFile(File.java:1879) at hudson.FilePath$12.invoke(FilePath.java:999) ... 16 more Terminating xvnc.
Oh sorry about that. It's created. hipp2:/tmp> ls -l | grep xtext drwxrwxr-x 2 genie.modeling.tmf.xtext modeling.tmf.xtext 4096 Feb 12 10:18 genie.modeling.tmf.xtext
(In reply to Dennis Huebner from comment #8) > Could you please create /tmp/genie.modeling.tmf.xtext again with appropriate > rights? Just an FYI, if you restart your HIPP instance it will create /tmp/<project> directory as part of the startup scripts.
(In reply to Thanh Ha from comment #10) > (In reply to Dennis Huebner from comment #8) > > Could you please create /tmp/genie.modeling.tmf.xtext again with appropriate > > rights? > > Just an FYI, if you restart your HIPP instance it will create /tmp/<project> > directory as part of the startup scripts. Good to know. I also added -Djava.io.tmpdir=/tmp/genie.modeling.tmf.xtext/ to ANT_ARGS ANT_OPTS JAVA_ARGS and JVM_OPTS. Is is possible to connect to hipp2 from build.eclipse.org? If I try to login with my user account (dhubner) my regular password is not accepted.
(In reply to Dennis Huebner from comment #11) > (In reply to Thanh Ha from comment #10) > > (In reply to Dennis Huebner from comment #8) > > > Could you please create /tmp/genie.modeling.tmf.xtext again with appropriate > > > rights? > > > > Just an FYI, if you restart your HIPP instance it will create /tmp/<project> > > directory as part of the startup scripts. > > Good to know. I also added -Djava.io.tmpdir=/tmp/genie.modeling.tmf.xtext/ > to ANT_ARGS ANT_OPTS JAVA_ARGS and JVM_OPTS. > Is is possible to connect to hipp2 from build.eclipse.org? If I try to login > with my user account (dhubner) my regular password is not accepted. That's a good point. I'll add it to my todo to set -Djava.io.tmpdir as standard on the Hudson image. I'm curious can the global key-value pairs take in variables as parameters? If so the startup script exports the variable JAVA_TMPDIR [1]. Perhaps it would be better to set it this way instead: -Djava.io.tmpdir=$JAVA_TMPDIR Assuming Hudson lets us define the global variables using variables. [1] https://hudson.eclipse.org/xtext/systemInfo
> Is is possible to connect to hipp2 from build.eclipse.org? If I try to login > with my user account (dhubner) my regular password is not accepted. It isn't, sorry. Since we're much more permissive with the hipp user accounts, we're trying to keep the rest of the server on a short leash.
(In reply to Denis Roy from comment #13) > > Is is possible to connect to hipp2 from build.eclipse.org? If I try to login > > with my user account (dhubner) my regular password is not accepted. > > It isn't, sorry. Since we're much more permissive with the hipp user > accounts, we're trying to keep the rest of the server on a short leash. It would be handy for my to have access. At the moment I have to create some additional shell build steps where I can run my commands. And I think hudson have even more permissions as I have.
(In reply to Dennis Huebner from comment #14) > It would be handy for my to have access. At the moment I have to create some > additional shell build steps where I can run my commands. And I think hudson > have even more permissions as I have. Can't you create the scripts in your git repo and then when HIPP checks it out it can run the scripts? or did I misunderstand what you're trying to do?
(In reply to Thanh Ha from comment #15) > (In reply to Dennis Huebner from comment #14) > > It would be handy for my to have access. At the moment I have to create some > > additional shell build steps where I can run my commands. And I think hudson > > have even more permissions as I have. > > Can't you create the scripts in your git repo and then when HIPP checks it > out it can run the scripts? or did I misunderstand what you're trying to do? I'm trying to own hipp2 :) I don't have to put my scripts into the git repo, adding a new shell build step in hudson is enough. But if you try to look around, find things and so on, it becomes annoying.
(In reply to Thanh Ha from comment #12) > I'm curious can the global key-value pairs take in variables as parameters? > > If so the startup script exports the variable JAVA_TMPDIR [1]. Perhaps it > would be better to set it this way instead: > > -Djava.io.tmpdir=$JAVA_TMPDIR > > Assuming Hudson lets us define the global variables using variables. > > [1] https://hudson.eclipse.org/xtext/systemInfo The answer to this question is, yes. Looks like I can pass $JAVA_TMPDIR here. I'd suggest using this variable to get the temp dir path so that if we ever change the URL in the scripts it'll get the correct directory.
> But if you try to look around, find things and so > on, it becomes annoying. The hipp servers are identical to build. Really. Same OS, same filesystems in the same locations. I'm not sure what you're expecting to see since your build jobs are run from a user account you can't access anyway.
(In reply to Thanh Ha from comment #17) > (In reply to Thanh Ha from comment #12) > > I'm curious can the global key-value pairs take in variables as parameters? > > > > If so the startup script exports the variable JAVA_TMPDIR [1]. Perhaps it > > would be better to set it this way instead: > > > > -Djava.io.tmpdir=$JAVA_TMPDIR > > > > Assuming Hudson lets us define the global variables using variables. > > > > [1] https://hudson.eclipse.org/xtext/systemInfo > > The answer to this question is, yes. Looks like I can pass $JAVA_TMPDIR > here. I'd suggest using this variable to get the temp dir path so that if we > ever change the URL in the scripts it'll get the correct directory. I updated the HIPP deployment image to include this parameter so future HIPP instances should get this.
From the Webmaster inbox: https://hudson.eclipse.org/cdt/job/cdt-8.3/ is failing with java.lang.RuntimeException: java.io.IOException: Could not create temp dir /tmp/config4139587641969896754equinox CDT and Xtext share an underlying host so just for grins I ran Denis scriptlet from yesterday: 33597 genie.modeling.tmf.xtext 662 genie.technology.tycho 288 genie.tools.cdt 193 genie.rt.equinox.p2 112 genie.modeling.sirius I've removed all of the Xtext files and directories. Than has updated the JAVA_TMPDIR and restarted Xtexts instance. -M.
That should have been: Thanh has updated the JAVA_TMPDIR and restarted Xtexts instance. No idea who that Than guy is. -M.
(In reply to Thanh Ha from comment #10) > (In reply to Dennis Huebner from comment #8) > > Could you please create /tmp/genie.modeling.tmf.xtext again with appropriate > > rights? > > Just an FYI, if you restart your HIPP instance it will create /tmp/<project> > directory as part of the startup scripts. After restart following directory was created: drwxrwxr-x 2 genie.modeling.tmf.xtext signers 4096 Feb 14 10:58 genie.modeling.tmf.xtext I think the right group should be modeling.tmf.xtext and not signers.
(In reply to Dennis Huebner from comment #22) > (In reply to Thanh Ha from comment #10) > > (In reply to Dennis Huebner from comment #8) > > > Could you please create /tmp/genie.modeling.tmf.xtext again with appropriate > > > rights? > > > > Just an FYI, if you restart your HIPP instance it will create /tmp/<project> > > directory as part of the startup scripts. > > After restart following directory was created: > drwxrwxr-x 2 genie.modeling.tmf.xtext signers 4096 Feb 14 10:58 > genie.modeling.tmf.xtext > > I think the right group should be modeling.tmf.xtext and not signers. You're right. While I do set the group if root restarts the process the script doesn't do it if a project resets it themselves. I updated the script to use "chgrp" when it's not root that started the process. I will deploy the script to all the hipp instances later today and should be fixed next time a project restarts their HIPP.
(In reply to Thanh Ha from comment #23) > I will deploy the script to all the hipp instances later today and should be > fixed next time a project restarts their HIPP. The updated script is now deployed and while I was there I started investigating and I think the java.io.tmpdir setting isn't helping us here. If I run "ls genie.*" I see all of the HIPPs are not using this space for their files (see below for sample output). It seems to me like all the Hudson tmp files are being created in this directory however project's specific jobs are still being created in /tmp. Either the individual jobs are not taking the setting or this is not the right setting. So I think this issue at this point is still not resolved. I will try to take a closer look later when I get a chance but if there's any Java experts that know how these jobs might be deciding where to create the temp files please feel free to jump in. Also is there a specific job I can clone to do some investigating work on my own, this would help greatly? Sample OUTPUT on HIPP2: # ls genie.* genie.modeling.sirius: jna7895955209275944775.tmp libs_jetty-http.jar9084422637028206612hudson libs_jetty-servlet-api.jar2679101633081975733hudson libs_jetty-web-app.jar5209691728181064137hudson libs_hudson-jetty-war-executable.jar7885356596628144658hudson libs_jetty-io.jar2430719531128580063hudson libs_jetty-servlet.jar8978703770030473852hudson libs_jetty-xml.jar4937824272175264917hudson libs_jetty-continuation.jar923231115637338083hudson libs_jetty-security.jar6453392106016647714hudson libs_jetty-util.jar2273015318089396098hudson libs_jetty.jar8568619246917316083hudson genie.modeling.tmf.xtext: jna3657845435170512288.tmp libs_jetty-http.jar5048764900253368135hudson libs_jetty-servlet-api.jar5444736652688696565hudson libs_jetty-web-app.jar1882986539285267815hudson libs_hudson-jetty-war-executable.jar314723378793146536hudson libs_jetty-io.jar2238461415982208068hudson libs_jetty-servlet.jar4574065702503415252hudson libs_jetty-xml.jar5772197082526897039hudson libs_jetty-continuation.jar6691913261443698538hudson libs_jetty-security.jar3916801324267335130hudson libs_jetty-util.jar5325417265358690640hudson libs_jetty.jar3302053016769374318hudson genie.mylyn: jna5033221658204536457.tmp libs_jetty-http.jar6765557460395532492hudson libs_jetty-servlet-api.jar3111006513106529736hudson libs_jetty-web-app.jar3569121364001757629hudson libs_hudson-jetty-war-executable.jar298445513104689909hudson libs_jetty-io.jar2352485724402829310hudson libs_jetty-servlet.jar1756029849213123175hudson libs_jetty-xml.jar5614854226675069816hudson libs_jetty-continuation.jar6111554201254389885hudson libs_jetty-security.jar8427330434722997656hudson libs_jetty-util.jar654871136297877078hudson libs_jetty.jar2094407466489050808hudson
Just a datapoint... Seems xtext is not cleaning up after itself as far as I can tell it creates folders like with numbers which I guess it's generating somehow like: 1392745239037-0 Each of these files seem to have a subdirectory inside such as "my" or "foo". "foo" directories are typically empty and "my" directories usually have "test" or "outer" directories in them. These directories in turn are empty. Whatever job from xtext that is creating these directories seem to be cleaning up files they are creating but for whatever reason aren't cleaning up the directories they are creating. In my opinion I think Xtext should fix these jobs to cleanup. A good test Framework should always remove everything it creates after the test completes in my opinion.
I ran some tests using Tycho project since I creates some tmp files as a test. What I found is that even though we set JVM_OPTs and ANT_OPTs etc... individual builder plugins might still ignore these settings. For example Maven 3 builder as far as I can tell does not add those options on as part of it's build (or at least ignores the tmpdir part). If I go to the job level configuration for a Maven build I can set MAVEN_OPTS to have -Djava.io.tmpdir= explicitly and it'll use the /tmp/genie.technology.cbi directory for the temp files. However if I set the MAVEN_OPTs globally as an environment variable Maven doesn't take affect. Or if I set this option in the Global builder default setting for some reason it doesn't get copied over to the job. So the only way to get this setting as far as I can tell to stick is to navigate to every individual Hudson job if they are using Maven 3 and make sure they have this setting. Unfortunately I don't think this is maintainable. On the plus side I think the tmp files Maven creates are not that many so it's not actually a big problem in my opinion. Even if we fix it for Maven though, not all projects use Maven and thus the builders for those other projects likely also ignore the default Java settings we configure. In the case of Xtext this appears to be using the Buckminster Hudson plugin. I can see that the Xtext team tried to set the Buckminster plugin to set the tmpdir setting in the global settings page but I think it's getting overwritten by the job configuration where they set additional JVM options. Considering we are still seeing the files created in /tmp I think it's safe to conclude that this setting is having no effect, maybe configuring it explicitly in the job config will help? Looking at Xtext which seems to be the project creating the most files and not cleaning up as I explained in comment 24 I think it's a bug in Xtext that's not fully cleaning up the temp directories it's creating. You can see traces that it at least tried to clean up but did not finish and left empty directories. Considering this I think the focus should be to fix inside whatever it is that Xtext is running to clean up after itself fully. Moving the tmpdir around won't really resolve the main problem anyway which is that the Xtext jobs are creating way too many files.
> Also is there a specific job I can clone to do some investigating work on my own, this would help greatly? Jobs which are producing this temp files are our test jobs. You can pick for example: https://hudson.eclipse.org/xtext/view/Xtext-Xtend/job/xtext-xtend-tests/ > Seems xtext is not cleaning up after itself as far as I can tell it creates > folders like with numbers which I guess it's generating somehow like: > > 1392745239037-0 This is the typical temp folder name if you are using guava temp file API. And we do. > In my opinion I think Xtext should fix these jobs to cleanup. A good test > Framework should always remove everything it creates after the test completes in my > opinion. This is also our opinion and we are working on that issue. (In reply to Thanh Ha from comment #26) > settings we configure. In the case of Xtext this appears to be using the > Buckminster Hudson plugin. I can see that the Xtext team tried to set the > Buckminster plugin to set the tmpdir setting in the global settings page but > I think it's getting overwritten by the job configuration where they set > additional JVM options. Considering we are still seeing the files created in > /tmp I think it's safe to conclude that this setting is having no effect, > maybe configuring it explicitly in the job config will help? We noticed the that buckminster doesn't get the right settings. As you can see, tmpdir parameter is already passed as it should: [workspace] $ /shared/common/jdk1.6.0-latest/bin/java -Dbuckminster.output.root=/home/hudson/genie.modeling.tmf.xtext/.hudson/jobs/xtext-xtend-tests/workspace/buckminster.output -Dbuckminster.temp.root=/home/hudson/genie.modeling.tmf.xtext/.hudson/jobs/xtext-xtend-tests/workspace/buckminster.temp -Djava.io.tmpdir=$JAVA_TMPDIR The problem is still, that one can execute "java" executable (programmatically or using some build-system plugin) without having java.io.tmpdir property set. > Considering this I think the focus should be to fix inside whatever it is > that Xtext is running to clean up after itself fully. You are right, especially when suggested solution doesn't work at the moment. We fixed the tmp files leaks we found. We also forced a new tmp file creation API for our tests. But we, and nobody, can't guaranty that some tmp files may be produced and not cleaned up. > ... Moving the tmpdir > around won't really resolve the main problem anyway which is that the Xtext > jobs are creating way too many files. Separate tmp dir location should help us investigate potential tmp file spots. As I already said nobody can't guaranty that their test-code is bug-free. And the most important is, that we can sweep the whole folder without disturbing other hipp2 projects.
I think we're done here. We moved xtext to a new server and I noticed /tmp seems to be pretty clean. I'm guess xtext resolved the issue in the tests.
