Community
Participate
Working Groups
After upgrade to 3.0.0 our hudson server stopping to respond after several days(4-7 days) of runtime. hudson.err.log contains messages like: 29.01.2013 04:36:30 hudson.triggers.SafeTimerTask run SCHWERWIEGEND: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@383e8192 failed java.lang.OutOfMemoryError: Java heap space 29.01.2013 04:36:38 hudson.triggers.SafeTimerTask run SCHWERWIEGEND: Timer task hudson.model.LoadStatistics$LoadStatisticsUpdater@4a89cfe4 failed java.lang.OutOfMemoryError: Java heap space 29.01.2013 04:36:54 hudson.triggers.SafeTimerTask run SCHWERWIEGEND: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@383e8192 failed java.lang.OutOfMemoryError: Java heap space Exception in thread "JmDNS(buildmaster.local.).State.Timer" Exception in thread "qtp542081238-1653" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 2013-01-29 04:41:35.701:WARN:oejs.AbstractConnector: java.lang.OutOfMemoryError: Java heap space Exception in thread "JmDNS(buildmaster.local.).Timer" java.lang.OutOfMemoryError: Java heap space i am not sure if this is a problem of hudson itself or of one of the installed plugins, so i have add the dump params on outofmemory: 2013-01-29 08:22:07 - Starting java -Xrs -Xmx768m -XX:MaxPermSize=128M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\temp\hudson_dumps -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "d:\hudson\hudson.war" --httpPort=8080 --ajp13Port=-1 i have analyized the dump, it seems all the memory is wasted in the class org.hudsonci.xpath.impl.Dom2Dom, field weakhashmap named "cache". It seems this WeakHashMap dont' do what it have to do, it prevents the objects from being GCed... May be hudson keeps somewhere strong references to the KEYS, which would prevent WeakHashMap from freeing the Values stored in the weakhashmap for GC? The memory dump is very big(~1gb) so i dont really want to upload it. Please let me know if you need more information.
Hi Waldemar, thanks for the report and also looking at the heap dump to find at the probable culprits. Could also let us know the plugins you have.
(In reply to comment #1) Bob could you please take a look at this. I changed the priority to P1
Copy&Paste from plugin manager: Copy To Slave Plugin null 1.4.1-h-1 Disable Downgrade to 1.4 Hudson MSBuild Plugin 1.14-h-1 Disable Downgrade to 1.11 Build Pipeline Plugin This plugin provides build pipeline functionality to Hudson and Jenkins. This allows a chain of jobs to be visualised in a new view. Manual jobs in the pipeline can be triggered by a user with the appropriate permissions manually confirming. Enable Publish Over CIFS 0.2 Disable Hudson REST Plugin This plugin adds REST support to Hudson 2.1.2 Disable Copy Artifact Plugin Adds a build step to copy artifacts from another project. 1.12 Disable SSH plugin You can use the SSH Plugin to run shell commands on a remote machine via ssh. 1.1 Disable Groovy Support Plugin 3.0.3 Disable Maven 2 job type plugin This plug-in is needed if you have legacy maven 2 job type. Preferred way to use maven in Hudson is by using maven as a builder in your Free Style job, which is provided by Maven 3 plugin. 2.2.2 Disable Downgrade to 2.2.1 Unpin Hudson Subversion Plug-in This plugin provides support to use Subversion as an SCM in a Hudson job 2.3.6-h-1 Disable Downgrade to 2.3.4 Unpin Batch Task Plugin This plugin adds batch tasks that are not regularly executed to projects, such as releases, integration, archiving, etc. 1.14 Disable Active Directory plugin With this plugin, you can configure Hudson to authenticate the username and the password through Active Directory. 1.17 Disable Hudson BIRT Charts Plugin This Plugin provides graphing support to Hudson using BIRT Chart Engine. 3.0.3 Disable ChuckNorris Plugin Displays a picture of Chuck Norris (instead of Hudson the butler) and a random Chuck Norris 'The Programmer' fact on each build page. 0.4 Disable Hudson :: Maven 3 :: SNAPSHOT Monitor 3.0.0 Enable Downgrade to 2.2.0 Maven 3 Build Plugin This plugin adds the Maven 3 build step to Hudson. 3.0.0 Enable Hudson CVS Plug-in Integrates Hudson with CVS SCM 2.2.0 Disable Downgrade to 2.1.0_1 Unpin Hudson SSH Slaves plugin Plugin allows you to manage slaves running on *nix machines over SSH. 3.0.2 Disable Downgrade to 2.1.1 Hudson Downstream build view This plug-in allows you to view the full status all the downstream builds so that we can graphically see that everything for this build has been completed successfully. 1.9-h-1 Disable Downgrade to 1.5 Hudson Parameterized Trigger plugin This plugin lets you trigger new builds when your build has completed, with various ways of specifying parameters for the new build. You can add multiple configurations: each has a list of projects to trigger, a condition for when to trigger them (based on the result of the current build), and a parameters section. 2.17-h-1 Disable Downgrade to 2.4 CIFS-Publisher Plugin This plugin uploads build artifacts to repository sites using CIFS (SMB) protocol. 1.1.10 Enable Hudson GIT plugin Integrates Hudson with GIT SCM 2.2.1-h-1 Disable Downgrade to 2.2.0 Unpin Favorite This plugin allows you to mark a job a favorite. This is controlled via a list view column you need to add to a view. You can then click on a star to favorite or unfavorite a job. There is also a job filter to allow you to create a view that only shows your favorites. 1.10-h-1 Disable XPath Provider Plugin XPath Service Provider for Jelly 1.0.1 Disable JNA Native Support Plugin This plugin provides support for Hudson to so some OS specific actions using Java Native Access interface. 3.0.2 Disable Sonar Plugin The plugin enables to trigger Sonar analyses from Jenkins/Hudson. It offers two different ways to trigger the Sonar analyses: Build step to trigger the analyses with the Sonar Runner Post-build action to trigger the analyses with Maven 2.0.1 Disable Downgrade to 2.0 Timestamper Adds timestamps to the Console Output. 1.2.1 Disable Token Macro Plugin This plugin adds reusable macro expansion capability for other plugins to use. 1.6-h-1 Disable Hudson Maven Release Plugin This plugin allows you to perform releases using the maven-release-plugin from Hudson to a repository. 0.10.0-h-1 Disable Downgrade to 0.9.1 Hudson Artifactory Plugin Integrates Artifactory to Hudson 2.1.3-h-2
Looking into the bug. Do you have the xpath-provider-plugin installed? If you do, this bug shouldn't happen.
Never mind, I see you do have xpath-provider installed. Thanks for the thorough bug report!
I see you have installed: XPath Provider Plugin XPath Service Provider for Jelly 1.0.1 Could you please update to 1.0.2? It appears that Plugin Central wasn't properly updated to the latest version.
Hello Bob, thanks for quick reply. I have updated this plugin to 1.0.2 and have restarted hudson. I'll let you know if i'll get OutOfMemory or not. It took about one week of runtime before the OutOfMemory appeared (i saw it most of the time on monday after weekend), so please give me some time.
Waldemar any update on this?
The last restart was on 30 Januar, now we have 5 Februar, so about 5 days are gone and the hudson is still running. Its eating about 550 mb right now according to the taskmanager and the memory consumption seems not to grow up. It looks like the update of the XPath Provider Plugin did help, so you can close the bug.
btw, it would be nice if you could add memory info (total/max/free memory) to the systemInfo page.
(In reply to comment #10) Yes, we can add the memory info to system log. Could you file an enhancement request (with importance set to enhancement) However, the memory information may not be accurate, if Hudson is deployed as a war in a container (tomcat) and some other wars are also deployed along with it
9 days uptime, the memory consumption has dropped from from 550 mb to ~400 mb according to task manager. I would say the problem was solved by updating the xpath-provider plugin.
Glad to hear that. Thanks.
