Re: [jetty-users] Need help with finding logs when Jetty killed

[<p-naitou@xxxxxxxx>]

Dear Silvio

Thank you for your reply.

I didn't think about kernel logs.
I will check more logs (kernel output and others).

Thanks,
Toshimichi Naito

________________________________________
差出人: jetty-users-bounces@xxxxxxxxxxx <jetty-users-bounces@xxxxxxxxxxx> が  の代理で送信Silvio Bierman <sbierman@xxxxxxxxxxxxxxxxxx>
送信日時: 2017年7月25日 18:08
宛先: jetty-users@xxxxxxxxxxx
件名: Re: [jetty-users] Need help with finding logs when Jetty killed

Dear Toshimichi,

I would start with checking /var/log/syslog and look for a kernel panic.
Under extreme situations the kernel may decide it needs to kill one or
more resource hungry processes. I have seen many Java processes being
killed because of out of memory situations, especially if the maximum
heap size (-Xmx...) is too large considering the available physical
memory and possible memory demands from other applications.

Lacking that the only situations where a Java server process would
suddenly die on me was when I employed a third party library that
decided to call System#exit under some error conditions. That was quite
a puzzler...

Good luck,

Silvio


On 25-07-17 10:47, p-naitou@xxxxxxxx wrote:
> Hi,
>
> We are using Jetty 9.3.9.v20160517.
> Sometimes, Jetty's process killed.
>
> We checked following logs, but couldn't find the cause.
> - /var/log/messages
> -- can't find oom killer or signal messages.
> - /var/jetty/logs/*.stderrout.log
> -- no exception messages.
> - /var/jetty/logs/*.request.log
> -- everytime final request didn't make response, but no relation on each endpoints.
> - application custom logs
> -- when hundle uncaught exception, put stacktrace on log file. but no logs.
>
> I tried some JAVA_OPTIONS settings and finally Jetty looks running stably.
> Probably "CompressedClassSpaceSize" or "MetaspaceSize" was too small.
> Because after increase two parameter's value, Jetty isn't killed.
>
> I have a question.
> - Is there no log when Jetty killed by "CompressedClassSpaceSize" and/or "MetaspaceSize" shortage?
> - How can I find evidence that "CompressedClassSpaceSize" and/or "MetaspaceSize" is wrong?
> - Is there a possibility that these guesses are wrong and this is Jetty's issue?
>
>
> Environment is follows:
> - aws ec2 t2.2xlarge(4vCPU, 16GiB), Amazon linux 2017.03
> - jetty-distribution-9.3.9.v20160517
> - OpenJDK Runtime Environment (build 1.8.0_131-b11)
> - hosts 3 web application(all apps are using Spring FW 4.1.7)
>
> seting files are follows:
> ----------------
> # cat /etc/default/jetty
> JETTY_HOME=/opt/jetty
> JETTY_BASE=/var/jetty
> JETTY_RUN=/var/run
>
> JAVA_OPTIONS="\
> -Xms16384m \
> -Xmx16384m \
> -XX:CompressedClassSpaceSize=80m \
> -XX:MetaspaceSize=128m \
> -XX:MaxMetaspaceSize=192m \
> -verbose:gc \
> -Xloggc:/tmp/jetty.gc.`date +%Y%m%d-%H%M%S`.log \
> -XX:+UseGCLogFileRotation \
> -XX:GCLogFileSize=10M \
> -XX:NumberOfGCLogFiles=5 \
> -XX:+PrintGC \
> -XX:+PrintGCDetails \
> -XX:+PrintGCDateStamps  \
> -XX:+PrintGCTimeStamps \
> -XX:+PrintClassHistogram \
> -XX:+HeapDumpOnOutOfMemoryError \
> -XX:HeapDumpPath=/tmp \
> -XX:+ExitOnOutOfMemoryError \
> -XX:ErrorFile=/tmp/jetty.hs_err_pid%p.log \
> -XX:+UseConcMarkSweepGC \
> -XX:CMSInitiatingOccupancyFraction=70 \
> -XX:+UseCMSInitiatingOccupancyOnly \
> -Dspring.profiles.active=stg \
> -Dfile.encoding=UTF-8 \
> "
> ----------------
> new CompressedClassSpaceSize is 200m
> new MaxMetaspaceSize is 512m
>
> ----------------
> # cat /var/jetty/start.ini
> jetty.server.dumpBeforeStop=true
> --module=home-base-warning
> --module=ext
> --module=resources
> --module=server
> jetty.threadPool.minThreads=10
> jetty.threadPool.maxThreads=100
> jetty.httpConfig.sendServerVersion=false
> --module=http
> jetty.http.host=0.0.0.0
> --module=deploy
> jetty.deploy.monitoredDir=webapps
> jetty.deploy.scanInterval=0
> jetty.deploy.extractWars=true
> --module=jsp
> --module=requestlog
> jetty.logging.dir=${jetty.base}/logs
> jetty.requestlog.dir=/logs
> jetty.requestlog.filePath=${jetty.requestlog.dir}/yyyy_mm_dd.request.log
> jetty.requestlog.filenameDateFormat=yyyy_MM_dd
> jetty.requestlog.timezone=JST
> --module=debug
> jetty.debug.timezone=JST
> --module=setuid
> ----------------
>
>
> Thanks,
> Toshimichi Naito
> _______________________________________________
> jetty-users mailing list
> jetty-users@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users