Bug 369941 - [api][dstore] cancelable threads not removed fast enough from Hashmap, resulting in OOM
Summary: [api][dstore] cancelable threads not removed fast enough from Hashmap, result...
Status: RESOLVED FIXED
Alias: None
Product: Target Management
Classification: Tools
Component: RSE (show other bugs)
Version: 3.0.3   Edit
Hardware: Other other
: P3 normal (vote)
Target Milestone: 3.4 M6   Edit
Assignee: David McKnight CLA
QA Contact: Martin Oberhuber CLA
URL:
Whiteboard:
Keywords: api
Depends on:
Blocks: 369950 369951
  Show dependency tree
 
Reported: 2012-01-27 10:23 EST by Onno Van den Troost CLA
Modified: 2012-02-06 09:40 EST (History)
1 user (show)

See Also:


Attachments
patch to automatically update cancellable threads after a download (4.72 KB, patch)
2012-01-27 10:41 EST, David McKnight CLA
no flags Details | Diff
update to add api for HEAD stream change (2.41 KB, patch)
2012-02-02 09:25 EST, David McKnight CLA
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Onno Van den Troost CLA 2012-01-27 10:23:49 EST
A customer reported an Out-Of-Memory error, and heapdump analysis showed that ServerCommandHandler was the main leak suspect.

Our developer already communicated with David M. about this:

<quote>
Looking at the heap dump, this part of heap occupation
is narrowed down to HashMap which stores the cancellable threads of UniversalDownloadHandler. 
The UniversalDownloadHandler instance is not removed from the HashMap soon after the download is finished.
The following actions are defined as the cancellable threads:
  Copy
  CreateFile
  CreateFolder
  Delete
  Query
  Rename
  Download
  Search
The timing when the thread instance is removed from the HashMap is the time when the thread is gone and the next cancellable command has come to the UniversalFileSystemMiner from a client.

Looking at the server log, about 15 clients were working at one process.
The total size of instances which the above HashMap has is about 150MB.
That is, about 15MB per client in average.
When all of the clients download some USS file and then works for MVS miners, all of the UniversalDownloadHandler instances are not removed.
In fact, the heap core dump said that the current thread was MVS miner.
 
After all, I think that Open RSE base should remove the  cancellable threads from the HashMap soon after the UniversalFileSystemMiner finishes the process for the corresponding command.

David,
What do you think about this problem ?
---
Completed threads are supposed to be removed from the cancellable threads HashMap on subsequent commands so I'm not sure why there would be an accumulation.  I've attached a dstore_miners.jar that automatically involves updateCancellableThreads() after download completion.
</quote>

The customer confirmed that the test fix worked.

We would like a backport of the fix to 30x and 32x.
Comment 1 David McKnight CLA 2012-01-27 10:41:54 EST
Created attachment 210192 [details]
patch to automatically update cancellable threads after a download
Comment 2 David McKnight CLA 2012-01-27 10:42:37 EST
To avoid API changes, I've used reflection to allow the UniversalDownloadHandler to access UniversalFileSystemMiner.updateCancellableThreads().
Comment 3 David McKnight CLA 2012-01-27 11:09:55 EST
I've committed the change to the HEAD stream and opened bug 369950 and bug 369951 for backporting.
Comment 4 David McKnight CLA 2012-02-02 09:25:13 EST
Created attachment 210446 [details]
update to add api for HEAD stream change
Comment 5 David McKnight CLA 2012-02-02 09:26:36 EST
For the HEAD stream change, I've changed private method UniversalFileSystemMiner.updateCancellableThreads() to be public so reflection isn't needed to make the call immediately after a download request completes.