Objects are constantly promoted to the Old Gen
<p>I'm working with a application which provides REST API. It works with  GET requests only and usually spends up to 100ms to handle even the most heavy requests.</p>

<p>Recently we started facing a problem that from time to time heap gets fullfilled and full GC takes a lot of time and it really affects our clients.</p>

<ul>
<li>Heap size - 5GB </li>
<li>Concurrent requests per second - up to 50</li>
<li>Each request consumes up to 1-2MB(There are few requests that consume 12 MB of heap) </li>
<li>Application uses Paralel GC for both Old and young geerations.</li>
<li>JDK 1.7</li>
</ul>

<p>From my point of view this application should promote some object in the Old Gen at the very beginning of work and after it all new objects should get removed from Eden or Survivor spaces. Thus full GC shoulld never come. However some objects keep get promoted into Old Gen and it's really confusing for me.</p>

<p>Several additional findings:</p>

<ul>
<li>We enabled JVM flag -XX:+PrintTenuringDistribution and according to this flag output new tenuring threshold becomes equal to 1 in few minutes after application starts, even without significant load.</li>
<li>We took a memory dump when the application consumed almsot all of allowed memory and according to the MAT analyzer statistics almost all of objects inn dump are unreachable(So, I can't understand why they get promoted into Old Gen)</li>
<li>Survivor spaces are almost not used. Looks like new objects get promoted into Old gen right from Eden space.</li>
<li>Minor GC comes each ~10-20 seconds.</li>
<li>Increasing Eden/Survivor spaces didn't help to solve the problem(tenuring threshold is still 1)</li>
</ul>

<p>So, could you please advice:</p>

<ul>
<li>For what reasons tenuring threshold becomes 1 so quickly?[<strong>Resolved</strong>]</li>
<li>What additional steps could I do to avoid promoting new objects into old gen?</li>
</ul>

<p>Please let me know if you need some additional info.</p>

<p><strong>Edit</strong>:</p>

<p>JVM parameters(<strong>Updated</strong>):</p>

<pre><code>-XX:MaxPermSize=512m -Xmx7g -Xmn4g -verbose:gc -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintTenuringDistribution
</code></pre>

<p>Application logs:</p>

<pre><code>Desired survivor size 520617984 bytes, new threshold 15 (max 15)
 [PSYoungGen: 3235245K-&gt;231763K(3665920K)] 3942774K-&gt;939308K(4760064K), 0.0905430 secs] [Times: user=0.31 sys=0.00, real=0.09 secs] 
603.561: [GC
Desired survivor size 521142272 bytes, new threshold 15 (max 15)
 [PSYoungGen: 3369299K-&gt;330881K(3684864K)] 4076844K-&gt;1038442K(4779008K), 0.1343473 secs] [Times: user=0.51 sys=0.00, real=0.13 secs] 
606.347: [GC
Desired survivor size 506462208 bytes, new threshold 15 (max 15)
 [PSYoungGen: 3507329K-&gt;215655K(3685376K)] 4214890K-&gt;923233K(4779520K), 0.0925060 secs] [Times: user=0.36 sys=0.00, real=0.09 secs] 
609.084: [GC
Desired survivor size 492306432 bytes, new threshold 15 (max 15)
 [PSYoungGen: 3392103K-&gt;213344K(3713536K)] 4099681K-&gt;920945K(4807680K), 0.0802729 secs] [Times: user=0.30 sys=0.00, real=0.08 secs] 
</code></pre>

<p><strong>Important detail</strong>:</p>

<p>Even when I start performance test with only 1 thread(that consumes ~10 MB per request) Old Gen keeps growing:
<a href="https://i.stack.imgur.com/N0hYL.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/N0hYL.png" alt="Old Gen with 1 thread perf test"></a></p>

<p>If I start GC it successfully cleanups memory</p>

<p>If I make a heap dump Mat Analyzer (or Yourkit) shows again that 80% of objects are unreachable</p>