java application crashed by suspicious jdbc memory leak
<p>I have been working on a java application which crawls page from Internet with http-client(version4.3.3). It uses one fixedThreadPool with 5 threads,each is a loop thread .The pseudocode is following.</p>

<pre><code>public class Spiderling extends Runnable{
  @Override
  public void run() {

    while (true) {
        T task = null;
        try {
            task = scheduler.poll();

            if (task != null) {
                if Ehcache contains task's config
                        taskConfig = Ehcache.getConfig;
                else{
                    taskConfig = Query task config from db;//close the conn every time
                    put taskConfig into Ehcache
                }


                spider(task,taskConfig);
            }

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
    LOG.error("spiderling is DEAD");
}
}
</code></pre>

<p>I am running it with following arguments <code>-Duser.timezone=GMT+8 -server -Xms1536m -Xmx1536m -Xloggc:/home/datalord/logs/gc-2016-07-23-10-28-24.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC</code> on a server(2 cpus,2G memory) and it crashes pretty regular about once in two or three days with no OutOfMemoryError and no JVM error log.</p>

<p>Here is my analysis；</p>

<ol>
<li>I analyse the gc log with GC-EASY,the report is <a href="http://gceasy.io/my-gc-report.jsp?p=L2hvbWUvcmFtL3VwbG9hZC9pbnN0YW5jZTEvc2hhcmVkLzIwMTYtNy0yNC9nYy0yMDE2LTA3LTIwLTE0LTA2LTU1LmxvZy0yLTEtMTI=" rel="nofollow">here</a>. The weird thing is the Old Gen increasing slowly until the allocated max heap size,but the Full Gc has never happened even once.</li>
<li>I suspect it might has memory leak,so I dump the heap map using cmd <code>jmap -dump:format=b,file=soldier.bin</code> and using the Eclipse MAT to analyze the dump file.Here is the problem suspect which object occupies 280+ M bytes.</li>
</ol>

<blockquote>
  <p>The class "com.mysql.jdbc.NonRegisteringDriver", 
  loaded by "sun.misc.Launcher$AppClassLoader @ 0xa0018490", occupies 281,118,144
  (68.91%) bytes. The memory is accumulated in one instance of
  "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "".</p>
  
  <p>Keywords
  com.mysql.jdbc.NonRegisteringDriver
  java.util.concurrent.ConcurrentHashMap$Segment[]
  sun.misc.Launcher$AppClassLoader @ 0xa0018490.</p>
</blockquote>

<p>I use c3p0-0.9.1.2 as mysql connection pool and mysql-connector-java-5.1.34 as jdbc connector and Ehcache-2.6.10 as memory cache.I have see all posts about 'com.mysql.jdbc.NonregisteringDriver memory leak' and still get no clue. </p>

<p>This problem has driven me crazy for several days, any advice or help will be appreciated!</p>

<p>**********************Supplementary description on 07-24****************</p>

<p>I use a JAVA WEB + ORM Framework called JFinal(github.com/jfinal/jfinal) which is open in github。
Here are some core code for further description about the problem.</p>

<pre><code>/**
 * CacheKit. Useful tool box for EhCache.
 * 
 */

public class CacheKit {

private static CacheManager cacheManager;
private static final Logger log = Logger.getLogger(CacheKit.class);

static void init(CacheManager cacheManager) {
    CacheKit.cacheManager = cacheManager;
}

public static CacheManager getCacheManager() {
    return cacheManager;
}

static Cache getOrAddCache(String cacheName) {
    Cache cache = cacheManager.getCache(cacheName);
    if (cache == null) {
        synchronized(cacheManager) {
            cache = cacheManager.getCache(cacheName);
            if (cache == null) {
                log.warn("Could not find cache config [" + cacheName + "], using default.");
                cacheManager.addCacheIfAbsent(cacheName);
                cache = cacheManager.getCache(cacheName);
                log.debug("Cache [" + cacheName + "] started.");
            }
        }
    }
    return cache;
}

public static void put(String cacheName, Object key, Object value) {
    getOrAddCache(cacheName).put(new Element(key, value));
}

@SuppressWarnings("unchecked")
public static &lt;T&gt; T get(String cacheName, Object key) {
    Element element = getOrAddCache(cacheName).get(key);
    return element != null ? (T)element.getObjectValue() : null;
}

@SuppressWarnings("rawtypes")
public static List getKeys(String cacheName) {
    return getOrAddCache(cacheName).getKeys();
}

public static void remove(String cacheName, Object key) {
    getOrAddCache(cacheName).remove(key);
}

public static void removeAll(String cacheName) {
    getOrAddCache(cacheName).removeAll();
}

@SuppressWarnings("unchecked")
public static &lt;T&gt; T get(String cacheName, Object key, IDataLoader dataLoader) {
    Object data = get(cacheName, key);
    if (data == null) {
        data = dataLoader.load();
        put(cacheName, key, data);
    }
    return (T)data;
}

@SuppressWarnings("unchecked")
public static &lt;T&gt; T get(String cacheName, Object key, Class&lt;? extends IDataLoader&gt; dataLoaderClass) {
    Object data = get(cacheName, key);
    if (data == null) {
        try {
            IDataLoader dataLoader = dataLoaderClass.newInstance();
            data = dataLoader.load();
            put(cacheName, key, data);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
    return (T)data;
}
</code></pre>

<p>}</p>

<p>I use CacheKit like <code>CacheKit.get("cfg_extract_rule_tree", extractRootId, new ExtractRuleTreeDataloader(extractRootId))</code>. and class <code>ExtractRuleTreeDataloader</code> will be called if find nothing in cache by <code>extractRootId</code>.</p>

<pre><code>public class ExtractRuleTreeDataloader implements IDataLoader {
public static final Logger LOG = LoggerFactory.getLogger(ExtractRuleTreeDataloader.class);
private int                ruleTreeId;

public ExtractRuleTreeDataloader(int ruleTreeId) {
    super();
    this.ruleTreeId = ruleTreeId;
}

@Override
public Object load() {
    List&lt;Record&gt; ruleTreeList = Db.find("SELECT * FROM cfg_extract_fule WHERE root_id=?", ruleTreeId);
    TreeHelper&lt;ExtractRuleNode&gt; treeHelper = ExtractUtil.batchRecordConvertTree(ruleTreeList);//convert List&lt;Record&gt; to and tree
    if (treeHelper.isValidTree()) {
        return treeHelper.getRoot();
    } else {
        LOG.warn("rule tree id :{} is an error tree #end#", ruleTreeId);
        return null;
    }
}
</code></pre>

<p>As I said before, I use JFinal ORM.The <code>Db.find</code> method code is </p>

<pre><code>public List&lt;Record&gt; find(String sql, Object... paras) {
    Connection conn = null;
    try {
        conn = config.getConnection();
        return find(config, conn, sql, paras);
    } catch (Exception e) {
        throw new ActiveRecordException(e);
    } finally {
        config.close(conn);
    }
}
</code></pre>

<p>and the <code>config close</code> method code is </p>

<pre><code>public final void close(Connection conn) {
    if (threadLocal.get() == null)      // in transaction if conn in threadlocal
        if (conn != null)
            try {conn.close();} catch (SQLException e) {throw new ActiveRecordException(e);}
}
</code></pre>

<p>There is no transaction in my code,so I am pretty sure the conn.close() will be called every time.</p>

<p>**********************more description on 07-28****************</p>

<p>First, I use Ehcache to store the taskConfigs in the memory. And the taskConfigs almost never change, so I want store them in the memory eternally and store them to disk if the memory can not store them all.</p>

<p>I use MAT to find out the GC Roots of NonRegisteringDriver, and the result is show in the following picture.
<a href="http://i.stack.imgur.com/NgDdB.png" rel="nofollow">The Gc Roots of NonRegisteringDriver</a></p>

<p>But I still don't understand why the default behavior of Ehcache lead memory leak.The taskConfig is a class extends the Model class.</p>

<pre><code>public class TaskConfig extends Model&lt;TaskConfig&gt; {
    private static final long    serialVersionUID = 5000070716569861947L;
    public static TaskConfig DAO              = new TaskConfig();

}
</code></pre>

<p>and the source code of Model is in this page(github.com/jfinal/jfinal/blob/jfinal-2.0/src/com/jfinal/plugin/activerecord/Model.java). And I can't find any reference (either directly or indirectly) to the connection object as @Jeremiah guessing.</p>

<p>Then I read the source code of NonRegisteringDriver, and don't understand why the map field <code>connectionPhantomRefs</code> of <code>NonRegisteringDriver</code> holds more than 5000 entrys of <code>&lt;ConnectionPhantomReference, ConnectionPhantomReference&gt;</code>,but find no <code>ConnectionImpl</code> in the queue field <code>refQueue</code> of <code>NonRegisteringDriver</code>. Because I see the cleanup code in class <code>AbandonedConnectionCleanupThread</code> which means it will move the <code>ref</code> in the <code>NonRegisteringDriver.connectionPhantomRefs</code> while getting abandoned connection <code>ref</code> from <code>NonRegisteringDriver.refQueue</code>.</p>

<pre><code>@Override
public void run() {
    threadRef = this;
    while (running) {
        try {
            Reference&lt;? extends ConnectionImpl&gt; ref = NonRegisteringDriver.refQueue.remove(100);
            if (ref != null) {
                try {
                    ((ConnectionPhantomReference) ref).cleanup();
                } finally {
                    NonRegisteringDriver.connectionPhantomRefs.remove(ref);
                }
            }

        } catch (Exception ex) {
            // no where to really log this if we're static
        }
    }
}
</code></pre>

<p>Appreciate the help offered by @Jeremiah ! </p>