{"action":"created","issue":{"url":"https://api.github.com/repos/embulk/embulk/issues/118","labels_url":"https://api.github.com/repos/embulk/embulk/issues/118/labels{/name}","comments_url":"https://api.github.com/repos/embulk/embulk/issues/118/comments","events_url":"https://api.github.com/repos/embulk/embulk/issues/118/events","html_url":"https://github.com/embulk/embulk/pull/118","id":59983599,"number":118,"title":"Fix NPE message issue because Buffer.release() is called twice","user":{"login":"hata","id":174427,"avatar_url":"https://avatars.githubusercontent.com/u/174427?v=3","gravatar_id":"","url":"https://api.github.com/users/hata","html_url":"https://github.com/hata","followers_url":"https://api.github.com/users/hata/followers","following_url":"https://api.github.com/users/hata/following{/other_user}","gists_url":"https://api.github.com/users/hata/gists{/gist_id}","starred_url":"https://api.github.com/users/hata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hata/subscriptions","organizations_url":"https://api.github.com/users/hata/orgs","repos_url":"https://api.github.com/users/hata/repos","events_url":"https://api.github.com/users/hata/events{/privacy}","received_events_url":"https://api.github.com/users/hata/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-03-05T17:15:29Z","updated_at":"2015-03-06T05:51:33Z","closed_at":"2015-03-06T04:58:24Z","pull_request":{"url":"https://api.github.com/repos/embulk/embulk/pulls/118","html_url":"https://github.com/embulk/embulk/pull/118","diff_url":"https://github.com/embulk/embulk/pull/118.diff","patch_url":"https://github.com/embulk/embulk/pull/118.patch"},"body":"This problem is that StdoutOutputPlugin shows NullPointerException when big data is shown because PooledBufferAllocator$NettyByteBuffer.release() has a debug code to check release() is called twice.\r\n\r\nThis problem occurs in TransactionalPageOutput.add() in StdoutOutputPlugin.open(). The code has like the following code:\r\n\r\n```\r\n  public void add(Page page) {\r\n    reader.setPage(page);\r\n    while (reader.nextRecord()) {\r\n      ...\r\n    }\r\n    page.release();\r\n  }\r\n```\r\n\r\nIt means that page.release() is called after reading records(e.g. page1) and then reader.setPage(page) is called when add() is called for another new Page instance(e.g. page2). reader.setPage(page) calls Page.release() for the previous page instance(page1) and then NullPointerException is printed by PooledBufferAllocator$NettyByteBufBuffer.release() because release() for page1 is called twice.\r\n\r\nMy fix is like the following chnages:\r\n\r\n- Added a new parameter, releasePage, to the constructor of PageReader to set a flag to call Page.release() in setPage()/close() or not.\r\n- Added the parameter to StdoutOutputPlugin and CsvFormatterPlugin to avoid calling Page.release(). CsvFormatterPlugin didn't call Page.release(). So, I added the call at the end of  TransactionalPageOutput.add() method.\r\n\r\nNote:\r\n- I checked StdoutOutputPlugin,NullOutputPlugin, and CsvFormatterPlugin. And personally, I thought it is good to call page.release() in add() method for memory usage although it only affects a little memory. (Or, there is other fix ways like removing page.release() from StdoutOutputPlugin.open().) So, it is ok for me to reject my current change.\r\n- CsvFormatterPlugin may not need to change in this fix. The code doesn't call PageReader.close(). So, I think either page.release() or PageReader.close() should be called even though the change is only relates a little memory footprint.\r\n\r\n\r\nTestcase:\r\nI think it is required to make two or more Page instances with stdout output plugin.\r\n\r\n```\r\n$ embulk example try1\r\n$ cd try1/csv/\r\n$ gunzip sample_01.csv.gz\r\n```\r\n\r\nCreate a big csv file. For example, run like the following script to generate about 500k csv file.\r\n\r\n```\r\n#!/bin/bash\r\n# Generate bigger csv file from a file.\r\nfile=sample_01.csv\r\na=0\r\nmax=2500\r\n\r\ncat $file\r\nwhile :\r\ndo\r\n    a=`expr $a + 1`\r\n    cat $file|grep -v \"comment\"\r\n    if [ $a -ge $max ]; then\r\n        break\r\n    fi\r\ndone\r\n```\r\n\r\nSave it as test.sh and run it under try1/csv directory.\r\n\r\n```\r\n$ sh test.sh > sample_500k.csv\r\n```\r\n\r\nThen create config.yml and run embulk.\r\n\r\n```\r\n$ cd ../..\r\n$ embulk guess try1/example.yml -o config.yml\r\n$ embulk run config.yml > test.log\r\n```\r\n\r\nI think you can see like the following NullPointerException several times.\r\n\r\n```\r\njava.lang.NullPointerException\r\n\tat org.embulk.exec.PooledBufferAllocator$NettyByteBufBuffer.release(PooledBufferAllocator.java:54)\r\n\tat org.embulk.spi.Page.release(Page.java:38)\r\n\tat org.embulk.standards.StdoutOutputPlugin$1.add(StdoutOutputPlugin.java:67)\r\n\tat org.embulk.spi.PageBuilder.doFlush(PageBuilder.java:197)\r\n\tat org.embulk.spi.PageBuilder.flush(PageBuilder.java:203)\r\n\tat org.embulk.spi.PageBuilder.addRecord(PageBuilder.java:182)\r\n\tat org.embulk.standards.CsvParserPlugin.run(CsvParserPlugin.java:198)\r\n\tat org.embulk.spi.FileInputRunner.run(FileInputRunner.java:145)\r\n\tat org.embulk.exec.LocalExecutor$6.call(LocalExecutor.java:630)\r\n\tat org.embulk.exec.LocalExecutor$6.call(LocalExecutor.java:616)\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\nPlease comment if you have any questions.\r\n\r\nThanks.\r\n"},"comment":{"url":"https://api.github.com/repos/embulk/embulk/issues/comments/77510543","html_url":"https://github.com/embulk/embulk/pull/118#issuecomment-77510543","issue_url":"https://api.github.com/repos/embulk/embulk/issues/118","id":77510543,"user":{"login":"frsyuki","id":40720,"avatar_url":"https://avatars.githubusercontent.com/u/40720?v=3","gravatar_id":"","url":"https://api.github.com/users/frsyuki","html_url":"https://github.com/frsyuki","followers_url":"https://api.github.com/users/frsyuki/followers","following_url":"https://api.github.com/users/frsyuki/following{/other_user}","gists_url":"https://api.github.com/users/frsyuki/gists{/gist_id}","starred_url":"https://api.github.com/users/frsyuki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frsyuki/subscriptions","organizations_url":"https://api.github.com/users/frsyuki/orgs","repos_url":"https://api.github.com/users/frsyuki/repos","events_url":"https://api.github.com/users/frsyuki/events{/privacy}","received_events_url":"https://api.github.com/users/frsyuki/received_events","type":"User","site_admin":false},"created_at":"2015-03-06T05:51:33Z","updated_at":"2015-03-06T05:51:33Z","body":"7611ade improved error message to include the stacktrace of last release() call:\r\n\r\n```\r\norg.embulk.exec.PooledBufferAllocator$BufferDoubleReleasedException: Detected double release() call of a buffer\r\n        at org.embulk.exec.PooledBufferAllocator$NettyByteBufBuffer.release(PooledBufferAllocator.java:49)\r\n        at org.embulk.spi.PageReader.close(PageReader.java:154)\r\n        at org.embulk.standards.StdoutOutputPlugin$1.close(StdoutOutputPlugin.java:77)\r\n        at org.embulk.exec.LocalExecutor$6.call(LocalExecutor.java:640)\r\n        at org.embulk.exec.LocalExecutor$6.call(LocalExecutor.java:616)\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:262)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: org.embulk.exec.PooledBufferAllocator$BufferReleasedBeforeAt\r\n        at org.embulk.exec.PooledBufferAllocator$NettyByteBufBuffer.release(PooledBufferAllocator.java:54)\r\n        at org.embulk.spi.Page.release(Page.java:38)\r\n        at org.embulk.standards.StdoutOutputPlugin$1.add(StdoutOutputPlugin.java:67)\r\n        at org.embulk.spi.PageBuilder.doFlush(PageBuilder.java:197)\r\n        at org.embulk.spi.PageBuilder.finish(PageBuilder.java:211)\r\n        at org.embulk.input.jdbc.AbstractJdbcInputPlugin.run(AbstractJdbcInputPlugin.java:241)\r\n        at org.embulk.exec.LocalExecutor$6.call(LocalExecutor.java:630)\r\n        ... 5 more\r\n```"}}