Torch out of memory in thread when using torch.serialize twice
<p>I'm trying to add a parallel dataloader to the <a href="https://github.com/AlexMili/torch-dataframe/" rel="nofollow">torch-dataframe</a> in order to add <a href="https://github.com/AlexMili/torch-dataframe/issues/24" rel="nofollow">torchnet compatibility</a>. I've used the <a href="https://github.com/torchnet/torchnet/blob/master/dataset/paralleldatasetiterator.lua" rel="nofollow">tnt.ParallelDatasetIterator</a> and <a href="https://github.com/AlexMili/torch-dataframe/blob/torchnet/helper_classes/paralleliterator.lua" rel="nofollow">changed it</a> so that:</p>

<ol>
<li>A basic batch is loaded outside the threads</li>
<li>The batch is serialized and sent to the thread</li>
<li>In the thread the batch is deserialized and converts the batch data to tensors</li>
<li>The tensors are returned in a table that has the <code>input</code>  and <code>target</code> keys in order to match the <a href="https://github.com/torchnet/torchnet/blob/master/engine/sgdengine.lua#L78-L109" rel="nofollow">tnt.Engine</a> setup.</li>
</ol>

<p><strong>The problem</strong> occurs the second time the <code>enque</code> is called with an error: <code>.../torch_distro/install/bin/luajit: not enough memory</code>. I'm currently only working with <a href="https://github.com/andresy/mnist" rel="nofollow">mnist</a> with an adapted <a href="https://github.com/AlexMili/torch-dataframe/blob/torchnet/examples/mnist_example.lua" rel="nofollow">mnist-example</a>. The <code>enque</code> loop now looks like this (with debugging memory output):</p>

<pre><code>-- `samplePlaceholder` stands in for samples which have been
-- filtered out by the `filter` function
local samplePlaceholder = {}

-- The enque does the main loop
local idx = 1
local function enqueue()
  while idx &lt;= size and threads:acceptsjob() do
    local batch, reset = self.dataset:get_batch(batch_size)

    if (reset) then
      idx = size + 1
    else
      idx = idx + 1
    end

    if (batch) then
      local serialized_batch = torch.serialize(batch)

      -- In the parallel section only the to_tensor is run in parallel
      --  this should though be the computationally expensive operation
      threads:addjob(
        function(argList)
          io.stderr:write("\n Start");
          io.stderr:write("\n 1: " ..tostring(collectgarbage("count")))
          local origIdx, serialized_batch, samplePlaceholder = unpack(argList)

          io.stderr:write("\n 2: " ..tostring(collectgarbage("count")))
          local batch = torch.deserialize(serialized_batch)
          serialized_batch = nil

          collectgarbage()
          collectgarbage()

          io.stderr:write("\n 3: " .. tostring(collectgarbage("count")))
          batch = transform(batch)

          io.stderr:write("\n 4: " .. tostring(collectgarbage("count")))
          local sample = samplePlaceholder
          if (filter(batch)) then
            sample = {}
            sample.input, sample.target = batch:to_tensor()
          end
          io.stderr:write("\n 5: " ..tostring(collectgarbage("count")))

          collectgarbage()
          collectgarbage()
          io.stderr:write("\n 6: " ..tostring(collectgarbage("count")))

          io.stderr:write("\n End \n");
          return {
            sample,
            origIdx
          }
        end,
        function(argList)
          sample, sampleOrigIdx = unpack(argList)
        end,
        {idx, serialized_batch, samplePlaceholder}
      )
    end
  end
end
</code></pre>

<p>I've sprinkled <code>collectgarbage</code> and also tried to remove any objects not needed. The memory output is rather straight forward:</p>

<pre><code> Start
 1: 374840.87695312
 2: 374840.94433594
 3: 372023.79101562
 4: 372023.85839844
 5: 372075.41308594
 6: 372023.73632812
 End 
</code></pre>

<p>The function that loops the <code>enque</code> is the non-ordered function that is trivial (the memory error is thrown at the second <code>enque</code> and the ):</p>

<pre><code>iterFunction = function()
  while threads:hasjob() do
    enqueue()
    threads:dojob()
    if threads:haserror() then
      threads:synchronize()
    end
    enqueue()

    if table.exact_length(sample) &gt; 0 then
      return sample
    end
  end
end
</code></pre>