{"push_id":675651770,"size":1,"distinct_size":1,"ref":"refs/heads/alloc","head":"5249579f770aab36a31f2f42283f62707ed10839","before":"ff04637f858d2e069c4d7cd0850c3cc19e1b5ec2","commits":[{"sha":"5249579f770aab36a31f2f42283f62707ed10839","author":{"email":"77634b9dffad39de322ac3e901bd5b8059fe2e65@yahoo.com","name":"Chris Westin"},"message":"DRILL-1942: Improve off-heap memory usage tracking\nThis is a reimplementation of the allocators to make tracking better. The\nre-implementation includes support for JMX MBeans for on-the-fly memory\nstatistics inspection. The new classes are BaseAllocator, RootAllocator, and\nChildAllocator. The BufferAllocator interface has been modified a bit.\n\nIn addition to that, the new allocator flattens the use of buffers in slices,\nso that a DrillBuf is always referring to an UnsafeDirectLittleEndian, and\nnever to another DrillBuf; this simplifies figuring out what's going on when\nsomething goes wrong. The new allocator (BaseAllocator, and its derived\nclasses RootAllocator and ChildAllocator) treats sharing buffers in the same\nway as slicing buffers, and sharing and ownership transfer of buffers and\ntheir slices both work.\n\nRecord batch and value vector classes previously had a mandatory cleanup()\nmethod. These classes have been made to include AutoCloseable as a base class,\nand cleanup() has been renamed to close(). This was done to get additional\nhelp from tools to find places that we need to close() things.\n\nAbstractRecordBatch:\n- cleanup() -> close()\n- hygiene\n\nAbstractSingleRecordBatch:\n- cleanup() -> close()\n- hygiene\n\nAccounter:\n- hygiene\n- TODOs\n\nAllocatorLedger:\n- created private interface for child allocators to communicate with parents\n\nAllocationReservation:\n- created\n\nAtomicRemainder:\n- TODOs\n\nBaseAllocator:\n- created common base class for buffer allocators\n\nBaseDataValueVector\n- released the data buffer in close()\n\nBitVector\n- fixed a bug in splitAndTransferTo(), which was missing a retain() call after\n  slicing a buffer for the transfer target.\n\nBootStrapContext:\n- hygiene\n\nBroadcastSenderRootExec:\n- cleanup() => close()\n- hygiene\n\nBufferAllocator:\n- replace preallocator with createReservation()\n- Closeable -> AutoCloseable, and add throws clause\n  - required minor changes to several other files\n    - created DrillCloseables and DrillAutoCloseables to make this easy\n- renamed the second form of BufferAllocator.takeOwnership to shareOwnership\n- clean up interface definition - no abstract needed\n- hygiene\n- some javadoc\n- TODOs\n\nChildAllocator\n- created\n\nDataServer\n- handle failed data transfers due to cancelled closed fragments\n\nDrillAutoCloseables:\n- add comment that null argument is supported\n- fix a typo in a comment\n\nDrillbit:\n- sychhronize ShutdownThread.run()'s action against a static to avoid\n  concurrent shutdown problems with metrics by serializing the shutdown\n  of multiple drillbits running in the same JVM.\n\nDrillBuf:\n- add some finals\n- minor formatting\n- fix spelling mixtake\n- TODOs\n\nDrillConfig:\n- hygiene\n\nExecConstants\n- debugging property for memory to ExecConstants\n- added ALLOCATION_POLICY property name\n\nExternalSortBatch:\n- only get allocator once, outside loop body\n- release buffers obtained from getBuffers()\n\nFakeAllocator:\n- switch from preallocator to AllocationReservation\n- private logger\n- TODOs\n\nFilterRecordBatch:\n- made some variables private\n\nFragmentContext:\n- getAllocator() shouldn't be @Deprecated\n- made logger private\n- TODOs\n\nForemanException:\n- removed unused fromThrowable()\n- commented out unused logger\n- formatting\n\nFragmentSetupException:\n- leave out the finals\n\nHashAggTemplate:\n- cleanup() -> close()\n- hygiene\n\nMappifyUtility:\n- remove unused imports\n\nMergeJoinBatchBuilder:\n- replace preallocator with AllocationReservation\n- implement close() to clean up reservation\n\nMSortTemplate:\n- made some variables private\n\nNestedConfig:\n- hygiene\n\nOperatorContext:\n- removed unused applyFragmentLimits member\n- made logger private\n- TODOs\n\nOutOfMemoryException:\n- add serialVersionUID to silence warning\n- minor formatting\n\nOutOfMemoryRuntimeException:\n- add serialVersionUID to silence warning\n- minor formatting\n\nPartitionSenderRootExec:\n- cleanup() -> close()\n- hygiene\n\nPooledByteBufAllocatorL:\n- finals\n- formatting\n- make logger private (and then comment out because it is unused)\n\nPrintingResultsListener:\n- TODOs\n\nPriorityQueueCopierTemplate:\n- removed unnecessary use of the pre-allocator\n- made some members private\n\nPriorityQueueTemplate:\n- removed unnecessary use of preallocator\n- made some members private\n\nRecordBatch:\n- interface now extends AutoCloseable instead of declaring it's own cleanup()\n- hygiene\n- TODOs\n\nRootAllocator:\n- created replacement for TopLevelAllocator\n\nScreenCreator:\n- clean() => close()\n- hygiene\n\nSimpleRootExec:\n- hygiene\n- fixup RecordBatch close()\n\nSingleSenderCreator:\n- cleanup() => close()\n- hygiene\n\nSortRecordBatchBuilder:\n- replace preallocation with AllocationReservation\n- implements AutoCloseable\n- implement close() to close reservation\n- formatting\n- hygiene\n\nStackTrace:\n- DRILL-2624: fixed numeric formatting for line numbers\n\nTestBaseAllocator:\n- created new test for allocators, unit tests them in isolation\n\nTopLevelAllocator:\n- replace preallocator with AllocationReservation implementation\n- eliminated @Deprecated constructors\n  - changed many other files to use remaining constructor\n- eliminate private constructor\n- add finals, private\n- formatting\n- TODOs\n\nVectorAccessible:\n- hygiene (mostly comments)\n\nExternalSortBatch:\nExpressionInterpreterTest:\nFilterRecordBatch:\nHashAggBatch:\nHashJoinBatch:\nIteratorValidatorBatchIterator:\nLimitRecordBatch:\nMergeJoinBatch:\nMergingRecordBatch:\nOrderedPartitionRecordBatch:\nProducerConsumerBatch:\nRemovingRecordBatch:\nScanBatch:\nSortBatch:\nStreamingAggBatch:\nStreamingWindowFrameRecordBatch:\nTopNBatch:\nTraceRecordBatch:\nUnionAllRecordBatch:\nUnorderedReceiverBatch:\nWriterRecordBatch:\n- cleanup() -> close()\n- close builders that use AllocationReservation\n- hygiene\n\nTestAllocators\nTestSimpleFragmentRun\nTestSimpleFunctions\nTestStringFunctions\nTestLoad\nTestRepeated\nTestStringFunctions\n... many others...\n- modify to use new allocator\n- some hygiene\n\nValueVector and RecordWriter templates:\n- added clauses to release buffers before references are overwritten\n- eliminated common subexpressions and repeated getAccessor()/getMutator()\n  calls in loops\n- hygiene","distinct":true,"url":"https://api.github.com/repos/cwestin/incubator-drill/commits/5249579f770aab36a31f2f42283f62707ed10839"}]}