{"push_id":734749917,"size":23,"distinct_size":23,"ref":"refs/heads/master","head":"0c538915b23d9ad4e410036c9ac18c3e928ce251","before":"2d191ab05fff3392697fbce8dd909003c86d34e1","commits":[{"sha":"a2eb6cc8774ab43475829b0b691e62739fbbe88b","author":{"email":"053520384a0ea670db2d31d536af97669647c081@gmail.com","name":"Aljoscha Krettek"},"message":"[FLINK-1967] Introduce (Event)time in Streaming\n\nThis introduces an additional timestamp field in StreamRecord. When using a\nSourceFunction and an auto-timestamp interval is set using the\nExecutionConfig, the timestamp is automatically set to System.currentTimeMillis()\nupon element emission. The timestamp can be manually set using an\nEventTimeSourceFunction.\n\nThis also changes the signature of the StreamOperators. They now get\na StreamRecord instead of the unwrapped value. This is necessary for\nthem to access the timestamp. Before, the StreamTask would unwrap the\nvalue from the StreamRecord, now this is moved one layer higher.\n\nThis also introduces watermarks to keep track of processing. At\na configurable interval the sources will emit watermarks that signify\nthat no records with a lower timestamp will be emitted in the future by\nthis source. The timestamps are broadcast, stream inputs wait for watermark\nevents on all input channels and forward the watermark to the\nStreamOperator once the watermark advances on all inputs. Operators are\nresponsible for forwarding the watermark once they know that no elements\nwith a previous timestamp will be emitted (this is mostly relevant for\nbuffering operations such as windows). Right now, all operators simply\nforward the watermark they receive.\n\nWhen using an EventTimeSourceFunction the system does not\nautomatically emit timestamps, the user is required to manually emit\nwatermarks using the SourceContext.\n\nNo watermarks will be emitted unless\nExecutionConfig.setAutoWatermarkInterval is used to set a watermark\ninterval.\n\nPer default timestamps and watermarks are completely disabled, there is\na switch in ExecutionConfig (enableTimstamps()) to enable them. This\nmeans that, out-of-box, the performance is not changed by adding this\nnew feature. If it is not used.\n\nThis commit contains fixes for other stuff that was discovered while\nimplementing the feature. See Jira issue numbers and descriptions below.\n\n[FLINK-2290/2295] Fix CoReader Event Handling and Checkpointing\n\nThis changes CoReader (now CoStreamingRecordReader) to reuse\nUnionGate for the input multiplexing. This way it will not lock in on\none specific input side and read events from both input sides.\n\nThis also enables an event listener for checkpoint barriers so that the\nTwoInputTask now reacts to those and correctly forwards them.\n\nThen, this adds CoStreamCheckpointintITCase to verify that checkpointing\nworks in topologies with TwoInputStreamTasks.\n\nThis also adds tests for OneInputStreamTask and TwoInputStreamTask\nthat check whether:\n - whether open()/close() of RichFunctions are correctly\n   called\n - Watermarks are correctly forwarded\n - Supersteps/checkpoint barriers are correctly forwarded and the\n   blocking of inputs works correctly\n\nAdd proper tests for Stream Operators\n\nThese test whether:\n - open()/close() on RichFunctions are called\n - Timestamps of emitted elements match the timestamp of the input\n   element\n - Watermarks are correctly forwarded\n\n[FLINK-2301] Fix Behaviour of BarrierBuffer and add Tests\n\nBefore, a CheckpointBarrier from a more recent Checkpoint would also\ntrigger unblocking while waiting on an older CheckpointBarrier. Now,\na CheckpointBarrier from a newer checkpoint will unblock all channels\nand start a new wait on the new Checkpoint.\n\nThe tests for OneInputStreamTask and TwoInputStreamTask check whether\nthe buffer behaves correctly when receiving CheckpointBarriers from more\nrecent checkpoints while still waiting on an older CheckpointBarrier.","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/a2eb6cc8774ab43475829b0b691e62739fbbe88b"},{"sha":"0f45f2b73445352bb80fed72474c9ffffdaceb3b","author":{"email":"180f55e8350155df1048ad760eeaa1cff137f014@apache.org","name":"Maximilian Michels"},"message":"[tests] add flag for synchronous execution of futures in TestingCluster\n\nThis adds a flag to TestingCluster to disabled asynchronous execution of\nfutures via the CallingThreadDispatcher.","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/0f45f2b73445352bb80fed72474c9ffffdaceb3b"},{"sha":"0f589aad899ca2ee230ff21655a38842928005f1","author":{"email":"180f55e8350155df1048ad760eeaa1cff137f014@apache.org","name":"Maximilian Michels"},"message":"[FLINK-2371] improve AccumulatorLiveITCase\n\nInstead of using Thread.sleep() to synchronize the checks of the\naccumulator values, we rely on message passing here to synchronize the\ntask process.\n\nTherefore, we let the task process signal to the task manager that it\nhas updated its accumulator values. The task manager lets the job\nmanager know and sends out the heartbeat which contains the\naccumulators. When the job manager receives the accumulators and has\nbeen notified previously, it sends a message to the subscribed test case\nwith the current accumulators.\n\nThis assures that all processes are always synchronized correctly and we\ncan verify the live accumulator results correctly.\n\nIn the course of rewriting the test, I had to change two things in the\nimplementation:\n\na) User accumulators are now immediately serialized as well. Otherwise,\nAkka does not serialize in local one VM setups and passes the live\naccumulator map through.\n\nb) The asynchronous update of the accumulators is disabled for\ntests (via the dispatcher flag of the TestingCluster). This was\nnecessary because we cannot guarantee when the Future for updating the\naccumulators is executed. In real setups this is neglectable.\n\nThis closes #925.","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/0f589aad899ca2ee230ff21655a38842928005f1"},{"sha":"1d373a7de39ac3fc14d69025122b04d27ec9d7ba","author":{"email":"180f55e8350155df1048ad760eeaa1cff137f014@apache.org","name":"Maximilian Michels"},"message":"[FLINK-2382] fix Flink live accumulators for TwoInputStreamTask","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/1d373a7de39ac3fc14d69025122b04d27ec9d7ba"},{"sha":"75d1639a5f82dc6b67cb6ab3f3824d1c9191fd23","author":{"email":"55535687dad626de987465d57be5008b6e327a91@apache.org","name":"Stephan Ewen"},"message":"[core] Remove obsolete OperatorUtil","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/75d1639a5f82dc6b67cb6ab3f3824d1c9191fd23"},{"sha":"44ee1c1b64b18db199a7e77492dc890a1234fcb0","author":{"email":"55535687dad626de987465d57be5008b6e327a91@apache.org","name":"Stephan Ewen"},"message":"[FLINK-2358] [dashboard] First part dashboard server backend\n\n - Adds a separate Maven project for easier maintenance. Also allows users to refer to runtime without web libraries.\n - Simple HTTP server based on Netty HTTP (slim dependency, since we use netty anyways)\n - REST URL parsing via Netty Router\n - Abstract stubs for handlers that deal with errors and request/response\n - First set of URL request handlers that produce JSON responses\n\nThis closes #677\nThis closes #623\nThis closes #297","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/44ee1c1b64b18db199a7e77492dc890a1234fcb0"},{"sha":"e86f451705315ea7d1ed4d3821e08dd2225d84c0","author":{"email":"55535687dad626de987465d57be5008b6e327a91@apache.org","name":"Stephan Ewen"},"message":"[FLINK-2358] [dashboard] Add server for static dashboard files (HTML, JS, CSS, ...)","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/e86f451705315ea7d1ed4d3821e08dd2225d84c0"},{"sha":"d59cebd8c3f643dc6da88924300e78f65f26c640","author":{"email":"55535687dad626de987465d57be5008b6e327a91@apache.org","name":"Stephan Ewen"},"message":"[FLINK-2358] [dashboard] Add first stub of angular.js based dashboard.\n\nAdd README.md that outlines build instructions.","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/d59cebd8c3f643dc6da88924300e78f65f26c640"},{"sha":"ea2b1b139eb221b2b1a81ee776a04e711c3f8af4","author":{"email":"55535687dad626de987465d57be5008b6e327a91@apache.org","name":"Stephan Ewen"},"message":"[FLINK-2358] [dashboard] Adding comments to web dashboard files, add LICENSE file entries","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/ea2b1b139eb221b2b1a81ee776a04e711c3f8af4"},{"sha":"29454de57edeeeb8b7ec404bfb8c08cba6e45781","author":{"email":"4877b5966827a9c06b3f725e99b7442e1fa1186a@gmail.com","name":"Gabor Gevay"},"message":"[FLINK-2377] [tests] Add reader.close() to readAllResultLines in TestBaseUtils\n\nThis closes #924","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/29454de57edeeeb8b7ec404bfb8c08cba6e45781"},{"sha":"1404db6a90080f3562ce0d8b1a62e466c6525734","author":{"email":"55535687dad626de987465d57be5008b6e327a91@apache.org","name":"Stephan Ewen"},"message":"[FLINK-2377] [tests] Make buffer closing logic robust to I/O failures.","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/1404db6a90080f3562ce0d8b1a62e466c6525734"},{"sha":"0818bec02c0bc90bae68f09d8d2e48dc0f6b9374","author":{"email":"c00d5696513395d25d45a1c8d3ac15935bb1a05c@pietros-MBP.lan","name":"Pietro Pinoli"},"message":"[FLINK-2362] [docs] Add Scala and Java documentation for the 'distinct()' transformation\n\nThis closes #922","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/0818bec02c0bc90bae68f09d8d2e48dc0f6b9374"},{"sha":"bbbfd22077623633363d656d66a00028e7fea631","author":{"email":"4877b5966827a9c06b3f725e99b7442e1fa1186a@gmail.com","name":"Gabor Gevay"},"message":"[FLINK-2376] Raise the time limit in testFindConnectableAddress under Windows\n\nThis closes #920","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/bbbfd22077623633363d656d66a00028e7fea631"},{"sha":"11b48633dd421fcdf5e3cdced908ce85c1aa399d","author":{"email":"6873ed71afbc9d34f235a1376a902115740ccfeb@apache.org","name":"vasia"},"message":"[FLINK-1943] [gelly] Added GSA compiler and translation tests\n\nThis closes #916","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/11b48633dd421fcdf5e3cdced908ce85c1aa399d"},{"sha":"f798df8d4251e74096681b70b2784e11ae1e89b5","author":{"email":"78b721c5bdf8fa1e5c15c24f06246c493ee3b282@apache.org","name":"Robert Metzger"},"message":"[docs] Document some missing configuration parameters\n\nThis closes #915","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/f798df8d4251e74096681b70b2784e11ae1e89b5"},{"sha":"efafaee83c01afd9e072a1a182a392097902bad8","author":{"email":"55535687dad626de987465d57be5008b6e327a91@apache.org","name":"Stephan Ewen"},"message":"[docs] Clean up completeness checker for configuration docs.","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/efafaee83c01afd9e072a1a182a392097902bad8"},{"sha":"50344d70deb6348923f42a1ddaaf0e570708bc71","author":{"email":"4877b5966827a9c06b3f725e99b7442e1fa1186a@gmail.com","name":"Gabor Gevay"},"message":"[FLINK-2198] [tests] Under Windows, skip tests that use setWritable\n\nThis closes #919","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/50344d70deb6348923f42a1ddaaf0e570708bc71"},{"sha":"148395bcd81a93bcb1473e4e93f267edb3b71c7e","author":{"email":"d55276e323c5bc117f09cf681c93927505a487e9@air-watch.com","name":"Ajay Bhat"},"message":"[FLINK-2018] [core] Add ParameterUtil.fromGenericOptionsParser()\n\nThis closes #720","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/148395bcd81a93bcb1473e4e93f267edb3b71c7e"},{"sha":"03320503e20c2038412c889d9d8b61821d4963af","author":{"email":"b36a4b8a3b519984e26e470684cfa9887ab74e9b@informatik.hu-berlin.de","name":"mjsax"},"message":"[FLINK-2304] Add named attribute access to Storm compatibility layer\n  - extended FlinkTuple to enable named attribute access\n  - extended BoltWrapper for user defined input schema\n  - extended FlinkTopologyBuilder to handle declared output schemas\n  - adapted JUnit tests\n  - added new examples and ITCases\n  - updated READMEs\n  - updated documentation\n\nCloses #878","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/03320503e20c2038412c889d9d8b61821d4963af"},{"sha":"13762149d404d51fd8d8106bba8bbe7bfdccbdf2","author":{"email":"b36a4b8a3b519984e26e470684cfa9887ab74e9b@informatik.hu-berlin.de","name":"mjsax"},"message":"[FLINK-1658] Rename AbstractEvent to AbstractTaskEvent and AbstractJobEvent\n\nDelete AbstractEvent in org.apache.flink.runtime.event.job\n    (see comment section in https://issues.apache.org/jira/browse/FLINK-1658)\n\nThis closes #929","distinct":true,"url":"https://api.github.com/repos/chiwanpark/flink/commits/13762149d404d51fd8d8106bba8bbe7bfdccbdf2"}]}