{"push_id":576003543,"size":1,"distinct_size":1,"ref":"refs/heads/beorn7/ingestion-tweaks","head":"8ccb1c4f6b199a3406b440236c254acf9ec74031","before":"760decb809df17b42b9966744730c4d95d767e63","commits":[{"sha":"8ccb1c4f6b199a3406b440236c254acf9ec74031","author":{"email":"91e32ae1d26bd312b370cf8378a5a83accca90a0@soundcloud.com","name":"beorn7"},"message":"Improve persisting chunks to disk.\n\nThis is done by bucketing chunks by fingerprint. If the persisting to\ndisk falls behind, more and more chunks are in the queue. As soon as\nthere are \"double hits\", we will now persist both chunks in one go,\ndoubling the disk throughput (assuming it is limited by disk\nseeks). Should even more pile up so that we end wit \"triple hits\", we\nwill persist those first, and so on.\n\nEven if we have millions of time series, this will still help,\nassuming not all of them are growing with the same speed. Series that\nget many samples and/or are not very compressable will accumulate\nchunks faster, and they will soon get double- or triple-writes.\n\nTo improve the chance of double writes,\n-storage.local.persistence-queue-capacity could be set to a higher\nvalue. However, that will slow down shutdown a lot (as the queue has\nto be worked through). So we leave it to the user to set it to a\nreally high value. A more fundamental solution would be to checkpoint\nnot only head chunks, but also chunks still in the persist queue. That\nwould be quite complicated for a rather limited use-case (running many\ntime series with high ingestion rate on slow spinning disks).","distinct":true,"url":"https://api.github.com/repos/prometheus/prometheus/commits/8ccb1c4f6b199a3406b440236c254acf9ec74031"}]}