Spring Batch custom completion policy for dynamic chunk size
<p><strong>Context</strong></p>

<p>We have a batch job that replicates localized country names (i.e. translations of country names to different languages) to our DB from the external one. The idea was to process all localized country names for a single country in 1 chunk (i.e. first chunk - all translations for Andorra, next chunk - all translations for U.A.E., etc.). We use <code>JdbcCursorItemReader</code> for reading external data + some oracle analytic functions to provide total number of translations available for the country: something like </p>

<pre><code>select country_code, language_code, localized_name, COUNT(1) OVER(PARTITION BY c_lng.country_code) as lng_count
from EXT_COUNTRY_LNG c_lng
order by c_lng.countty_code, c_lng.language_code
</code></pre>

<p><strong>Problem</strong></p>

<p>So cutting this input by chunks looks simple: stop chunk when you've read the exact amount of rows specified in <code>lng_count</code> and start a new one with the next read row, but it appears not to be so simple practically :(</p>

<p>First thing to try is a custom completion policy. But the problem is, it doesn't have access to the last item, read by <code>ItemReader</code> - you should explicitly put it to context in reader and get it back in policy. Don't like it 'cause it requires additional reader modifications/adding reader listeners. Moreover I don't like the same item being serialized/deserialized back and forth. And I don't feel like <code>JobContext</code>/<code>StepContext</code> is a good place for such data.</p>

<p>There's also <code>RepeatContext</code> which looks like a better place for such data, but I was not able to get to it <em>easily</em>...</p>

<p>So finally we end up with solution like this:</p>

<pre><code>@Bean(name = "localizedCountryNamesStep")
@JobScope
public Step insertCountryStep(
        final StepBuilderFactory stepBuilderFactory,
        final MasterdataCountryNameReader countryNameReader,
        final MasterdataCountryNameProcessor countryNameProcessor,
        final MasterdataCountryNameWriter writer) {
    /* Use the same fixed-commit policy, but update it's chunk size dynamically */
    final SimpleCompletionPolicy policy = new SimpleCompletionPolicy();
    return stepBuilderFactory.get("localizedCountryNamesStep")
            .&lt;ExtCountryLng, LocalizedCountryName&gt; chunk(policy)
            .reader(countryNameReader)
            .listener(new ItemReadListener&lt;ExtCountryLng&gt;() {

                @Override
                public void beforeRead() {
                    // do nothing
                }

                @Override
                public void afterRead(final ExtCountryLng item) {
                    /* Update the cunk size after every read: consequent reads 
                    inside the same country = same chunk do nothing since lngCount is always the same there */
                    policy.setChunkSize(item.getLngCount());
                }

                @Override
                public void onReadError(final Exception ex) {
                    // do nothing
                }
            })
            .processor(countryNameProcessor)
            .writer(writer)
            .faultTolerant()
            .skip(RuntimeException.class)
            .skipLimit(Integer.MAX_VALUE) // Batch does not support unlimited skip
            .retryLimit(0) // this solution disables only retry, but not recover
            .build();
}
</code></pre>

<p>It's working, it requires minimum code changes, but it's still a bit ugly for me. So I'm wondering, is there another elegant way to do a dynamic chunk size in Spring Batch when all the required information is already available at the <code>ItemReader</code>?</p>