How to specify multiple input paths to a Dataflow job
<p>I want to run a Dataflow job over multiple inputs from Google Cloud Storage, but the paths I want to pass to the job can't be specified with just the <code>*</code> glob operator.</p>

<p>Consider these paths:</p>

<pre><code>gs://bucket/some/path/20160208/input1
gs://bucket/some/path/20160208/input2
gs://bucket/some/path/20160209/input1
gs://bucket/some/path/20160209/input2
gs://bucket/some/path/20160210/input1
gs://bucket/some/path/20160210/input2
gs://bucket/some/path/20160211/input1
gs://bucket/some/path/20160211/input2
gs://bucket/some/path/20160212/input1
gs://bucket/some/path/20160212/input2
</code></pre>

<p>I want my job to work on the files in the <code>20160209</code>, <code>20160210</code> and <code>20160211</code> directories, but not on <code>20160208</code> (the first) and <code>20160212</code> (the last). In reality there's a lot of more dates, and I want to be able to specify an arbitrary range of dates for my job to work on.</p>

<p><a href="https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/io/TextIO.Read#from-java.lang.String-" rel="nofollow">The docs for <code>TextIO.Read</code></a> say:</p>

<blockquote>
  <p>Standard Java Filesystem glob patterns ("*", "?", "[..]") are supported.</p>
</blockquote>

<p>But I can't get this to work. There's a link to <a href="http://docs.oracle.com/javase/tutorial/essential/io/find.html" rel="nofollow">Java Filesystem glob patterns </a>, which in turn links to <a href="https://docs.oracle.com/javase/8/docs/api/java/nio/file/FileSystem.html#getPathMatcher-java.lang.String-" rel="nofollow">getPathMatcher(String)</a>, that lists all the globbing options. One of them is <code>{a,b,c}</code>, which looks exactly like what I need, however, if I pass <code>gs://bucket/some/path/201602{09,10,11}/*</code> to <code>TextIO.Read#from</code> I get "Unable to expand file pattern".</p>

<p>Maybe the docs mean that <em>only</em> <code>*</code>, <code>?</code> and <code>[…]</code> are supported, and if that is the case, how can I construct a glob that Dataflow will accept and that can match an arbitrary date range like the one I describe above?</p>

<p><strike><em>Update:</em> I've figured out that I can write a chunk of code to so that I can pass in the path prefixes as a comma separated list, create an input from each and use the <code>Flatten</code> transform, but that seems like a very inefficient way of doing it. It looks like the first step reads all input files and immediately write them out again to the temporary location on GCS. Only when all the inputs have been read and written the actual processing starts. This step is completely unnecessary in the job I'm writing. I want the job to read the first file, start processing it and read the next, and so on.</strike> This just caused a ton other problems, I'll try to make it work, but it feels like a dead end because of the initial rewriting.</p>