Azure Blob support in Apache Beam?
<p>I'm wondering if Apache Beam supports windows azure storage blob files(wasb) IO. Is there any support yet?</p>

<p>I'm asking because I've deployed an apache beam application to run a job on an Azure Spark cluster and basically it's impossible to IO wasb files from the associated storage container with that spark cluster. Is there any alternative solution?</p>

<p><strong>Context</strong>: I'm trying to run the <a href="https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java" rel="nofollow noreferrer">WordCount example</a> on my Azure Spark Cluster. Already had set some components as stated <a href="https://stackoverflow.com/a/37764434/1757779">here</a> believing that would help me. Below is the part of my code where I'm setting the hadoop configuration: </p>

<pre><code>final SparkPipelineOptions options = PipelineOptionsFactory.create().as(SparkPipelineOptions.class);

options.setAppName("WordCountExample");
options.setRunner(SparkRunner.class);
options.setSparkMaster("yarn");
JavaSparkContext context = new JavaSparkContext();
Configuration conf = context.hadoopConfiguration();
conf.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem");
conf.set("fs.azure.account.key.&lt;storage-account&gt;.blob.core.windows.net",
         "&lt;key&gt;");
options.setProvidedSparkContext(context);
Pipeline pipeline = Pipeline.create(options);
</code></pre>

<p>But unfortunately I keep ending with the following error:</p>

<pre><code>java.lang.IllegalStateException: Failed to validate wasb://&lt;storage-container&gt;@&lt;storage-account&gt;.blob.core.windows.net/user/spark/kinglear.txt
at org.apache.beam.sdk.io.TextIO$Read$Bound.apply(TextIO.java:288)
at org.apache.beam.sdk.io.TextIO$Read$Bound.apply(TextIO.java:195)
at org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76)
at org.apache.beam.runners.spark.SparkRunner.apply(SparkRunner.java:129)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:400)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:323)
at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:58)
at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:173)
at spark.example.WordCount.main(WordCount.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627)
Caused by: java.io.IOException: Unable to find handler for  wasb://&lt;storage-container&gt;@&lt;storage-account&gt;.blob.core.windows.net/user/spark/kinglear.txt
at org.apache.beam.sdk.util.IOChannelUtils.getFactory(IOChannelUtils.java:187)
at org.apache.beam.sdk.io.TextIO$Read$Bound.apply(TextIO.java:283)
... 13 more
</code></pre>

<p>I was thinking about implementing a customized IO for Apache Beam in this case for Azure Storage Blobs if comes to that as a solution, I'd like to check with the community if that's an alternative solution.</p>