Mongodb-hadoop-connector create lots of partitions even if I add query to it
<p>I have a really large mongo table, I wanna do some analyze on it using spark , it's so large that I don't want to load the whole database. But it looks like it always scan the whole database and split them into tons of partitions even if I pass a <code>mongo.input.query</code> to it. I am using <a href="https://github.com/mongodb/mongo-hadoop" rel="nofollow">mongo-hadoop</a> to load it, and my codes look like this:</p>

<pre><code>val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)

val mongoConfig = new Configuration()

val beginDate = new Date(2016 - 1900,6,7)

println("the begin data is: =========== &gt;" + beginDate)

val beginId = new ObjectId(beginDate, 0,0.toShort,0)

mongoConfig.set("mongo.input.uri",
    "mongodb://mymongoduri/mongodb.mongocollection")

val queryStr = """{"_id": {"$gt" : {"$oid":"beginDate" }}}""".replace("beginDate", beginId.toString)
mongoConfig.set("mongo.input.query", queryStr)
mongoConfig.set("mongo.input.fields", """{ "its.src":-1, "its._id":-1, "its.cid": -1}""")

val documents = sc.newAPIHadoopRDD(
  mongoConfig,                // Configuration
  classOf[MongoInputFormat],  // InputFormat
  classOf[Object],            // Key type
  classOf[BSONObject])        // Value type

val OUTPUT_PATH = if(ENV == Some("dev")){
  s"./result"
} else{
  s"s3://${OUTPUT_BUCKET}/output/graph/${beginDate}"
}

documents.saveAsNewAPIHadoopFile(
  OUTPUT_PATH,
  classOf[Object],
  classOf[BSONObject],
  classOf[BSONFileOutputFormat[Object, BSONObject]]
)
</code></pre>

<p>It ends up with tons of empty files in s3, which is not my expected result (and it waste lots of money).</p>

<p>I have read the document, it said <code>mongo.input.query</code> only <code>filter the input collection with a query</code> , can I just load the data as what I query ? Not just filtering them.</p>

<p>Or, <strong>Can I just store those partition which is not empty?</strong></p>