How to filter a map<String, Int> in a data frame : Spark / Scala
<p>I am trying to get the count individual column to publish metrics. I have a I have a <code>df [customerId : string, totalRent : bigint, totalPurchase: bigint, itemTypeCounts: map&lt;string, int&gt; ]</code> </p>

<p>Right now I am doing :</p>

<pre><code>val totalCustomers = df.count

val totalPurchaseCount = df.filter("totalPurchase &gt; 0").count

val totalRentCount = df.filter("totalRent &gt; 0").count


publishMetrics("Total Customer",  totalCustomers )
publishMetrics("Total Purchase",  totalPurchaseCount )
publishMetrics("Total Rent",  totalRentCount )

publishMetrics("Percentage of Rent",  percentage(totalRentCount, totalCustomers) )
publishMetrics("Percentage of Purchase",  percentage(totalPurchaseCount, totalCustomers) )

private def percentageCalc(num: Long, denom: Long): Double = {
val numD: Long = num
val denomD: Long = denom
return if (denomD == 0.0) 0.0
else (numD / denomD) * 100
}
</code></pre>

<p>But I am not sure how do I do this for itemTypeCounts which is a map.  I want count and percentage based on each key entry. The issue is the key value is dynamic , I mean there is no way I know the key value before hand. Can some one tell me how do get count for each key values. I am new to scala/spark, any other efficient approaches to get the counts of each columns are much appreciated. </p>

<p>Sample data :</p>

<pre><code>customerId : 1
totalPurchase : 17
totalRent : 0
itemTypeCounts : {"TV" : 4, "Blender" : 2}

customerId : 2
totalPurchase : 1
totalRent : 1
itemTypeCounts : {"Cloths" : 4}

customerId : 3
totalPurchase : 0
totalRent : 10
itemTypeCounts : {"TV" : 4}
</code></pre>

<p>So the output is :</p>

<pre><code>totalCustomer : 3
totalPurchaseCount : 2 (2 customers with totalPurchase &gt; 0)
totalRent : 2 (2 customers with totalRent &gt; 0)
itemTypeCounts_TV : 2
itemTypeCounts_Cloths  : 1
itemTypeCounts_Blender  : 1
</code></pre>