saving a list of rows to a Hive table in pyspark
<p>I have a pyspark app.  I copied a hive table to my hdfs directory, &amp; in python I <code>sqlContext.sql</code> a query on this table. Now this variable is a dataframe I call <code>rows</code>.  I need to randomly shuffle the <code>rows</code>, so I had to convert them to a list of rows <code>rows_list = rows.collect()</code>.  So then I <code>shuffle(rows_list)</code> which shuffles the lists in place.   I take the amount of random rows I need <code>x</code>:</p>

<p><code>for r in range(x):
        allrows2add.append(rows_list[r])
</code>
Now I want to save allrows2add as a hive table OR append an existing hive table (whichever is easier to do).  The problem is that I can not do this:</p>

<p><code>all_df = sc.parallelize(allrows2add).toDF()</code> Cant do this, schema can't be inferred
<code>ValueError: Some of types cannot be determined by the first 100 rows, please try again with sampling</code></p>

<p>without putting in the whole schema.  The schema of <code>rows</code> has 117 columns, so I don't want to type them out.  Is there a way to extract the schema of <code>rows</code> to help me make allrows2add a dataframe or somehow save as a hive table?
I can do 
<code>rows.printSchema()</code> but not sure how to get it into a schema format as a variable to pass <code>toDF()</code> without having to parse all of that text</p>

<p>Thanks</p>

<p>Adding for loop info</p>

<pre><code>#Table is a List of Rows from small Hive table I loaded using
#query = "SELECT * FROM Table"
#Table = sqlContext.sql(query).collect()

for i in range(len(Table)):

    rows = sqlContext.sql(qry)
    val1 = Table[i][0]
    val2 = Table[i][1]
    count = Table[i][2]
    x = 100 - count

#hivetemp is a table that I copied from Hive to my hfs using:
#create external table IF NOT EXISTS hive temp LIKE hivetableIwant2copy LOCATION "/user/name/hiveBackup";
#INSERT OVERWRITE TABLE hivetemp SELECT * FROM hivetableIwant2copy;

    query = "SELECT * FROM hivetemp WHERE col1&lt;&gt;\""+val1+"\" AND col2 ==\""+val2+"\" ORDER BY RAND() LIMIT "+str(x)

    rows = sqlContext.sql(query)
    rows = rows.withColumn("col4", lit(10))
    rows = rows.withColumn("col5", lit(some_string))
#writing to parquet is heck slow AND I can't work with pandas due to the library not installed on the server
    rows.saveAsParquetFile("rows"+str(i)+".parquet")
#tried this before and heck slow also
    #rows_list = rows.collect()
    #shuffle(rows_list)
</code></pre>