Gower distance on big data
<p>I want to perform clustering on geographical data to classify types of landscape within my region.</p>

<p>My data consist of five variables (temperature, amplitude of temperature, precipitation, altitude and soil type) for each field of regular grid. I have a bit more than 1  million fields (=rows in data frame).</p>

<p>Four of the variables are numeric and soil type is categorical variable described as a number. (Numeric data are already standardized.) I decided to count Gower distance dissimilarity matrix and to perform PCA and hierarchical clustering on this matrix. However, the data is too big.</p>

<pre><code>   SOIL  PREC     TEMP     ALT      AMP
0  6     1.000    1.146    0.157   -0.579
1  6     0.948    1.224    0.154   -0.579
2  5     1.000    1.146    0.201   -0.662
3  6     1.078    1.093    0.177   -0.620
4  6     1.000    1.146    0.182   -0.620
5  6     1.000    1.146    0.186   -0.599
</code></pre>

<p>I don't want to sample, because variables are in gradients. I tried to count frequencies and perform Gower distance on smaller data, but it's still too big.</p>

<p>I think I might 
(1) manually chunk the big dataset, 
(2) to each chunk matrix add two extra rows with max and min values of the variables as a "description" of extent of each variable for the distance analysis, 
(3) count the dissimilarity matrix for each chunk with <a href="http://www.inside-r.org/packages/cran/StatMatch/docs/gower.dist" rel="nofollow">gower.dist</a> function,
(4) remove the extra rows and 
(5) merge all of the chunk dissimilarity matrices into one big dissimilarity matrix.</p>

<p>Do you consider this a correct and working way? Do you have any other suggestions how to deal this issue?</p>

<p>Is it correct to perform PCA on dissimilarity matrix?</p>