The castle was extended, and in 1582 the Town's College (the precursor of the University of Edinburgh) was founded.
The University of Edinburgh started as a much smaller castle. 
The University of Edinburgh didn't start as a much smaller castle.