Good description and simple explanation is given here
Covariance matrix
The variance of a variable is a measure of the dispersion of the values taken by the variable around its mean value.
The covariance matrix generalizes the concept of variance to random vectors, or sets of random variables.
PCA and covariance matrix both are well explained. You can see some illustrations with different covariance matrix types.
All materials are from this site http://www.aiaccess.net/English/Glossaries/GlosMod/e_gm_covariance_matrix.htm#Animation_covariance%20matrix
September 13th, 2010 at 09:45
An excellent tutorial on COV and PCA see here http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf