The answer is by comparing genetic values to genetic variation ina population. To calculate heritability, the following formulae are used.
H2 = VG/VP
Where H = Heritability
VG = Genetic variation
VP = Phenotypic variation
Heritability, therefore, incorporates the effects dominance and epistasis. In this case, therefore, the phenotypic variation is the ability to play the flute while genetic variation in the genotype of people able to play the flute.