The sector model for proteins suggests that the main units of fitness are sparse, physically connected networks of amino acid residues that connect distant functional surface sites through the protein core. Here, we develop a bacterial two-hybrid assay for PDZ function coupled to deep sequencing to globally test this model through saturation mutagenesis. We show that the pattern of mutational effects is in fact well described by the evolutionary conservation and correlations between sequence positions. However, a more specific test of correlations is to examine second-order mutational effects – the non-independence of pairs of mutations. Indeed, we show that the context-dependence of mutational effects on the background of a variation at the primary specificity site on the PDZ ligand is exclusively localized within sectors.