Boundary-locality and perturbative structure of entanglement spectra in gapped systems