Constraining black hole mimickers with gravitational wave observations