Correlations imposed by the unitary limit between few-nucleon systems, nuclear matter, and neutron stars