Rolling Stones Down Potential Hills: Reshaping Gravitational Aggregates