Optimizing High-Speed Satellite Communication with High-Order Modulation via Adaptive Signal Processing at Both Ends of the Link