Text this: Symmetries of the gravitational scattering in the absence of peeling