Text this: Are queries and keys always relevant? A case study on transformer wave functions