Mine were the same, probably because I powder coated the bracket and the washer, (or maybe it just didn't fit in the first place)
They do need to fit tight in that pocket so they don't move around. Grind whatever is needed off one side of each washer and place that side inboard or outboard on each one so the hole stays "centered" on both the front and back washers for each side relative to each other.
Also I needed to put a chamfers along the two edges on mine to make sure they sat all the way in against the face since they were actually resting up on the inside radius of the bracket.
M