One of the most ubiquitous challenges in analyses at the LHC is event reconstruction, whereby heavy resonance particles (such as top quarks, Higgs bosons, or vector bosons) must be reconstructed from the detector signatures left behind by their decay products. This is particularly challenging when all decay products have similar or identical signatures, such as all-jet events. Existing methods typically require the evaluation of every possible permutation of these events to find the "best" assignment. In this work, we present a novel neural network architecture ``SPANet'', or Symmetry Preserving Attention networks. By casting this problem as a set assignment problem on a variable size set, and embedding our knowledge of the symmetries in the problem into the neural network architecture, we demonstrate that these problems can be solved efficiently even in cases for which existing methods are intractable. We demonstrate the approach using a suite of progressively more complex benchmarks, going from all-hadronic ttbar, via ttH, to 4top final states, and provide an easy to use and flexible software package to design and train networks for arbitrary final states.