There has been much interest recently in discrete toy models of holography called tensor networks. This is based primarily on the fact that entanglement between boundary regions is given by a minimal cut of the network, in a way reminiscent of the Ryu-Takayanagi formula in the case of a static bulk geometry. These models have two related shortcomings: they can't reproduce the full maximin condition relevant in the general case, and they can't reproduce the fact that the areas of two intersecting RT surfaces are non-commuting operators in the semi-classical theory. This can be traced back to the fact that the tensor networks don't satisfy any discretised version of the Hamiltonian constraint of gravity. To remedy this situation, we go back to pure GR in three dimensions, which can be rewritten as a topological Chern-Simons theory. A lattice version of Chern-Simons theory, known as the quantum double model or the Levin-Wen model, naturally satisfies a Hamiltonian constraint and also has an analog of non-commuting areas. We propose that these models can be used as a starting point for building holographic tensor networks.