With the great promise of deep learning, discoveries of new particles at the Large Hadron Collider (LHC) may be imminent. Following the discovery of a new particle in an all-hadronic channel, deep learning can also be used to identify the quantum numbers of the new particle. We show that convolutional neural networks (CNNs) using jet images can significantly improve upon existing techniques to identify the quantum chromodynamic (QCD) representation (`color’) of a two-prong jet using its substructure. In addition to demonstrating the capabilities of CNNs for quantum color tagging, we study what information in the jet radiation pattern is useful for classification. These techniques improve the categorization of new particles and are an important addition to the growing jet substructure toolkit, for searches and measurements at the LHC now and in the future.