Machine learning methods are becoming ubiquitous across the LHC and particle physics. However, the exploration of such techniques within the field in low latency, low power FPGA hardware has only just begun. There is great potential to improve trigger and data acquisition performance, more generally for pattern recognition problems, and potentially beyond. We present a case study for using neural networks in FPGAs. Our study takes jet substructure as an example since it is a field familiar with machine learning, but lessons are far-reaching. We map out resource usage and latency versus types of machine learning algorithms and their hyper-parameters to identify the problems in particle physics that would benefit. We develop a package based on High Level Synthesis (HLS) to build network architectures which is readily accessible to a broad user base.