Intermediate band (IB) materials are a novel class of materials that, like semiconductors, have a band gap but also have an extra set of allowed electronic levels entirely contained within the semiconductor band gap, allowing sub-gap photon absorption. Solar cells made from such materials have the potential to radically improve photovoltaic efficiencies, similar to triple-junction cells. IB materials are also explored for use as infrared photodetectors. Current IB devices are made from three classes of materials: quantum dots, highly-mismatched alloys, and hyperdoped semiconductors. None has simultaneously achieved high sub-gap absorption and sufficient carrier lifetime. I will describe theoretical and experimental work to understand carrier lifetimes and their impact on device efficiencies. I will introduce a figure of merit, which predicts the potential effectiveness of candidate IB materials for both photodetectors and solar cells in advance of device fabrication. This figure of merit captures the tradeoff between enhanced absorption and enhanced recombination within an IB material, and it suggests a path toward efficient IB materials. I will give examples of measurements of the figure of merit and demonstrate a method for theoretical predictions for new systems.