We present work on the detection of neutral particles via nuclear recoils in gas-filled Time Projection Chambers (TPCs). We employ Gas Electron Multipliers (GEMs) to amplify the signal and silicon pixel electronics to read out the projected track. These technologies allow ionization in the target gas to be detected with low noise, improved position and time resolution, and high efficiency. We review experimental results obtained in previous years, and report on ongoing simulation studies and construction of the first prototype at the University of Hawaii. We also present prospects of using such detectors to perform direction-sensitive searches for WIMP dark matter and fast neutron from fissionable material.