A core component of particle tracking algorithms in LHC experiments is the Kalman Filter. Its capability to iteratively model dynamics (linear or non-linear) in noisy data makes it powerful for state estimation and extrapolation in a combinatorial track builder (the CKF). In practice, the CKF computational cost scales quadratically with the detector occupancy and will become a heavy burden on computing resources during the HL-LHC. Alternatively, recurrent neural networks (RNNs) have also been shown to be effective at state estimation and modeling non-linear dependencies in sequence data. We therefore propose an approach to particle track building that utilizes an RNN-based tree search algorithm. The RNN model, utilizing Long Short-term Memory (LSTM) units, takes a sequence of candidate track hits and predicts the location of a hit on a successive detector layer. We have developed two types of prediction models: one which produces single value predictions and another which produces Gaussian predictions in the form of central values and a covariance matrix. The latter model is thus able to express uncertainty in its predictions and is trained using a log-likelihood cost function. The hit predictions are used to select candidate hits from the data. By including an additional overall track quality estimate, we are able to efficiently explore the combinatorial search tree by prioritizing and extrapolating only the most likely hits until the track is fully formed. In this contribution we will present the RNN track building algorithm and discuss its performance for track-finding on simulated ACTS data compared to traditional solutions.