DescriptionSafety-critical applications (e.g., autonomous vehicles, human-machine teaming, and automated medical diagnosis) often require the use of computational agents that are capable of understanding and reasoning about the high-level content of real-world scene images in order to make rational and grounded decisions that can be trusted by humans. Many of these agents rely on machine learning-based models which are increasingly being treated as black-boxes. One way to increase model interpretability is to make explainability a core principle of the model, e.g., by forcing deep neural networks to explicitly learn grounded and interpretable features. In this thesis, I provide a high-level overview of the field of explainable/interpretable machine learning and review some existing approaches for interpreting neural networks used for computer vision tasks. I also introduce four novel approaches for making convolutional neural networks (CNNs) more interpretable by utilizing explainability as a guiding principle when designing the model architecture. Finally, I discuss some possible future research directions involving explanation-driven machine learning.