Many AI tools are being tested through clinical trials or implementation studies to test their effects on actual outcomes. However, because these tools are so dependent on patterns learned from data and how this relates to data encountered in real-life settings, the patterns at one health institution might be significantly different from those of another. This opens institutions to potential dangers such as inefficient translation, bias, and operational problems, late detection of model failures, patient harm, and loss of trust in health care systems, squandering the potential benefits of AI to improving lives. This talk will cover our recent work that contrasts clinician cognition and AI – to better understand the context of use and the research needed to ensure safe and effective implementation. It will also introduce the Collaboration for trANslational Artificial Intelligence tRIals (CANAIRI) Project. The collaboration brings together world leading researchers across the globe to develop consensus-based standards of best practices and key capabilities for conducting Health AI evaluations. We advocate for a widening of the current view on the value of these trials toward one that is sociotechnical in nature, operationalized through a set of evaluative processes as recommendations for healthcare settings. When scoped holistically, we believe that translational trials can provide a consistent basis to make evidence-based decisions about AI integration and operationalize institutional accountabilities. The ability to conduct these trials, is a core capability for any health setting which wishes to utilize AI. Translating this technology safely and responsibly into health care settings requires multidisciplinary expertise alongside of complex data sets that encompass the entire health care system.