Jedidiah Carlson

Biological research is increasingly reliant on complex computational pipelines, often involving multiple steps of QCing, transforming, processing, ingesting, and summarizing data on the journey from raw biological material to scientific insights. For any one of these tasks, there could be dozens of published software tools that have been developed to parse data for subsequent analysis. A ubiquitous problem for computational biologists is simply choosing which tool to use for their particular dataset and research questions. Though independent benchmarking studies are routinely published to compare different software tools at a high level, these benchmarks often focus on broad measures of output data quality and/or computational performance using heavily vetted "gold standard" datasets. In reality, it is difficult to make judgement calls about which tools is best suited for specific data, research questions, or practical constraints. Moreover, the pace at which new tools are published makes it difficult for researchers to stay on top of (much less implement and test) emerging methods. BioBakeoff is a collection of AI agent skills that creates sandboxes for on-the-fly benchmarking of various tools for common bioinformatic tasks. Triggering the agent prompts users to provide the following: 1. Sample data 2. Relevant research question(s) being pursued by the analysis 3. Any practical constraints (compute resources, timeline, cost, etc.) With this information, BioBakeoff then spins up a customized Docker orchestrator that runs a set of candidate tools (without the user ever needing to deal with complex installations/dependencies or tool-specific outputs), aggregates the results using standardized benchmarking statistics, and produces a brief custom report detailing how the tools performed relative to one another, along with suggestions and caveats to help inform researchers' decision about which tool to proceed with. ### Notes Currently BioBakeoff focuses on variant calling as a proof-of-concept, but I will soon be extending it to other genomics analysis tasks. I have intentionally kept this as a low-level implementation rather than getting fancy with a GUI, etc. Ideally, this agent can be hooked into existing platforms and workflows.