Draft blog post for the M2D2 blog
“Science is often described as an iterative and cumulative process, a puzzle solved piece by piece, with each piece contributing a few hazy pixels of a much larger picture.” — Emperor of all Maladies, Siddhartha Mukherjee
The Scientific Method prescribes a systematic approach to gaining knowledge through observation, forming hypotheses, and experimentation. Popularized during the Renaissance, this principle has been at the core of the rapid technological growth that followed. Progress in science has led to technological advancement, which in turn has enabled further scientific progress, resulting in a continually improving “hazy picture” of the universe. Figure 1 shows a simplified version of the Scientific Method.
To make the illustration concrete, consider the drug discovery process. It begins with the observation of a phenomenon in nature - the symptoms of a disease. These observations are then incorporated into our existing models of biology and medicine. Based on these observations and prior knowledge, several hypotheses can be formulated regarding the disease - the cause, mechanism of action, and potential therapies. These hypotheses are tested through experiments - detecting the presence of viral agents in affected organs, observing genetic pathways, testing therapies on isolated cells in-vitro, etc. At this point, completing the cycle, we return to the phase of observation, this time considering the effect of the designed experiment on the phenomenon. This cycle results in a constantly improving understanding of the phenomenon - improving our knowledge about biology and medicine, and increasingly precise and effective experiments - leading to better therapies.
Figure 1: (Simplified) Illustration of the scientific method.
The scientific method can be viewed as two complementary phases of computation and experimentation. Experimentation serves as an interface to the real world, where the phenomenon of interest is observed, intervened upon and its effects measured. Computation consists of analyzing the observations and experimental outcomes, formulating hypotheses and designing experiments to test said hypotheses. In reality the distinction between the two is often blurred. Computation and experiment have a symbiotic relationship - each one is incomplete in isolation without the other - which ultimately leads to progress. Historically, each of these phases can take considerable amounts of time. However, advancements in natural sciences have revolutionized the scale and precision with which experiments can be performed. At the same time advances in fields like machine learning have opened new avenues to accelerate computation. In this post, we focus on methods that enable us to accelerate the computation phase with data-driven approaches.
The computation phase deals with two distinct problems:
Recall the drug discovery example. Say we have identified a target protein responsible for the ailment, one can collect experimental data about the structure and binding behavior of the protein with ligands through in-vitro experiments, and build a computational model that captures this behavior, i.e. docking. Using this model, we can design ligands that can inhibit the activity of the said protein.
In reality, however, each of the steps in the example above are extremely non-trivial, resulting in long timelines for drug discovery. Recent developments in machine learning are an exciting avenue, as they enable us to build large-scale complex models of physical systems, formulate hypotheses and design experiments to accelerate the computational phase of scientific discovery.
In the last few decades, machine learning has enabled remarkable technological advances ranging from superhuman Go players to protein folding. These advances have been enabled, in part, by availability of extremely large datasets. A lot of the the approaches also assume the availability of a well specified objective to optimize. This leads us to two critical challenges in leveraging ML approaches for scientific discovery.
The first critical challenge in leveraging learning based approaches for scientific discovery is that of limited data. By design, machine learning approaches rely on access to large datasets to extract useful patterns. But owing to fundamental limitations, it can be extremely expensive or impossible to obtain large amounts of data in many applications of interest. Going back to the drug discovery example, it can be extremely difficult to obtain experimental data for small-molecules binding with a target protein, at the scale required for machine learning methods. Limited data introduces uncertainty in the models we can learn, which needs to be accounted for when formulating hypotheses with the model, as it can useful for guiding the search for novel hypotheses and experiments to disambiguate them. Bayesian models offer a principled approach to deal with limited data by modelling the posterior over functions that fit the data, however, owing to approximations required to scale to realistic data, they can underestimate the true uncertainty. [17]
Machine learning approaches often assume access to some reward signal to evaluate quality of designs. For instance, for designing drug-like molecules, the true objective is to find drug-like molecules that inhibit the target protein within the human body. This objective, however, potentially cannot be specified as a simple scalar reward. In practice, the binding energy of the molecule with the target protein is used as the reward signal to search for molecules. The binding energy alone cannot not account for a lot of the factors that can influence the effect of the drug molecule within the human body. Thus, a molecule that just minimizes this binding energy can provide no effect in the actual environment. This makes it critical to find diverse hypotheses (in this case molecules) to account for the underspecification and uncertainty in the reward signal. Widely used approaches to tackle such problems like reinforcement learning and Bayesian optimization aim to discover a single maximizer of the the reward signal, not accounting for underspecification of the reward signal itself. [15, 16]