Reidar Brumer Bratvold

Ressi Bonti Muhammad, Sergey Alyaev, Reidar Brumer Bratvold

Trajectory adjustment decisions throughout the drilling process, called geosteering, affect subsequent choices and information gathering, thus resulting in a coupled sequential decision problem. Previous works on applying decision optimization methods in geosteering rely on greedy optimization or approximate dynamic programming (ADP). Either decision optimization method requires explicit uncertainty and objective function models, making developing decision optimization methods for complex and realistic geosteering environments challenging to impossible. We use the Deep Q-Network (DQN) method, a model-free reinforcement learning (RL) method that learns directly from the decision environment, to optimize geosteering decisions. The expensive computations for RL are handled during the offline training stage. Evaluating DQN needed for real-time decision support takes milliseconds and is faster than the traditional alternatives. Moreover, for two previously published synthetic geosteering scenarios, our results show that RL achieves high-quality outcomes comparable to the quasi-optimal ADP. Yet, the model-free nature of RL means that by replacing the training environment, we can extend it to problems where the solution to ADP is prohibitively expensive to compute. This flexibility will allow applying it to more complex environments and make hybrid versions trained with real data in the future.

Ressi Bonti Muhammad, Apoorv Srivastava, Sergey Alyaev et al.

Geosteering, a key component of drilling operations, traditionally involves manual interpretation of various data sources such as well-log data. This introduces subjective biases and inconsistent procedures. Academic attempts to solve geosteering decision optimization with greedy optimization and Approximate Dynamic Programming (ADP) showed promise but lacked adaptivity to realistic diverse scenarios. Reinforcement learning (RL) offers a solution to these challenges, facilitating optimal decision-making through reward-based iterative learning. State estimation methods, e.g., particle filter (PF), provide a complementary strategy for geosteering decision-making based on online information. We integrate an RL-based geosteering with PF to address realistic geosteering scenarios. Our framework deploys PF to process real-time well-log data to estimate the location of the well relative to the stratigraphic layers, which then informs the RL-based decision-making process. We compare our method's performance with that of using solely either RL or PF. Our findings indicate a synergy between RL and PF in yielding optimized geosteering decisions.

Sergey Alyaev, Reidar Brumer Bratvold, Sofija Ivanova et al.

Geosteering workflows are increasingly based on the quantification of subsurface uncertainties during real-time operations. As a consequence operational decision making is becoming both better informed and more complex. This paper presents an experimental web-based decision support system, which can be used to both aid expert decisions under uncertainty or further develop decision optimization algorithms in controlled environment. A user of the system (either human or AI) controls the decisions to steer the well or stop drilling. Whenever a user drills ahead, the system produces simulated measurements along the selected well trajectory which are used to update the uncertainty represented by model realizations using the ensemble Kalman filter. To enable informed decisions the system is equipped with functionality to evaluate the value of the selected trajectory under uncertainty with respect to the objectives of the current experiment. To illustrate the utility of the system as a benchmark, we present the initial experiment, in which we compare the decision skills of geoscientists with those of a recently published automatic decision support algorithm. The experiment and the survey after it showed that most participants were able to use the interface and complete the three test rounds. At the same time, the automated algorithm outperformed 28 out of 29 qualified human participants. Such an experiment is not sufficient to draw conclusions about practical geosteering, but is nevertheless useful for geoscience. First, this communication-by-doing made 76% of respondents more curious about and/or confident in the presented technologies. Second, the system can be further used as a benchmark for sequential decisions under uncertainty. This can accelerate development of algorithms and improve the training for decision making.

Sergey Alyaev, Reidar Brumer Bratvold, Sofija Ivanova et al.

With the steady growth of the amount of real-time data while drilling, operational decision-making is becoming both better informed and more complex. Therefore, as no human brain has the capacity to interpret and integrate all decision-relevant information from the data, the adoption of advanced algorithms is required not only for data interpretation but also for decision optimization itself. However, the advantages of the automatic decision-making are hard to quantify. The main contribution of this paper is an experiment in which we compare the decision skills of geosteering experts with those of an automatic decision support system in a fully controlled synthetic environment. The implementation of the system, hereafter called DSS-1, is presented in our earlier work [Alyaev et al. "A decision support system for multi-target geosteering." Journal of Petroleum Science and Engineering 183 (2019)]. For the current study we have developed an easy-to-use web-based platform which can visualize and update uncertainties in a 2D geological model. The platform has both user and application interfaces (GUI and API) allowing us to put human participants and DSS-1 into a similar environment and conditions. The results of comparing 29 geoscientists with DSS-1 over three experimental rounds showed that the automatic algorithm outperformed 28 participants. What's more, no expert has beaten DSS-1 more than once over the three rounds, giving it the best comparative rating among the participants. By design DSS-1 performs consistently, that is, identical problem setup is guaranteed to yield identical decisions. The study showed that only two experts managed to demonstrate partial consistency within a tolerance but ended up with much lower scores.