Publication

Refereed papers

1. Keisuke Fujii, Koh Takeuchi, Atsushi Kuribayashi, Naoya Takeishi, Yoshinobu Kawahara, Kazuya Takeda

   Estimating Counterfactual Treatment Outcomes Over Time in Complex Multi-Agent Scenarios

   IEEE Transactions on Neural Networks and Learning Systems, 2024

   (in press)

   Abs arXiv ver.

   Evaluation of intervention in a multi-agent system, e.g., when humans should intervene in autonomous driving systems and when a player should pass to teammates for a good shot, is challenging in various engineering and scientific fields. Estimating the individual treatment effect (ITE) using counterfactual long-term prediction is practical to evaluate such interventions. However, most of the conventional frameworks did not consider the time-varying complex structure of multi-agent relationships and covariate counterfactual prediction. This may lead to erroneous assessments of ITE and difficulty in interpretation. Here we propose an interpretable, counterfactual recurrent network in multi-agent systems to estimate the effect of the intervention. Our model leverages graph variational recurrent neural networks and theory-based computation with domain knowledge for the ITE estimation framework based on long-term prediction of multi-agent covariates and outcomes, which can confirm the circumstances under which the intervention is effective. On simulated models of an automated vehicle and biological agents with time-varying confounders, we show that our methods achieved lower estimation errors in counterfactual covariates and the most effective treatment timing than the baselines. Furthermore, using real basketball data, our methods performed realistic counterfactual predictions and evaluated the counterfactual passes in shot scenarios.

2. Keisuke Fujii, Kazushi Tsutsui, Atom Scott, Hiroshi Nakahara, Naoya Takeishi, Yoshinobu Kawahara

   Adaptive Action Supervision in Reinforcement Learning from Real-World Multi-Agent Demonstrations

   In Proceedings of the 16th International Conference on Agents and Artificial Intelligence, pp. 27–39, 2024

   Abs arXiv ver.

   Modeling of real-world biological multi-agents is a fundamental problem in various scientific and engineering fields. Reinforcement learning (RL) is a powerful framework to generate flexible and diverse behaviors in cyberspace; however, when modeling real-world biological multi-agents, there is a domain gap between behaviors in the source (i.e., real-world data) and the target (i.e., cyberspace for RL), and the source environment parameters are usually unknown. In this paper, we propose a method for adaptive action supervision in RL from real-world demonstrations in multi-agent scenarios. We adopt an approach that combines RL and supervised learning by selecting actions of demonstrations in RL based on the minimum distance of dynamic time warping for utilizing the information of the unknown source dynamics. This approach can be easily applied to many existing neural network architectures and provide us with an RL model balanced between reproducibility as imitation and generalization ability to obtain rewards in cyberspace. In the experiments, using chase-and-escape and football tasks with the different dynamics between the unknown source and target environments, we show that our approach achieved a balance between the reproducibility and the generalization ability compared with the baselines. In particular, we used the tracking data of professional football players as expert demonstrations in football and show successful performances despite the larger gap between behaviors in the source and target environments than the chase-and-escape task.

3. Keisuke Fujii, Naoya Takeishi, Yoshinobu Kawahara, Kazuya Takeda

   Decentralized Policy Learning with Partial Observation and Mechanical Constraints for Multi-person Modeling

   Neural Networks, 171:40–52, 2024

   Abs arXiv ver.

   Extracting the rules of real-world multi-agent behaviors is a current challenge in various scientific and engineering fields. Biological agents independently have limited observation and mechanical constraints; however, most of the conventional data-driven models ignore such assumptions, resulting in lack of biological plausibility and model interpretability for behavioral analyses. Here we propose sequential generative models with partial observation and mechanical constraints in a decentralized manner, which can model agents’ cognition and body dynamics, and predict biologically plausible behaviors. We formulate this as a decentralized multi-agent imitation-learning problem, leveraging binary partial observation and decentralized policy models based on hierarchical variational recurrent neural networks with physical and biomechanical penalties. Using real-world basketball and soccer datasets, we show the effectiveness of our method in terms of the constraint violations, long-term trajectory prediction, and partial observation. Our approach can be used as a multi-agent simulator to generate realistic trajectories using real-world data.

4. Maciej Falkiewicz, Naoya Takeishi, Imahn Shekhzadeh, Antoine Wehenkel, Arnaud Delaunoy, Gilles Louppe, Alexandros Kalousis

   Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probability

   In Advances in Neural Information Processing Systems 36, pp. 1082–1099

   Abs arXiv ver. Code

   Bayesian inference allows expressing the uncertainty of posterior belief under a probabilistic model given prior information and the likelihood of the evidence. Predominantly, the likelihood function is only implicitly established by a simulator posing the need for simulation-based inference (SBI). However, the existing algorithms can yield overconfident posteriors (Hermans et al., 2022) defeating the whole purpose of credibility if the uncertainty quantification is inaccurate. We propose to include a calibration term directly into the training objective of the neural model in selected SBI techniques. By introducing a relaxation of the classical formulation of calibration error we enable end-to-end backpropagation. The proposed method is not tied to any particular neural model and brings moderate computational overhead compared to the profits it introduces. It is directly applicable to existing computational pipelines allowing reliable black-box posterior inference. We empirically show on six benchmark problems that the proposed method achieves competitive or better results in terms of coverage and expected posterior density than the previously existing approaches.

5. Naoya Takeishi, Yoshinobu Kawahara

   A Characteristic Function for Shapley-Value-Based Attribution of Anomaly Scores

   Transactions on Machine Learning Research, 2023

   Abs Code

   In anomaly detection, the degree of irregularity is often summarized as a real-valued anomaly score. We address the problem of attributing such anomaly scores to input features for interpreting the results of anomaly detection. We particularly investigate the use of the Shapley value for attributing anomaly scores of semi-supervised detection methods. We propose a characteristic function specifically designed for attributing anomaly scores. The idea is to approximate the absence of some features by locally minimizing the anomaly score with regard to the to-be-absent features. We examine the applicability of the proposed characteristic function and other general approaches for interpreting anomaly scores on multiple datasets and multiple anomaly detection methods. The results indicate the potential utility of the attribution methods including the proposed one.

6. Naoya Takeishi, Alexandros Kalousis

   Deep Grey-Box Modeling With Adaptive Data-Driven Models Toward Trustworthy Estimation of Theory-Driven Models

   In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, pp. 4089–4100, 2023

   Abs arXiv ver. Code Poster

   The combination of deep neural nets and theory-driven models, which we call deep grey-box modeling, can be inherently interpretable to some extent thanks to the theory backbone. Deep grey-box models are usually learned with a regularized risk minimization to prevent a theory-driven part from being overwritten and ignored by a deep neural net. However, an estimation of the theory-driven part obtained by uncritically optimizing a regularizer can hardly be trustworthy when we are not sure what regularizer is suitable for the given data, which may harm the interpretability. Toward a trustworthy estimation of the theory-driven part, we should analyze regularizers’ behavior to compare different candidates and to justify a specific choice. In this paper, we present a framework that enables us to analyze a regularizer’s behavior empirically with a slight change in the neural net’s architecture and the training objective.

7. Keisuke Fujii, Koh Takeuchi, Atsushi Kuribayashi, Naoya Takeishi, Yoshinobu Kawahara, Kazuya Takeda

   Estimating Counterfactual Treatment Outcomes over Time in Complex Multi-Vehicle Simulation

   In Proceedings of the 30th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, 2022

   Best Poster Paper Award winner

   Abs

   Evaluation of intervention in a multi-agent system, e.g., when humans should intervene in autonomous driving systems, is challenging in various engineering and scientific fields. Estimating the individual treatment effect (ITE) using counterfactual long-term prediction is practical to evaluate such interventions. However, most of the conventional frameworks did not consider the time-varying complex structure of multi-agent relationships and covariate counterfactual prediction. Here we propose an interpretable, counterfactual recurrent network in multi-agent systems to estimate the effect of the intervention. Our model leverages graph variational recurrent neural networks and theory-based computation with domain knowledge for the ITE estimation framework based on long-term prediction of multi-agent covariates and outcomes, which can confirm the circumstances under which the intervention is effective. On simulated models of an automated vehicle with time-varying confounders, we show that our methods achieved lower estimation errors in counterfactual covariates.

8. Naoya Takeishi, Keisuke Fujii, Koh Takeuchi, Yoshinobu Kawahara

   Discriminant Dynamic Mode Decomposition for Labeled Spatiotemporal Data Collections

   SIAM Journal on Applied Dynamical Systems, 21(2):1030–1058, 2022

   Abs arXiv ver.

   Extracting coherent patterns is one of the standard approaches toward understanding spatiotemporal data. Dynamic mode decomposition (DMD) is a powerful tool for extracting coherent patterns, but the original DMD and most of its variants do not consider label information, which is often available as side information of spatiotemporal data. In this work, we propose a new method for extracting distinctive coherent patterns from labeled spatiotemporal data collections such that they contribute to major differences in a labeled set of dynamics. We achieve such pattern extraction by incorporating discriminant analysis into DMD. To this end, we define a kernel function on subspaces spanned by sets of dynamic modes and develop an objective to take both reconstruction goodness as DMD and class-separation goodness as discriminant analysis into account. We illustrate our method using a synthetic dataset and several real-world datasets. The proposed method can be a useful tool for exploratory data analysis for understanding spatiotemporal data.

9. Naoya Ozaki, Kanta Yanagida, Takuya Chikazawa, Nishanth Pushparaj, Naoya Takeishi, Ryuki Hyodo

   Asteroid Flyby Cycler Trajectory Design Using Deep Neural Networks

   Journal of Guidance, Control, and Dynamics, 45(8):1496–1511, 2022

   Abs arXiv ver.

   Asteroid exploration has been attracting more attention in recent years. Nevertheless, we have just visited tens of asteroids, whereas we have discovered more than 1 million bodies. As our current observation and knowledge should be biased, it is essential to explore multiple asteroids directly to better understand the remains of planetary building materials. One of the mission design solutions is utilizing asteroid flyby cycler trajectories with multiple Earth gravity assists. An asteroid flyby cycler trajectory design problem is a subclass of global trajectory optimization problems with multiple flybys, involving a trajectory optimization problem for a given flyby sequence and a combinatorial optimization problem to decide the sequence of the flybys. As the number of flyby bodies grows, the computation time of this optimization problem expands maliciously. This paper presents a new method to design asteroid flyby cycler trajectories utilizing a surrogate model constructed by deep neural networks approximating trajectory optimization results. Because one of the bottlenecks of machine learning approaches is the heavy computation time to generate massive trajectory databases, we propose an efficient database generation strategy by introducing pseudo-asteroids satisfying the Karush– Kuhn– Tucker conditions. The numerical result applied to Japan Aerospace Exploration Agency’s DESTINY+ mission shows that the proposed method is practically applicable to space mission design and can significantly reduce the computational time for searching asteroid flyby sequences.

10. Keisuke Fujii, Naoya Takeishi, Kazushi Tsutsui, Emyo Fujioka, Nozomi Nishiumi, Ryoya Tanaka, Mika Fukushiro, Kaoru Ide, Hiroyoshi Kohno, Ken Yoda, Susumu Takahashi, Shizuko Hiryu, Yoshinobu Kawahara

    Learning Interaction Rules from Multi-Animal Trajectories via Augmented Behavioral Models

    In Advances in Neural Information Processing Systems 34, pp. 11108–11122, 2021

    Abs arXiv ver.

    Extracting the interaction rules of biological agents from movement sequences pose challenges in various domains. Granger causality is a practical framework for analyzing the interactions from observed time-series data; however, this framework ignores the structures and assumptions of the generative process in animal behaviors, which may lead to interpretational problems and sometimes erroneous assessments of causality. In this paper, we propose a new framework for learning Granger causality from multi-animal trajectories via augmented theory-based behavioral models with interpretable data-driven models. We adopt an approach for augmenting incomplete multi-agent behavioral models described by time-varying dynamical systems with neural networks. For efficient and interpretable learning, our model leverages theory-based architectures separating navigation and motion processes, and the theory-guided regularization for reliable behavioral modeling. This can provide interpretable signs of Granger-causal effects over time, i.e., when specific others cause the approach or separation. In experiments using synthetic datasets, our method achieved better performance than various baselines. We then analyzed multi-animal datasets of mice, flies, birds, and bats, which verified our method and obtained novel biological insights.

11. Naoya Takeishi, Alexandros Kalousis

    Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling

    In Advances in Neural Information Processing Systems 34, pp. 14809–14821, 2021

    Abs arXiv ver. Code Poster

    Integrating physics models within machine learning models holds considerable promise toward learning robust models with improved interpretability and abilities to extrapolate. In this work, we focus on the integration of incomplete physics models into deep generative models. In particular, we introduce an architecture of variational autoencoders (VAEs) in which a part of the latent space is grounded by physics. A key technical challenge is to strike a balance between the incomplete physics and trainable components such as neural networks for ensuring that the physics part is used in a meaningful manner. To this end, we propose a regularized learning method that controls the effect of the trainable components and preserves the semantics of the physics-based latent variables as intended. We not only demonstrate generative performance improvements over a set of synthetic and realworld datasets, but we also show that we learn robust models that can consistently extrapolate beyond the training distribution in a meaningful manner. Moreover, we show that we can control the generative process in an interpretable manner.

12. Naoya Takeishi, Yoshinobu Kawahara

    Learning Dynamics Models with Stable Invariant Sets

    In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pp. 9782–9790, 2021

    Abs arXiv ver. Code Slides

    Invariance and stability are essential notions in dynamical systems study, and thus it is of great interest to learn a dynamics model with a stable invariant set. However, existing methods can only handle the stability of an equilibrium. In this paper, we propose a method to ensure that a dynamics model has a stable invariant set of general classes such as limit cycles and line attractors. We start with the approach by Manek and Kolter (2019), where they use a learnable Lyapunov function to make a model stable with regard to an equilibrium. We generalize it for general sets by introducing projection onto them. To resolve the difficulty of specifying a to-be stable invariant set analytically, we propose defining such a set as a primitive shape (e.g., sphere) in a latent space and learning the transformation between the original and latent spaces. It enables us to compute the projection easily, and at the same time, we can maintain the model’s flexibility using various invertible neural networks for the transformation. We present experimental results that show the validity of the proposed method and the usefulness for long-term prediction.

13. Anand Srinivasan, Naoya Takeishi

    An MCMC Method for Uncertainty Set Generation via Operator-Theoretic Metrics

    In Proceedings of the 2020 IEEE Conference on Decision and Control, pp. 2714–2719, 2020

    Abs arXiv ver.

    Model uncertainty sets are required in many robust optimization problems, such as robust control and prediction with uncertainty, but there is no definite methodology to generate uncertainty sets for nonlinear dynamical systems. In this paper, we propose a method for model uncertainty set generation via Markov chain Monte Carlo. The proposed method samples from distributions over dynamical systems via metrics over transfer operators and is applicable to general nonlinear systems. We adapt Hamiltonian Monte Carlo for sampling high-dimensional transfer operators in a computationally efficient manner. We present numerical examples to validate the proposed method for uncertainty set generation.

14. Naoya Takeishi, Yoshinobu Kawahara

    Learning Multiple Nonlinear Dynamical Systems with Side Information

    In Proceedings of the 2020 IEEE Conference on Decision and Control, pp. 3206–3211, 2020

    Abs PDF Slides

    We address the problem of learning multiple dynamical systems, which is a kind of multi-task learning (MTL). The existing methods of MTL do not apply to learning dynamical systems in general. In this work, we develop a regularization method to perform MTL for dynamical systems appropriately. The proposed method is based on an operator-theoretic metric on dynamics that is agnostic of model parametrization and applicable even for nonlinear dynamics models. We calculate the proposed MTL-like regularization by estimating the metric from trajectories generated during training. Learning timevarying systems can be regarded as a special case of the usage of the proposed method. The proposed regularizer is versatile as we can straightforwardly incorporate it into offthe-shelf gradient-based optimization methods. We show the results of experiments on synthetic and real-world datasets, which exhibits the validity of the proposed regularizer.

15. Naoya Takeishi, Yoshinobu Kawahara

    Knowledge-Based Regularization in Generative Modeling

    In Proceedings of the 29th International Joint Conference on Artificial Intelligence, pp. 2390–2396, 2020

    Abs arXiv ver. Slides

    Prior domain knowledge can greatly help to learn generative models. However, it is often too costly to hard-code prior knowledge as a specific model architecture, so we often have to use general-purpose models. In this paper, we propose a method to incorporate prior knowledge of feature relations into the learning of general-purpose generative models. To this end, we formulate a regularizer that makes the marginals of a generative model to follow prescribed relative dependence of features. It can be incorporated into off-the-shelf learning methods of many generative models, including variational autoencoders and generative adversarial networks, as its gradients can be computed using standard backpropagation techniques. We show the effectiveness of the proposed method with experiments on multiple types of datasets and generative models.

16. Yoshiyuki Anzai, Takehisa Yairi, Naoya Takeishi, Yuichi Tsuda, Naoko Ogawa

    Visual Localization for Asteroid Touchdown Operation Based on Local Image Features

    Astrodynamics, 4(2):149–161, 2020

    Abs

    In an asteroid sample-return mission, accurate position estimation of the spacecraft relative to the asteroid is essential for landing at the target point. During the missions of Hayabusa and Hayabusa2, the main part of the visual position estimation procedure was performed by human operators on the Earth based on a sequence of asteroid images acquired and sent by the spacecraft. Although this approach is still adopted in critical space missions, there is an increasing demand for automated visual position estimation, so that the time and cost of human intervention may be reduced. In this paper, we propose a method for estimating the relative position of the spacecraft and asteroid during the descent phase for touchdown from an image sequence using state-of-the-art techniques of image processing, feature extraction, and structure from motion. We apply this method to real Ryugu images that were taken by Hayabusa2 from altitudes of 20 km-500 m. It is demonstrated that the method has practical relevance for altitudes within the range of 5-1 km. This result indicates that our method could improve the efficiency of the ground operation in the global mapping and navigation during the touchdown sequence, whereas full automation and autonomous on-board estimation are beyond the scope of this study. Furthermore, we discuss the challenges of developing a completely automatic position estimation framework.

17. Keisuke Fujii, Naoya Takeishi, Motokazu Hojo, Yuki Inaba, Yoshinobu Kawahara

    Physically-Interpretable Classification of Biological Network Dynamics for Complex Collective Motions

    Scientific Reports, 10(1):3005, 2020

    Abs

    Understanding biological network dynamics is a fundamental issue in various scientific and engineering fields. Network theory is capable of revealing the relationship between elements and their propagation; however, for complex collective motions, the network properties often transiently and complexly change. A fundamental question addressed here pertains to the classification of collective motion network based on physically-interpretable dynamical properties. Here we apply a data-driven spectral analysis called graph dynamic mode decomposition, which obtains the dynamical properties for collective motion classification. Using a ballgame as an example, we classified the strategic collective motions in different global behaviours and discovered that, in addition to the physical properties, the contextual node information was critical for classification. Furthermore, we discovered the label-specific stronger spectra in the relationship among the nearest agents, providing physical and semantic interpretations. Our approach contributes to the understanding of principles of biological complex network dynamics from the perspective of nonlinear dynamical systems.

18. Naoya Takeishi

    Kernel Learning for Data-Driven Spectral Analysis of Koopman Operators

    In Proceedings of the 11th Asian Conference on Machine Learning, pp. 956–971, 2019

    Abs Poster

    Spectral analysis of the Koopman operators is a useful tool for studying nonlinear dynamical systems and has been utilized in various branches of science and engineering for purposes such as understanding complex phenomena and designing a controller. Several methods to compute the Koopman spectral analysis have been studied, among which data-driven methods are attracting attention. We focus on one of the popular data-driven methods, which is based on the Galerkin approximation of the operator using a basis estimated in a data-driven manner via the diffusion maps algorithm. The performance of this method with a finite amount of data depends on the choice of the kernel function used in diffusion maps, which creates a need for kernel selection. In this paper, we propose a method to learn the kernel function adaptively to obtain better performance in approximating spectra of the Koopman operator using the Galerkin approximation with diffusion maps. The proposed method depends on the multiple kernel learning scheme, and our objective function is based on the idea that a diffusion operator should commute with the Koopman operator. We also show the effectiveness of the proposed method empirically with numerical examples.

19. Keisuke Fujii, Naoya Takeishi, Benio Kibushi, Motoki Kouzaki, Yoshinobu Kawahara

    Data-Driven Spectral Analysis for Coordinative Structures in Periodic Human Locomotion

    Scientific Reports, 9(1):16755, 2019

    Abs

    Living organisms dynamically and flexibly operate a great number of components. As one of such redundant control mechanisms, low-dimensional coordinative structures among multiple components have been investigated. However, structures extracted from the conventional statistical dimensionality reduction methods do not reflect dynamical properties in principle. Here we regard coordinative structures in biological periodic systems with unknown and redundant dynamics as a nonlinear limit-cycle oscillation, and apply a data-driven operator-theoretic spectral analysis, which obtains dynamical properties of coordinative structures such as frequency and phase from the estimated eigenvalues and eigenfunctions of a composition operator. Using segmental angle series during human walking as an example, we first extracted the coordinative structures based on dynamics; e.g. the speed-independent coordinative structures in the harmonics of gait frequency. Second, we discovered the speed-dependent time-evolving behaviours of the phase by estimating the eigenfunctions via our approach on the conventional low-dimensional structures. We also verified our approach using the double pendulum and walking model simulation data. Our results of locomotion analysis suggest that our approach can be useful to analyse biological periodic phenomena from the perspective of nonlinear dynamical systems.

20. Naoya Takeishi

    Shapley Values of Reconstruction Errors of PCA for Explaining Anomaly Detection

    In Proceedings of the 2019 International Conference on Data Mining Workshops, pp. 793–798, 2019

    The published version has typos in Eqs. (14) and (18), which are fixed in the arXiv version.

    Abs arXiv ver. Code Slides

    We present a method to compute the Shapley values of reconstruction errors of principal component analysis (PCA), which is particularly useful in explaining the results of anomaly detection based on PCA. Because features are usually correlated when PCA-based anomaly detection is applied, care must be taken in computing a value function for the Shapley values. We utilize the probabilistic view of PCA, particularly its conditional distribution, to exactly compute a value function for the Shapely values. We also present numerical examples, which imply that the Shapley values are advantageous for explaining detected anomalies than raw reconstruction errors of each feature.

21. Riku Sasaki, Naoya Takeishi, Takehisa Yairi, Koichi Hori

    Neural Gray-Box Identification of Nonlinear Partial Differential Equations

    In Proceedings of the 16th Pacific Rim International Conference on Artificial Intelligence, pp. 309–321, 2019

    Lecture Notes in Computer Science, vol. 11671

    Abs

    Many branches of the modern computational science and engineering are based on numerical simulations, for which we must prepare appropriate equations that well reflect the behavior of real-world phenomena and numerically solve them. For these purposes, we may utilize the data-driven identification and simulation technique of nonlinear partial differential equations (NPDEs) using deep neural networks (DNNs). A potential issue of the DNN-based identification and simulation in practice is the high variance due to the complexity of DNNs. To alleviate it, we propose a simple yet efficient way to incorporate prior knowledge of phenomena. Specifically, we can often anticipate what kinds of terms are present in a part of an appropriate NPDE, which should be utilized as prior knowledge for identifying the remaining part of the NPDE. To this end, we design DNN’s inputs and the loss function for identification according to the prior knowledge. We present the results of the experiments conducted using three different types of NPDEs: the Korteweg– de Vries equation, the Navier– Stokes equation, and the Kuramoto– Sivashinsky equation. The experimental results show the effectiveness of the proposed method, i.e., utilizing known terms of an NPDE.

22. Ryo Sakagami, Naoya Takeishi, Takehisa Yairi, Koichi Hori

    Visualization Methods for Spacecraft Telemetry Data Using Change-Point Detection and Clustering

    Transactions of the Japan Society for Aeronautical and Space Sciences, Aerospace Technology Japan, 17(2):244–252, 2019

    Abs

    For secure operation of spacecraft, automatic or assistive health monitoring systems utilizing telemetry data are important. However, it is difficult to utilize them comprehensively because they consist of myriad heterogeneous variables. Although various monitoring systems focusing on only a few variables or homogeneous variables have been suggested, a definitive method to deal with the relationship among multiple heterogeneous variables has not yet. This paper proposes a new visualization framework that aims to show the correlation rules underlying multiple variables of spacecraft telemetry data. The proposed framework consists of a change-point detection algorithm based on subspace identification, clustering methods using dimensionality reduction, and a visualization method using heatmaps. In experiments conducted with real telemetry data obtained from JAXA spacecraft SDS-4, the proposed framework demonstrated effective visualizations that reflected the correlations among variables expected from mechanical characteristics of the satellite. Despite differences in scales and/or units, this framework succeeded in visualizing dynamic correlations not only among continuous variables but also among continuous and discrete variables. This framework can be utilized as an initial stage of anomaly detection focusing on the relationship among multiple variables, as well as a method to perceive the overall state of the spacecraft at a glance.

23. Naoya Takeishi, Takehisa Yairi, Yoshinobu Kawahara

    Factorially Switching Dynamic Mode Decomposition for Koopman Analysis of Time-Variant Systems

    In Proceedings of the 2018 IEEE Conference on Decision and Control, pp. 6402–6408, 2018

    Abs PDF Slides

    The modal decomposition based on the spectra of the Koopman operator has gained much attention in various areas such as data science and optimal control, and dynamic mode decomposition (DMD) has been known as a data-driven method for this purpose. However, there is a fundamental limitation in DMD and most of its variants; these methods are based on the premise that the target system is time-invariant at least within the data at hand. In this work, we aim to compute DMD on time-varying dynamical systems. To this end, we propose a probabilistic model that has factorially switching dynamic modes. In the proposed model, which is based on probabilistic DMD, observation at each time is expressed using a subset of dynamic modes, and the activation of the dynamic modes varies over time. We present an approximate inference method using expectation propagation and demonstrate the modeling capability of the proposed method with numerical examples of temporally-local events and transient phenomena.

24. Rem Hida, Naoya Takeishi, Takehisa Yairi, Koichi Hori

    Dynamic and Static Topic Model for Analyzing Time-Series Document Collections

    In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 516–520, 2018

    Abs

    For extracting meaningful topics from texts, their structures should be considered properly. In this paper, we aim to analyze structured time-series documents such as a collection of news articles and a series of scientific papers, wherein topics evolve along time depending on multiple topics in the past and are also related to each other at each time. To this end, we propose a dynamic and static topic model, which simultaneously considers the dynamic structures of the temporal topic evolution and the static structures of the topic hierarchy at each time. We show the results of experiments on collections of scientific papers, in which the proposed method outperformed conventional models. Moreover, we show an example of extracted topic structures, which we found helpful for analyzing research activities.

25. Naoya Takeishi, Yoshinobu Kawahara, Takehisa Yairi

    Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition

    In Advances in Neural Information Processing Systems 30, pp. 1130–1140, 2017

    Abs Code Poster

    Spectral decomposition of the Koopman operator is attracting attention as a tool for the analysis of nonlinear dynamical systems. Dynamic mode decomposition is a popular numerical algorithm for Koopman spectral analysis; however, we often need to prepare nonlinear observables manually according to the underlying dynamics, which is not always possible since we may not have any a priori knowledge about them. In this paper, we propose a fully data-driven method for Koopman spectral analysis based on the principle of learning Koopman invariant subspaces from observed data. To this end, we propose minimization of the residual sum of squares of linear least-squares regression to estimate a set of functions that transforms data into a form in which the linear regression fits well. We introduce an implementation with neural networks and evaluate performance empirically using nonlinear dynamical systems and applications.

26. Naoya Takeishi, Takehisa Yairi

    Visual Monocular Localization, Mapping, and Motion Estimation of a Rotating Small Celestial Body

    Journal of Robotics and Mechatronics, 29(5):856–863, 2017

    Abs Code

    In the exploration of a small celestial body, it is important to estimate the position and attitude of the spacecraft, as well as the geometric properties of the target celestial body. In this paper, we propose a method to concurrently estimate these quantities in a highly automatic manner when measurements from an attitude sensor, inertial sensors, and a monocular camera are given. The proposed method is based on the incremental optimization technique, which works with models for sensor fusion, and a tailored initialization scheme developed to compensate for the absence of range sensors. Moreover, we discuss the challenges in developing a fully automatic navigation framework.

27. Naoya Takeishi, Yoshinobu Kawahara, Takehisa Yairi

    Subspace Dynamic Mode Decomposition for Stochastic Koopman Analysis

    Physical Review E, 96(3):033310, 2017

    The published version has a typo in the main algorithm, which is fixed in the arXiv version.

    Abs arXiv ver. Code

    The analysis of nonlinear dynamical systems based on the Koopman operator is attracting attention in various applications. Dynamic mode decomposition (DMD) is a data-driven algorithm for Koopman spectral analysis, and several variants with a wide range of applications have been proposed. However, popular implementations of DMD suffer from observation noise on random dynamical systems and generate inaccurate estimation of the spectra of the stochastic Koopman operator. In this paper, we propose subspace DMD as an algorithm for the Koopman analysis of random dynamical systems with observation noise. Subspace DMD first computes the orthogonal projection of future snapshots to the space of past snapshots and then estimates the spectra of a linear model, and its output converges to the spectra of the stochastic Koopman operator under standard assumptions. We investigate the empirical performance of subspace DMD with several dynamical systems and show its utility for the Koopman analysis of random dynamical systems.

28. Naoya Takeishi, Yoshinobu Kawahara, Takehisa Yairi

    Sparse Nonnegative Dynamic Mode Decomposition

    In Proceedings of the 2017 IEEE International Conference on Image Processing, pp. 2682–2686, 2017

    Abs

    Dynamic mode decomposition (DMD) is a method to extract coherent modes from nonlinear dynamical systems. In this paper, we propose an extension of DMD, sparse nonnegative DMD, which generates a nonlinear and sparse modal representation of dynamics. In particular, this makes DMD more suitable for video processing. We reformulate DMD as a block-multiconvex optimization problem to impose constraints and regularizations directly on the structures of the estimated dynamic modes. We introduce the results of experiments with synthetic data and a surveillance video dataset and show that sparse nonnegative DMD can extract part-based dynamic modes from video streams.

29. Naoya Takeishi, Yoshinobu Kawahara, Yasuo Tabei, Takehisa Yairi

    Bayesian Dynamic Mode Decomposition

    In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 2814–2821, 2017

    Abs Code Slides

    Dynamic mode decomposition (DMD) is a data-driven method for calculating a modal representation of a nonlinear dynamical system, and it has been utilized in various fields of science and engineering. In this paper, we propose Bayesian DMD, which provides a principled way to transfer the advantages of the Bayesian formulation into DMD. To this end, we first develop a probabilistic model corresponding to DMD, and then, provide the Gibbs sampler for the posterior inference in Bayesian DMD. Moreover, as a specific example, we discuss the case of using a sparsity-promoting prior for an automatic determination of the number of dynamic modes. We investigate the empirical performance of Bayesian DMD using synthetic and real-world datasets.

30. Takehisa Yairi, Naoya Takeishi, Tetsuo Oda, Yuta Nakajima, Naoki Nishimura, Noboru Takata

    A Data-Driven Health Monitoring Method for Satellite Housekeeping Data Based on Probabilistic Clustering and Dimensionality Reduction

    IEEE Transactions on Aerospace and Electronic Systems, 53(3):1384–1401, 2017

    Abs

    In the operation of artificial satellites, it is very important to monitor the health status of the systems and detect any symptoms of anomalies in the housekeeping data as soon as possible. Recently, the data-driven approach to the system monitoring problem, in which statistical machine learning techniques are applied to the large amount of measurement data collected in the past, has attracted considerable attention. In this paper, we propose a new data-driven health monitoring and anomaly detection method for artificial satellites based on probabilistic dimensionality reduction and clustering, taking into consideration the miscellaneous characteristics of the spacecraft housekeeping data. We applied our method to the telemetry data of the small demonstration satellite 4 (SDS-4) of the Japan Aerospace Exploration Agency (JAXA) and evaluated its effectiveness. The results show that the proposed system provides satellite operators with valuable information for understanding the health status of the system and inferring the causes of anomalies.

31. Naoya Takeishi, Takehisa Yairi, Naoki Nishimura, Yuta Nakajima, Noboru Takata

    Dynamic Grouped Mixture Models for Intermittent Multivariate Sensor Data

    In Proceedings of the 20th Pacific Asia Conference on Knowledge Discovery and Data Mining, pp. 221–232, 2016

    Lecture Notes in Computer Science, vol. 9652

    Abs PDF Slides

    For secure and efficient operation of engineering systems, it is of great importance to watch daily logs generated by them, which mainly consist of multivariate time-series obtained with many sensors. This work focuses on challenges in practical analyses of those sensor data: temporal unevenness and sparseness. To handle the unevenly and sparsely spaced multivariate time-series, this work presents a novel method, which roughly models temporal information that still remains in the data. The proposed model is a mixture model with dynamic hierarchical structure that considers dependency between temporally close batches of observations, instead of every single observation.

32. Naoya Takeishi, Takehisa Yairi, Yuichi Tsuda, Fuyuto Terui, Naoko Ogawa, Yuya Mimasu

    Simultaneous Estimation of Shape and Motion of an Asteroid for Automatic Navigation

    In Proceedings of the 2015 IEEE International Conference on Robotics and Automation, pp. 2861–2866, 2015

    Abs PDF Poster

    In an asteroid exploration and sample return mission, accurate estimation of the shape and motion of the target asteroid is essential for selecting a touchdown site and navigating a spacecraft during touchdown operation. In this work, we present an automatic estimation method for the shape and motion of an asteroid, which is planned to be tested in future exploration missions including Japanese Hayabusa-2 [1]. Our task is to estimate the shape and rotation axis of the asteroid, as well as positions of the spacecraft from optical images. The proposed method is based on the expectation conditional-maximization (ECM) framework that consists of an auxiliary particle filter and nonlinear optimization techniques. One of our technical contributions is the estimation of the direction of rotation axis of the asteroid from monocular camera images, which are taken by the moving spacecraft. We conducted two experiments with synthetic data and an asteroid mock-up to show the validity of the proposed method and to present the numerical accuracy.

33. Naoya Takeishi, Akira Tanimoto, Takehisa Yairi, Yuichi Tsuda, Fuyuto Terui, Naoko Ogawa, Yuya Mimasu

    Evaluation of Interest-Region Detectors and Descriptors for Automatic Landmark Tracking on Asteroids

    Transactions of the Japan Society for Aeronautical and Space Sciences, 58(1):45–53, 2015

    Abs

    The asteroid explorer Hayabusa-2, which is scheduled to be launched in 2014, is going to perform a global mapping mission after it arrives at the target asteroid. Although most of the global mapping sequence will be the same as that of its predecessor Hayabusa, several automation technologies are planned to be tested to reduce the workload of the operators. In particular, the structure from motion (SFM) and simultaneous localization and mapping (SLAM) techniques are expected to significantly contribute to the automation of asteroid shape estimation and visual spacecraft navigation. These frameworks require automatic landmark tracking on the asteroid surface, but no previous work has discussed the method that should be used to track images of the asteroid taken in space, where the absence of scattering light causes dramatic changes in appearance. In this study, we evaluated the performances of SIFT, SURF, BRISK, ORB, Harris-Affine, Hessian-Affine and MSER for images of the asteroid. We found that SIFT is acceptable for use, while SURF, BRISK and ORB can be used with careful parameter tuning. The affine-invariant detectors might contribute to more accurate tracking, but using them is more challenging owing to an extra normalizing process.

34. Naoya Takeishi, Takehisa Yairi

    Anomaly Detection from Multivariate Time-Series with Sparse Representation

    In Proceedings of the 2014 IEEE International Conference on Systems, Man, and Cybernetics, pp. 2651–2656, 2014

    Abs PDF

    Anomaly detection from sensor data is an important data mining application for efficient and secure operation of complicated systems. In this study, we propose a novel anomaly detection method for multivariate time-series to capture relationships of variables and time-domain correlations simultaneously, without assuming any generative models of signals. The supposed framework in this study is a semi-supervised anomaly detection where we seek unusual parts of test data compared with reference data. The proposed method is based on feature extraction with sparse representation and relationship learning with dimensionality reduction. Our idea comes from the similarity between a sparse feature matrix extracted from multivariate time-series and a term-document matrix. We conducted experiments with synthetic and simulated data, and confirmed that the proposed method successfully detected anomalies in multivariate time-series signals. Especially, it demonstrated superior performance with anomalies in which only relationships of time-series patterns are changed from reference data (multivariate anomalies).

35. Akira Tanimoto, Naoya Takeishi, Takehisa Yairi, Yuichi Tsuda, Fuyuto Terui, Naoko Ogawa, Yuya Mimasu

    Fast Estimation of Asteroid Shape and Motion for Spacecraft Navigation

    In Proceedings of the 2013 IEEE International Conference on Robotics and Biomimetics, pp. 1550–1555, 2013

    Abs

    In this paper, we consider fast simultaneous estimation problem of the geometric shape of the asteroid and the relative motion of the spacecraft. In asteroid exploration missions, the information of asteroid shape and motion is needed to find suitable landing sites and navigate the spacecraft safely. In the previous HAYABUSA mission, large part of the estimation was performed manually by ground operators. We propose an efficient automatic estimation method using the image feature matching and matrix decomposition based fast 3D reconstruction techniques. Preliminary experiment results are also shown.

Preprints

1. João A. Cândido Ramos, Lionel Blondé, Naoya Takeishi, Alexandros Kalousis

   Sample-Efficient On-Policy Imitation Learning from Observations

   arXiv:2306.09805

   Abs

   Imitation learning from demonstrations (ILD) aims to alleviate numerous shortcomings of reinforcement learning through the use of demonstrations. However, in most real-world applications, expert action guidance is absent, making the use of ILD impossible. Instead, we consider imitation learning from observations (ILO), where no expert actions are provided, making it a significantly more challenging problem to address. Existing methods often employ on-policy learning, which is known to be sample-costly. This paper presents SEILO, a novel sample-efficient on-policy algorithm for ILO, that combines standard adversarial imitation learning with inverse dynamics modeling. This approach enables the agent to receive feedback from both the adversarial procedure and a behavior cloning loss. We empirically demonstrate that our proposed algorithm requires fewer interactions with the environment to achieve expert performance compared to other state-of-the-art on-policy ILO and ILD methods.

2. Chun Fui Liew, Danielle DeLatte, Naoya Takeishi, Takehisa Yairi

   Recent Developments in Aerial Robotics: A Survey and Prototypes Overview

   arXiv:1711.10085

   Abs

   In recent years, research and development in aerial robotics (i.e., unmanned aerial vehicles, UAVs) has been growing at an unprecedented speed, and there is a need to summarize the background, latest developments, and trends of UAV research. Along with a general overview on the definition, types, categories, and topics of UAV, this work describes a systematic way to identify 1,318 high-quality UAV papers from more than thirty thousand that have been appeared in the top journals and conferences. On top of that, we provide a bird’s-eye view of UAV research since 2001 by summarizing various statistical information, such as the year, type, and topic distribution of the UAV papers. We make our survey list public and believe that the list can not only help researchers identify, study, and compare their work, but is also useful for understanding research trends in the field. From our survey results, we find there are many types of UAV, and to the best of our knowledge, no literature has attempted to summarize all types in one place. With our survey list, we explain the types within our survey and outline the recent progress of each. We believe this summary can enhance readers’ understanding on the UAVs and inspire researchers to propose new methods and new applications.

Non-refereed presentations

1. Naoya Takeishi

   Toward Bayesian Deep Grey-box Modeling

   In International Conference on Scientific Computing and Machine Learning, Kyoto & Online, 2024

   Abs

   Combining scientific models and deep neural networks (deep grey-box or hybrid modeling) is expected to be a promising strategy for building robust, partly interpretable, and data-adaptive models. This paper presents a preliminary study to develop a framework for learning deep grey-box models with uncertainty quantification via Bayesian inference of both scientific models and neural nets.

2. Naoya Takeishi, Alexandros Kalousis

   Variational Autoencoder with Differentiable Physics Engine for Human Gait Analysis and Synthesis

   In Deep Generative Models and Downstream Applications Workshop, Online, 2021

   Abs Poster

   We address the task of learning generative models of human gait. As gait motion always follows the physical laws, a generative model should also produce outputs that comply with the physical laws, particularly rigid body dynamics with contact and friction. We propose a deep generative model combined with a differentiable physics engine, which outputs physically plausible signals by construction. The proposed model is also equipped with a policy network conditioned on each sample. We show an example of the application of such a model to style transfer of gait.

3. Naoya Takeishi

   Towards Intelligent Asteroid Exploration: Visual SLAM in Space

   In The 23rd SANKEN International Symposium, Awaji, Japan, 2020

   (invited talk)
4. Keisuke Fujii, Naoya Takeishi, Yoshinobu Kawahara

   Interpretable Classification of Complex Collective Motions Using Graph Dynamic Mode Decomposition

   In Workshop on Machine Learning for Trajectory, Activity, and Behavior, Nagoya, Japan, 2019
5. Naoya Takeishi, Kosuke Akimoto

   Knowledge-Based Distant Regularization in Learning Probabilistic Models

   In The 8th International Workshop on Statistical Relational AI, Stockholm, Sweden, 2018

   Abs arXiv ver.

   Exploiting the appropriate inductive bias based on the knowledge of data is essential for achieving good performance in statistical machine learning. In practice, however, the domain knowledge of interest often provides information on the relationship of data attributes only distantly, which hinders direct utilization of such domain knowledge in popular regularization methods. In this paper, we propose the knowledge-based distant regularization framework, in which we utilize the distant information encoded in a knowledge graph for regularization of probabilistic model estimation. In particular, we propose to impose prior distributions on model parameters specified by knowledge graph embeddings. As an instance of the proposed framework, we present the factor analysis model with the knowledge-based distant regularization. We show the results of preliminary experiments on the improvement of the generalization capability of such model.

6. Taichi Kitamura, Naoya Takeishi, Takehisa Yairi, Koichi Hori

   Abnormal Sound Detection for Rotary Parts in Noisy Environment by One-Class SVM and Non-Negative Matrix Factorization

   In Asia Pacific Conference of the Prognostics and Health Management Society, Jeju, Korea, 2017

   Abs

   In this paper, we introduce a data-driven method for detecting abnormal sound from rotary machines, which is due to small scratches on the surface of bearings, in a manufacturing plant. Since it is difficult to obtain a sufficient amount of anomalous data beforehand, we assume only normal data are available for training a model. The challenge in our situation is the presence of high-level ambient noise, which makes detecting small anomalous sound very hard. In the proposed method, feature vectors are extracted by applying short-time Fourier transform, and one-class SVM is trained on normal data and used to discriminate normal and anomalous data. In addition, ambient noise is removed using nonnegative matrix factorization before extracting features to overcome the problem of noise superimposition and to improve the discrimination precision. We show the results of an experiment using actual sound data obtained from rotary machines with ambient noise.

7. Riku Sasaki, Naoya Takeishi, Takehisa Yairi, Koichi Hori, Kazunari Ide, Hiroyoshi Kubo

   A Health Monitoring Method for Wind Power Generators With Hidden Markov and Probabilistic Principal Components Analysis Models

   In Asia Pacific Conference of the Prognostics and Health Management Society, Jeju, Korea, 2017

   Abs

   In this work, we propose a data-driven health monitoring method for wind power generators, which learns an empirical model from the time-series sensor data and detects irregularities or faults in the turbines and blades. Our main objective is to predict any symptoms of faults as early as possible before the generators fall into malfunction. The data obtained from the wind power generators are strongly correlated multidimensional time-series with multiple states. In this study, we take such features into account and develop the probabilistic model for them, namely, hidden Markov and probabilistic principal component analysis. Once the model is learned with the data that contain no faulty events, it can be used to detect faults in new data by comparing the original sensor values and reconstructed ones. In this research, we apply this method to synthetic data and real-world wind turbine data and show the results of experiments to confirm the availability of the proposed method.

8. Naoya Takeishi, Takehisa Yairi

   Dynamic Visual Simultaneous Localization and Mapping for Asteroid Exploration

   In The 13th International Symposium on Artificial Intelligence, Robotics and Automation in Space, Beijing, P.R.China, 2016

   Abs PDF

   In asteroid explorations, it is indispensable to estimate the shape of the target asteroid, which can be solved in a manner similar to one of simultaneous localization and mapping (SLAM). This work proposes a SLAM framework dedicated to the asteroid exploration, which considers both rigid body dynamics of the asteroid and motion of the spacecraft, estimating asteroid’s shape, centroid, rotational axis, angular velocity and phase, as well as spacecraft’s attitude and position relative to the asteroid. Experimental results with artificially generated data show that the proposed method can accurately estimate these quantities using images of a monocular camera and measurements of attitude and inertia sensors.

9. Kosuke Akimoto, Naoya Takeishi, Takehisa Yairi, Koichi Hori, Naoki Nishimura, Noboru Takata

   Tree-Based Nonparametric Prediction of Normal Sensor Measurement Range Using Temporal Information

   In The 13th International Symposium on Artificial Intelligence, Robotics and Automation in Space, Beijing, P.R.China, 2016

   Abs PDF

   Currently, limit-checking on telemetry sensor data of a spacecraft is widely used to detect its faults and anomalous behavior. Since classical limit-checking usually considers only a priori fixed pair of upper and lower bounds for each sensor variable, it sometimes fails to detect phenomena that are anomalous only in certain operating modes. To handle this problem, we present a method to predict normal ranges of sensor measurements adaptively based on status variables of telemetry data and temporal information. In the proposed method, a regression tree is learned using status variables, and each data point is labeled according to the terminal node of the tree it reached. Three new temporal features are generated from the sequence of the label, and a quantile regression forest is learned using both status variables and the generated features. Normal ranges are calculated from approximate distribution predicted using the quantile regression forest. We apply this method to actual telemetry data with simulated anomalies, and confirmed that the proposed method can detect temporal anomalies with a lower false alarm rate than the previous method.

10. Naoya Takeishi

    Automatic Landmark Recognition for Asteroid by Image Features

    In The 29th International Symposium on Space Technology and Science, Nagoya, Japan, 2013

    Abs

    In exploration missions of asteroids, we have to recognize visual signatures (landmarks) on an asteroid’s surface to estimate the geometric shape of the asteroid and the relative position of the spacecraft. This recognition process was conducted manually in the mission of MUSES-C (Hayabusa), and it required a tremendous amount of time and efforts. The purpose of our work is to develop an automatic landmark recognition system. We used scale invariant feature transform, random sampling consensus, graph model of keypoint-match and relative position of keypoint-pair on rigid body. The system succeeded in recognizing landmarks on the asteroid’s surface automatically within experimental images, while there was a little erroneous recognition. The error rate of recognition was about 2 percent. The landmarks detected with this system can be used to reconstruct the solid shape of the asteroid through techniques called structure from motion or simultaneous localization and mapping. Moreover, since some of the landmarks were easy for human operators to recognize, this system might be linked to the previous manual system.