Multimodal Sequence Classification of force-based instrumented hand manipulation motions using LSTM-RNN deep learning models

Abhinaba Bhattacharjee; Terry Loghmani; Sohel Anwar; Lexi Whitinger

Abstract

The advent of mobile ubiquitous computing enabled sensor informatics of human movements leveraged to model and build deep learning classifiers for cognitive AI. Expanding deep learning approaches for classifying instrumented hand manipulation tasks, especially the art of manual therapy and soft tissue manipulation, can potentially augment practitioner's performance and enhance fidelity with computer assisted guidelines. This paper introduces a dataset of 3D force profiles and manipulation motion sequences of controlled soft tissue manipulation stroke pattern applications in thoracolumbar, upper thigh and calf regions of a human subject performed by five experienced manual therapists. The multimodal 3D force, 3D accelerometer and gyro raw data were preprocessed and experimentally fed into a multilayer Long Short-Term Memory (LSTM) based Recurrent Neural Network (RNN) deep learning model to observe sequence classifications of two manipulation motion techniques (Linear "Strumming" motion and curvilinear "J-Stroke" arched motion) of manual therapy performed using a handheld, localizing Quantifiable Soft Tissue Manipulation (QSTM) medical tool. Each of these motion sequences were further labeled with corresponding best practice technique from validated video tapes and reclassified into "Correct" and "Incorrect" practice based on defined criteria. The deep learning model resulted in 90-95% classification accuracy for individual intra-therapist reduced dataset. The classification accuracy varied in between 74%-89% range, when trained with variable feature set combinations for the complete spectrum of inter-therapist dataset. Clinical Relevance-AI guided online practice classifications can be leveraged to curb practice inconsistencies, optimize training using data informed guidelines, and study progression of pain and healing towards advancing manual therapy.

Keywords: Long Short Term Memory; Multimodal Sequence Classification; Manual Therapy; Motion Recognition; Machine Learning

Links
[Full text PDF][Bibtex]