In order to implement the real-time detection of abnormality of elder and devices in an empty nest home,multi-modal joint sensors are used to collect discrete action sequences of behavior,and the improved hierarchical hidden Markov model is adopted to Abstract these discrete action sequences captured by multi-modal joint sensors into an occupant’s high-level behavior—event,then structure representation models of occupant normality are modeled from large amounts of spatio-temporal data. These models are used as classifiers of normality to detect an occupant’s abnormal behavior.In order to express context information needed by reasoning and detection,multi-media ontology （MMO） is designed to annotate and reason about the media information in the smart monitoring system.A pessimistic emotion model （PEM） is improved to analyze multi-interleaving events of multi-active devices in the home.Experiments demonstrate that the PEM can enhance the accuracy and reliability for detecting active devices when these devices are in blind regions or are occlusive. The above approach has good performance in detecting abnormalities involving occupants and devices in a real-time way.
