The recognition of human activities in the controlled area is an important issue for surveillance systems to keep pedestrians safe from tripping, crowding and overcrowding. Automating the surveillance systems need many procedures, which makes this process not easy, as it requires separating people from the background and determining their body parts and properties and then tracking all their different activities on the surveillance video. The main purpose of this study is to help in automating the surveillance systems by presenting a full framework to detect abandonment event in different scenarios using computer vision. To achieve this goal, this system is composed of three main stages: human detection, carried object detection and tracking with abandonment event detection. For human detection, it detects human by using a mixture of Gaussian, and aggregate channel features (ACF) detector. Moreover, it uses star skeleton with centroid and time-series graph to detect the carried objects. As for tracking, this study adapts the BuM-NLV technique with nonlinear voting. The framework was tested on a dataset that was recorded by the researchers, known as ABdetect. And it showed a precision of 86% in human detection and 25% in carried object detection. As for tracking it showed an accuracy of 85% and a 95% for abandonment event detection. This shows that this framework for detecting abandoned objects is resilient even when some of the component has lower than optimal performance.