Humanoid social robots that interact with people need to be capable of interpreting the social behavior of their interaction partners in order to respond in a socially appropriate way. In this paper, we present a social behavior interpretation system that enables a humanoid robot to recognize human social behavior by analyzing communicative signals. The system receives the constructed RGB-D scene from a Kinect sensor, extracts information about body gesture and head pose from the scene using Microsoft Kinect SDK, and recognizes eight human social behaviors using a Hidden Markov Model (HMM). We trained the eight-state HMM with a corpus of 35 recorded human-human interaction scenes. The evaluation of the system shows a weighted average recognition rate of 81% for all states. © 2014 IEEE.