Due to the cost-down of video capture devices, video surveillance systems are widely applied to monitor people, objects, or environment. The most common application is people's monitoring, such as detecting illegal entrants in restricted areas and detecting elderly people's fall in house. Currently, news frequently reports that children have accidents occurred in house. Although, the Children's Welfare Law prohibits parents from leaving children alone who are under the age of six and required to take care all the day, the accidents are still occurred, occasionally, due to parents' or babysitters' carelessness. This study will develop an automatic and intelligent in-house video surveillance system to detect, track and analyze children's motions. This system uses digital image processing techniques to determine if the children have dangerous behavior, such as climbing balcony wall, playing with fire, and opening door to go out. When the dangerous behavior is detected, the system will automatically alert parents or babysitter.