Abstract: The Ontology-based image feature database is built through feature extraction（including color，shape，and texture feature of image，etc.） and mapping feature to ontology.Taking the Chinese affective picture system as the training sample，the association between the image feature and the affective semantics of image is learned through machine learning，then these association rules are represented by SWRL（Semantic Web Rule Language），thus the image affective semantics mapping rule database can be built.The SWRL-based rules and ontology-based image features are used altogether in the affective semantic reasoning engine to deduce the affection of image，in this way the affective semantics of image can be recognized.