关键词: 信息熵, 模糊综合评判, 相似数据, 属性约简, 粗糙集

Abstract: Aiming at the problem of large redundancy of data in big data, an approximately duplicated data detecting method based on Shannon entropy and fuzzy integrated evaluation is proposed. Firstly, attributes in data set are reduced based on Shannon entropy, and then fuzzy integrated evaluation method is adopted to get the weights of the attributes after their reduction, lastly the approximately data is detected according to the reduced attributes and their weights. Theoretical analysis and experimental comparison show that this method has high detection accuracy and efficiency in approximately data detecting of structured big data set.

Key words: information entropy, fuzzy integrated evaluation, approximately data, attribute reduction, rough set