基于去中心化索引的IPFS数据获取方法研究

doi:10.3778/j.issn.1002-8331.2101-0109

摘要/Abstract

摘要： 星际文件系统（interplanetary file system，IPFS）实现了去中心化存储，可以满足日益增长的数据存储需求，然而IPFS仅提供一种精确的数据获取方式，在缺乏数据唯一标识时无法查找数据。现有的IPFS数据获取方法削弱了IPFS的去中心化，仅实现了关键字搜索，对长查询语句进行关键词搜索加重了网络负担。为此，提出了一种去中心化混合索引的IPFS数据获取方法——IPFS-DDAM。提取数据的关键词及中心语句以建立关键词索引及句子索引；使用分布式哈希表（distributed Hash table，DHT）存储索引，数据内容相似的句子索引存储相邻，实现了句子索引的邻近范围搜索及关键词索引的精确搜索；改进缓存存储机制，减少冗余存储；通过在公开数据集上的仿真实验证明了该方法的有效性，并且减少了网络负担。

关键词: 星际文件系统, 分布式哈希表, 去中心化, 关键词索引, 句子索引, 缓存

Abstract: The interplanetary file system（IPFS） is a decentralized file system that can meet the growing demand of data storage. However, only a precise way to retrieve data is provided by IPFS, so the data cannot be found in the absence of a unique data identifier. The current data acquisition method weakens the decentralization of IPFS, and only realizes keyword search. In order to solve them effectively, a data acquisition method based on a decentralized mixtureindex mechanism is proposed. Keywords and central sentences are extracted from data and used to build keyword indexes and sentence index respectively. All indexes are stored at distributed Hash table（DHT）. Keyword indexes realize precise search based on the Hash value of keywords. And the sentence indexes with similar content are stored neighboring, consequently, search operations can be executed in the neighborhood of nodes that storing the sentence index. The redundant data is reduced by improving the storage mechanism of cache. The experiments on public data sets prove the effectiveness of the proposed method.

Key words: interplanetary file system（IPFS）, distributed Hash table（DHT）, decentralized, keyword index, sentence index, cache

石秋娥, 周喜, 王轶. 基于去中心化索引的IPFS数据获取方法研究[J]. 计算机工程与应用, 2022, 58(3): 83-90.

SHI Qiu’e, ZHOU Xi, WANG Yi. Research of IPFS Data Acquisition Method Based on Decentralized Index[J]. Computer Engineering and Applications, 2022, 58(3): 83-90.

参考文献

[1] BENET J.IPFS-content addressed，versioned，p2p file system[J].arXiv：1407.3561，2014.
[2] JANSEN B J，SPINK A，SARACEVIC T.Real life，real users，and real needs：a study and analysis of user queries on the web[J].Information Processing & Management，2000，36（2）：207-227.
[3] MIKOLOV T，CHEN K，CORRADO G，et al.Efficient estimation of word representations in vector space[J].arXiv：1301.3781，2013.
[4] BRUIN M D.Search engine for the interplanetary filesystem[EB/OL].[2020-12-21].https：//github.com/ipfs-search/ipfs-search.
[5] KHUDHUR N，FUJITA S.Siva-the IPFS search engine[C]//Proceedings of the 2019 Seventh International Symposium on Computing and Networking（CANDAR），Nagasaki，Japan，Nov 26-29，2019.Los Alamitos：IEEE Computer SOC，2019：150-156.
[6] KLEMS M，EBERHARDT J，TAI S，et al.Trustless intermediation in blockchain-based decentralized service marketplaces[C]//International Conference on Service Oriented Computing，Malaga，Spain，Nov 13-16，2017.Switzerland：Springer，2017：731-739.
[7] ZHU L，XIAO C，GONG X.Keyword search in decentralized storage systems[J].Electronics，2020，9（12）：2041.
[8] TANG C，XU Z，DWARKADAS S.Peer-to-peer information retrieval using self-organizing semantic overlay networks[C]//Proceedings of the 2003 Conference on Applications，Technologies，Architectures，and Protocols for Computer Communications，Karlsruhe，Aug 25-29，2003.New York：ASSOC Computing Machinery，2003：175-186.
[9] 程学旗，吕建明，周昭涛.基于对等网络的全文信息检索[J].计算机研究与发展，2004，41（12）：2148-2155.
CHENG X Q，Lü J M，ZHOU Z T.P2P full text information retrieval based on centroid method[J].Journal of Computer Research and Development，2004，41（12）：2148-2155.
[10] REYNOLDS P，VAHDAT A.Efficient peer-to-peer keyword searching[C]//Proceedings of the ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing，Rio de Janeiro，Jun 15-20，2003.Berlin，Springer-verlag，2003：21-40.
[11] HASSANZADEH-NAZARABADI Y，KUP?U A，OZKASAP O.Decentralized and locality aware replication method for DHT-based P2P storage systems[J].Future Generation Computer Systems，2018，84（7）：32-46.
[12] JOUNG Y J，YANG L W，FANG C T.Keyword search in DHT-based peer-to-peer networks[J].IEEE Journal on Selected Areas in Communications，2007，25（1）：46-61.
[13] LI D，CAO J，LU X，et al.Efficient range query processing in peer-to-peer systems[J].IEEE Transactions on Knowledge & Data Engineering，2009，21（1）：78-91.
[14] TANG Y，ZHOU S，XU J.LIGHT：a query-efficient yet low-maintenance indexing scheme over DHTs[J].IEEE Transactions on Knowledge and Data Engineering，2010，22（1）：59-75.
[15] HIDALGO N，ARANTES L，SENS P，et al.ECHO：efficient complex query over DHT overlays[J].Journal of Parallel and Distributed Computing，2016，88（2）：31-45.
[16] NGOM B，MAKPANGOU M.Summary prefix tree：an over DHT indexing data structure for efficient superset search[C]//Proceedings of the 2017 IEEE 16th International Symposium on Network Computing and Applications（NCA），Cambridge，Oct 30-Nov 01，2017.New York：IEEE，2017：323-327.
[17] 董祥千，郭兵，沈艳，等.一种高效安全的去中心化数据共享模型[J].计算机学报，2018，41（5）：1021-1036.
DONG X Q，GUO B，SHEN Y，et al.An efficient and secure decentralizing data sharing model[J].Chinese Journal of Computers，2018，41（5）：1021-1036.
[18] 聂梦飞，庞晓琼，陈文俊，等.基于以太坊区块链的公平可搜索加密方案[J].计算机工程与应用，2020，56（4）：69-75.
NIE M F，PANG X Q，CHEN W J，et al.Fair searchable encryption scheme based on ethereum blockchain[J].Computer Engineering and Applications，2020，56（4）：69-75.
[19] RATNASAMY S，FRANCIS P，HANDLEY M，et al.A scalable content-addressable network[C]//Proceedings of the 2001 Conference on Applications，Technologies，Architectures，and Protocols for Computer Communications，San Diego，Aug 27-31，2001.New York：ASSOC Computing Machinery，2001：161-172.
[20] STOICA I，MORRIS R，KARGER D，et al.Chord：a scalable peer-to-peer lookup service for internet applications[J].ACM SIGCOMM Computer Communication Review，2001，31（4）：149-160.
[21] MAYMOUNKOV P，MAZIERES D.Kademlia：a peer-to-peer information system based on the XOR metric[C]//Proceedings of the International Workshop on Peer-to-Peer Systems，Cambridge，Mar 07-08，2002.Berlin：Springer-Verlag，2002：53-65.
[22] 王昱华，江林，胡志刚，等.基于DHT的P2P系统负载均衡算法[J].计算机工程与应用，2015，51（23）：100-105.
WANG Y H，JIANG L，HU Z G，et al.Load balancing algorithm for DHT-based P2Psystem[J].Computer Engineering and Applications，2015，51（23）：100-105.
[23] SALTON G，BUCKLEY C.Term-weighting approaches in automatic text retrieval[J].Information Processing & Management，1988，24（5）：513-523.
[24] BRODER A Z.On the resemblance and containment of documents[C]//Proceedings Compression and Complexity of SEQUENCES 1997，Salerno，Jun 11-13，1997.Los Alamitos：IEEE Computer SOC，1998：21-29.
[25] Yahoo!Webscope dataset L18 anonymized yahoo! search logs with relevance judgments，version 1.0[EB/OL].[2020-12-21].http：//labs.yahoo.com/Academic_Relations.
[26] 唐善成，白云悦，马付玉.中文语义相似度训练集[EB/OL].[2020-12-21].https：//github.com/IAdmireu/ChineseSTS.
TANG S C，BAI Y Y，MA F Y.Chinese semantic text similarity trainning dataset[EB/OL].[2020-12-21].https：//github.com/IAdmireu/ChineseSTS.
[27] ABELLO J，PARDALOS P M，RESENDE M G.Handbook of massive data sets[EB/OL].[2020-12-21].https：//link.springer.com/book/10.1007%2F978-1-4615-0005-6.