计算机工程与应用 ›› 2011, Vol. 47 ›› Issue (16): 80-82.
• 网络、通信、安全 • 上一篇 下一篇
何绍荣，鲜乾坤
收稿日期:
修回日期:
出版日期:
发布日期:
HE Shaorong，XIAN Qiankun
Received:
Revised:
Online:
Published:
摘要： 针对日益突出的网页访问问题，设计了一种新型分布式Web Spider。该分布式Web Spider采用中央控制节点来协调各个web spider的行为，利用宽度优先搜索来获得高质量的网页，通过对DNS缓存来提高访问web server的速度，以增加并行线程数量的方式来增加网页下载速度，并能动态地加入web spider节点和子中央控制节点，具有很强的灵活性和扩张能力。实验结果表明该分布式Web Spider作为搜索引擎的前端能够快速有效地下载网页，具有较好的性能。
关键词: 中央控制节点, 宽度优先搜索, 线程, 搜索引擎
Abstract: For the increasingly prominent web access problems，a New Distributed Web Spider（NDWS） is proposed.NDWS uses central control node to coordinate actions of all web spiders，employs breadth-first search to obtain high-quality web pages，caches DNS to improve speed of access to web server，increases number of concurrent threads to increase download speed of web pages.NDWS also can dynamically add web spider node and sub-central-control-node so that NDWS has strong flexibility and expansion capability.Experimental results show that as a front-end of search engine，NDWS can quickly and efficiently download web pages，and has better performance.
Key words: central control node, breadth-first search, thread, search engine
何绍荣，鲜乾坤. 新型分布式Web Spider的设计[J]. 计算机工程与应用, 2011, 47(16): 80-82.
HE Shaorong，XIAN Qiankun. Design of new distributed web spider[J]. Computer Engineering and Applications, 2011, 47(16): 80-82.
0 / 推荐
导出引用管理器 EndNote|Ris|BibTeX
链接本文: http://cea.ceaj.org/CN/
http://cea.ceaj.org/CN/Y2011/V47/I16/80
