Abstract: The combination of Handlers in virtual machine can protect key codes in the program, and these Handlers are the main target for software reverse analysts to attack. Aiming at the reduction method for dynamic extraction and static analysis of Handlers, virtual machine protection method based on Handler obfuscation is proposed. Firstly, various equivalent instruction rules are used to generate different equivalence Handlers, and then all Handlers are divided and disordered by random scrambling algorithm, and they are restructured by constructing jump table, finally random address array is used to hide the data of Handler scheduling address table and execution jump table. Experiments and analysis show that the generation, segmentation and disorder of diverse Handlers increase the difficulty of dynamic extraction and analysis, the Handler address table and a jump table hidden enhances the difficulty of static reverse analysis.

Key words: virtual machine protection, equivalent instruction replacement, segmentation disorder, diversity, table hidden

摘要： 按照一定顺序执行虚拟指令处理函数（Handler）可完成程序关键代码的保护，其为软件逆向分析者攻击的重点对象。针对“动态提取，静态分析”的Handler攻击方法，提出一种基于Handler混淆增强的虚拟机保护方法。运用等价指令替换规则生成多种等价Handler序列，对所有Handler进行变长切分和随机乱序，通过构建跳转表对乱序序列进行重组，构建随机地址数组对Handler调度地址表和执行跳转表进行隐藏。实验和分析表明：多样化Handler生成、切分和乱序增加了动态提取和分析的难度，Handler地址表和跳转表的隐藏增加了抵御静态逆向分析的难度，从而提升了虚拟机保护强度。

关键词: 虚拟机保护, 等价指令替换, 切分乱序, 多样化, 表隐藏