面向数据可视化的自然语言接口研究综述

doi:10.3778/j.issn.1002-8331.2310-0167

摘要/Abstract

摘要： 数据可视化领域长期以来的目标是寻找直接从自然语言生成可视化的解决方案，而自然语言接口（NLI）的研究为该领域提供了新的解决办法。该接口接受自然语言形式的查询和表格数据集作为输入，并输出与之对应的可视化渲染。在作为一种辅助输入方式的同时，传统用户需将分析意图转化为一系列逻辑操作并与之进行交互（如编程指令或图形化界面操作方式等），与利用面向数据可视化的自然语言接口（DV-NLI）相结合，能够使用户专注于可视化任务，而无需担心如何操作可视化工具。近年来，随着大语言模型（LLM）GPT-3、GPT-4的兴起，将LLM与可视化相结合已成为研究热点。对现有的DV-NLI进行了全面的回顾，并进行了新的研究补充。按照其实现方法，将DV-NLI分为符号化NLP方法、深度学习模型方法、大语言模型方法三类，对每个分类下的相关技术进行分析论述。最后，总结并展望DV-NLI的未来工作。

关键词: 数据可视化, 自然语言接口, 机器学习, 神经网络模型, 大语言模型

Abstract: The long-standing goal in the field of data visualization has been to find a solution that directly generates visualizations from natural language. Research on natural language interfaces (NLI) provides a new approach to this field. This interface accepts queries in the form of natural language and tabular datasets as input and outputs corresponding visualization renderings. Simultaneously, as an auxiliary input method, traditional users need to convert analytical intents into a series of logical operations and interact with them, such as programming instructions or graphical interface operations. Combining the use of natural language interfaces for data visualization (DV-NLI) enables users to focus on visualization tasks without worrying about how to operate visualization tools. In recent years, with the rise of large language models (LLM) such as GPT-3 and GPT-4, research on integrating LLM with visualization has become a hot topic. This paper provides a comprehensive review of existing DV-NLIs and supplements them with the latest research. Based on their implementation methods, DV-NLIs are categorized into symbolic NLP methods, deep learning model methods, and large language model methods. It also analyzes and discusses relevant techniques under each category. Finally, the paper summarizes and looks forward to future work in DV-NLI.

Key words: data visualization, natural language interface, machine learning, neural network model, large language model

高帅, 奚雪峰, 郑倩, 崔志明, 盛胜利. 面向数据可视化的自然语言接口研究综述[J]. 计算机工程与应用, 2024, 60(15): 24-41.

GAO Shuai, XI Xuefeng, ZHENG Qian, CUI Zhiming, SHENG Shengli. Review of Research on Natural Language Interfaces for Data Visualization[J]. Computer Engineering and Applications, 2024, 60(15): 24-41.

参考文献

[1] 骆昱宇, 秦雪迪, 谢宇鹏, 等. 智能数据可视分析技术综述[J]. 软件学报, 2024, 35(1): 356-404.
LUO Y Y, QIN X D, XIE Y P, et al. Intelligent data visualization analysis techniques: a survey[J]. Journal of Software, 2024, 35(1): 356-404.
[2] 夏佳志, 李杰, 陈思明, 等. 可视化与人工智能交叉研究综述[J]. 中国科学: 信息科学, 2021, 51(11): 1777-1801.
XIA J Z, LI J, CHEN S M, et al. A survey on interdisciplinary research of visualization and artificial intelligence[J]. Scientia Sinica (Informationis), 2021, 51: 1777-1801.
[3] SHEN L, SHEN E, LUO Y, et al. Towards natural language interfaces for data visualization: a survey[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(6): 3121-3144.
[4] 陶钧, 张宇, 陈晴, 等. 智能可视化与可视分析[J]. 中国图象图形学报, 2023, 28(6): 1909-1926.
TAO J, ZHANG Y, CHEN Q, et al. Intelligent visualization and visual analytics[J]. Journal of Image and Graphics, 2023, 28(6): 1909-1926.
[5] MADDIGAN P, SUSNJAK T. Chat2VIS: generating data visualisations via natural language using ChatGPT, codex and GPT-3 large language models[J]. arXiv:2302.02094, 2023.
[6] MADDIGAN P, SUSNJAK T. Chat2VIS: fine-tuning data visualisations using multilingual natural language text and pre-trained large language models[J]. arXiv:2303.14292, 2023.
[7] GAO T, DONTCHEVA M, ADAR E, et al. DataTone: managing ambiguity in natural language interfaces for data visualization[C]//Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, 2015: 489-500.
[8] SRINIVASAN A, LEE B, HENRY RICHE N, et al. InChorus: designing consistent multimodal interactions for data visua-
lization on tablet devices[C]//Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020: 1-13.
[9] JIA D, IRGER A, BESANCON L, et al. VOICE: visual oracle for interaction, conversation, and explanation[J]. arXiv:2304.04083, 2023.
[10] NARECHANIA A, SRINIVASAN A, STASKO J. NL4DV: a toolkit for generating analytic specifications for data visua-
lization from natural language queries[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(2): 369-379.
[11] METOYER R, ZHI Q, JANCZUK B, et al. Coupling story to visualization: using textual analysis as a bridge between data and interpretation[C]//Proceedings of the 23rd International Conference on Intelligent User Interfaces, 2018: 503-507.
[12] LAI C, LIN Z, JIANG R, et al. Automatic annotation synchronizing with textual description for visualization[C]//Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020.
[13] WANG Y, HOU Z, SHEN L, et al. Towards natural language-based visualization authoring[J]. IEEE Transactions on Visua-
lization and Computer Graphics, 2023, 29(1): 1222-1232.
[14] LUO Y, TANG N, LI G, et al. Natural language to visualization by neural machine translation[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28(1): 217-226.
[15] SONG Y, ZHAO X, WONG R C W, et al. RGVisNet: a hybrid retrieval-generation neural framework towards automatic data visualization generation[C]//Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022: 1646-1655.
[16] BELINKOV Y, GLASS J. Analysis methods in neural language processing: a survey[J]. Transactions of the Association for Computational Linguistics, 2019, 7: 49-72.
[17] YOUNG T, HAZARIKA D, PORIA S, et al. Recent trends in deep learning based natural language processing[J]. IEEE Computational Intelligence Magazine, 2018, 13(3): 55-75.
[18] LOPER E, BIRD S. NLTK: the natural language toolkit[C]//Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain, July 21-26, 2004.
[19] MANNING C, SURDEANU M, BAUER J, et al. The Stanford CoreNLP natural language processing toolkit[C]//Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2014: 55-60.
[20] LEE B, ISENBERG P, RICHE N H, et al. Beyond mouse and keyboard: expanding design considerations for information visualization interactions[J]. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(12): 2689-2698.
[21] SRINIVASAN A, NYAPATHY N, LEE B, et al. Collecting and characterizing natural language utterances for specifying data visualizations[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021: 464.
[22] RAKOTONDRAVONY N, DING Y, HARRISON L. Probablement, wahrscheinlich, likely? a cross-language study of how people verbalize probabilities in iconarray visualizations[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 1189-1199.
[23] SUN Y, LEIGH J, JOHNSON A, et al. Articulate: asemi-automated model for translating natural languagequeries into meaningful visualizations[C]//Proceedings of the 10th International Symposium on Smart Graphics, Banff, Canada, June 24-26, 2010: 184-195.
[24] SETLUR V, BATTERSBY S E, TORY M, et al. Eviza: a natural language interface for visual analysis[C]//Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan, 2016: 365-377.
[25] HOQUE E, SETLUR V, TORY M, et al. Applying pragmatics principles for interaction with visual analytics[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(1): 309-318.
[26] LUO Y, QIN X, TANG N, et al. DeepEye: creating good data visualizations by keyword search[C]//Proceedings of the 2018 International Conference on Management of Data, 2018: 1733-1736.
[27] YU B, SILVA C T. FlowSense: a natural language interface for visual data exploration within a dataflow system[J]. IEEE Transactions on Visualizationand Computer Graphics, 2020, 26(1): 1-11.
[28] SRINIVASAN A, SETLUR V. Snowy: recommending utterances for conversational visual analysis[C]//Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology, 2021: 864-880.
[29] LIU C, HAN Y, JIANG R, et al. ADVISor: automatic visua-
lization answer for natural-language question on tabular data[C]//Proceedings of the IEEE 14th Pacific Visualization Symposium (PacificVis), Tianjin, China, 2021: 11-20.
[30] LUO Y, TANG N, LI G, et al. Synthesizing natural language to visualization (NL2VIS) benchmarks from NL2SQL benchmarks[C]//Proceedings of the 2021 International Conference on Management of Data, 2021: 1235-1247.
[31] LIANG P, YE D, ZHU Z, et al. C5: toward better conversation comprehension and contextual continuity for ChatGPT[J]. arXiv:2308.05567, 2023.
[32] SRINIVASAN A, STASKO J. Orko: facilitating multimodal interaction for visual exploration and analysis of networks[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(1): 511-521.
[33] WEN Z, ZHOU M X, AGGARWAL V. An optimization-based approach to dynamic visual context management[C]//Proceedings of the IEEE Symposium on Information Visua-
lization, 2005: 187-194.
[34] SRINIVASAN A, DRUCKER S M, ENDERT A, et al. Augmenting visualizations with interactive data facts to facilitate interpretation and communication[J]. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(1): 672-681.
[35] DIBIA V, DEMIRALP ?. Data2VIS: automatic generation of data visualizations using sequence-to-sequence recurrent neural networks[J]. IEEE Computer Graphics and Applications, 2019, 39(5): 33-46.
[36] TORY M, SETLUR V. Do what I mean, not what I say! design considerations for supporting intent and context in analytical conversation[C]//Proceedings of the 2019 IEEE Conference on Visual Analytics Science and Technology (VAST), Vancouver, BC, Canada, 2019: 93-103.
[37] JIANG Q, SUN G, DONG Y, et al. DT2VIS: a focus+context answer generation system to facilitate visual exploration of tabular data[J]. IEEE Computer Graphics and Applications, 2021, 41(5): 45-56.
[38] SETLUR V, BATTERSBY S, WONG T. GeoSneakPique: visual autocompletion for geospatial queries[C]//Proceedings of the 2021 IEEE Visualization Conference (VIS), New Orleans, LA, USA, 2021: 166-170.
[39] SRINIVASAN A, LEE B, STASKO J. Interweaving multimodal interaction with flexible unit visualizations for data exploration[J]. IEEE Transactions onVisualization and Computer Graphics, 2021, 27(8): 3519-3533.
[40] QIAN C, SUN S, CUI W, et al. Retrieve-then-adapt: Example-based automatic generation for proportion-related infographics[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(2): 443-452.
[41] SETLUR V, KUMAR A. Sentifiers: interpreting vague intent modifiers in visual analysis using word co-occurrence and sentiment analysis[C]//Proceedings of the 2020 IEEE Visua-
lization Conference (VIS), 2020: 216-220.
[42] BRYAN C, MA K L, WOODRING J. Temporal summary images: an approach to narrative visualization via interactive annotation generation and placement[J]. IEEE Transactions on Visualization and Computer Graphics, 2017, 23(1): 511-520.
[43] CUI W, ZHANG X, WANG Y, et al. Text-to-Viz: automatic generation of infographics from proportion-related natural language statements[J]. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(1): 906-916.
[44] FULDA J, BREHMER M, MUNZNER T. TimeLineCurator: interactive authoring of visual timelines from unstructured text[J]. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(1): 300-309.
[45] HEARST M, TORY M, SETLUR V. Toward interface defaults for vague modifiers in natural language interfaces for visual analysis[C]//Proceedings of the 2019 IEEE Visualization Conference (VIS), Vancouver, BC, Canada, 2019: 21-25.
[46] HUANG J, XI Y, HU J, et al. FlowNL: asking the flow data in natural languages[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 1200-1210.
[47] ARUNKUMAR A, SHARMA S, AGRAWAL R, et al. LINGO: visually debiasing natural language instructions to support task diversity[J]. Computer Graphics Forum, 2023, 42(3): 409-421.
[48] SRINIVASAN A, SETLUR V. BOLT: a natural language interface for dashboard authoring[C]//Proceedings of the 25th Eurographics Conference on Visualization (Short Papers), Leipzig, Germany, June 12-16, 2023: 7-11.
[49] ZHAO J, FAN M, FENG M. ChartSeer: interactive steering exploratory visual analysis with machine intelligence[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28(3): 1500-1513.
[50] ZHAO J, XU S, CHANDRASEGARAN S, et al. ChartStory: automated partitioning, layout, and captioning of charts into comic-style narratives[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(2): 1384-1399.
[51] SHEN L, ZHANG Y, ZHANG H, et al. Data Player: automatic generation of data videos with narration-animation interplay[J]. arXiv:2308.04703, 2023.
[52] FENG Y, WANG X, PAN B, et al. XNLI: explaining and diagnosing NLI-based visual data analysis[J]. arXiv:2301.
10385, 2023.
[53] GUO Y, CAO N, QI X, et al. Urania: visualizing data analysis pipelines for natural language-based data exploration[J]. arXiv:2306.07760, 2023.
[54] KAVAZ E, RODRíGUEZ I, PUIG A, et al. A conversational data visualisation platform for hierarchical multivariate data[C]//Proceedings of the 25th Eurographics Conference on Visualization, Posters, Leipzig, Germany, June 12-16, 2023: 1-3.
[55] WANG Y, SHEN L, YOU Z, et al. WonderFlow: narration-centric design of animated data videos[J]. arXiv:2308.04040, 2023.
[56] GUO Y, CAO N, CAI L, et al. Datamator: an authoring tool for creating datamations via data query decomposition[J]. Applied Sciences, 2023, 13(17): 9709.
[57] JOHN R J L, POTTI N, PATEL J M. Ava: from data to insights through conversations[C]//Proceedings of the 8th Biennial Conference on Innovative Data Systems Research, Chaminade, CA, USA, January 8-11, 2017.
[58] KUMAR A, DI EUGENIO B, AURISANO J, et al. Towards multimodal coreference resolution for exploratory data visualization dialogue: context-based annotation and gesture identification[C]//Proceedings of the 21st Workshop on the Semantics and Pragmatics of Dialogue, 2017.
[59] MURILLO-MORALES T, MIESENBERGER K. Audial: a natural language interface to make statistical charts accessible to blind persons[C]//Proceedings of the17th International Conference on Computers Helping People with Special Needs, Lecco, Italy, Sep 9-11, 2020: 373-384.
[60] BACCI F, CAU F M, SPANO L D. Inspecting data using natural language queries[C]//Proceedings of the 20th International Conference on Computational Science and Its Applications, Cagliari, Italy, July 1-4, 2020: 771-782.
[61] MITRI M. Story analysis using natural language processing and interactive dashboards[J]. Journal of Computer Information Systems, 2022, 62(2): 216-226.
[62] MASSON D, MALACRIA S, CASIEZ G, et al. Charagraph: interactive generation of charts for realtime annotation of data-rich paragraphs[C]//Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023: 146.
[63] CAO Y, E J L, CHEN Z, et al. DataParticles: block-based and language-oriented authoring of animated unit visualizations[C]//Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023: 808.
[64] KIM Y H, LEE B, SRINIVASAN A, et al. Data@Hand: fostering visual exploration of personal data on smartphones leveraging speech and touch interaction[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021.
[65] DHAMDHERE K, MCCURLEY K S, NAHMIAS R, et al. Analyza: exploring data with conversation[C]//Proceedings of the 22nd International Conference on Intelligent User Interfaces, 2017: 493-504.
[66] KATO T, MATSUSHITA M, MAEDA E. Answering it with charts: dialogue in natural language and charts[C]//Proceedings of the 19th International Conference on Computational Linguistics, 2002: 1-7.
[67] HULLMAN J, DIAKOPOULOS N, ADAR E. Contextifier: automatic generation of annotated stock visualizations[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2013: 2707-2716.
[68] SETLUR V, TORY M, DJALALI A. Inferencing underspecified natural language utterances in visual analysis[C]//Proceedings of the 24th International Conference on Intelligent User Interfaces, 2019: 40-51.
[69] FAST E, CHEN B, MENDELSOHN J, et al. Iris: a conversational agent for complex tasks[C]//Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018: 473.
[70] GAO T, HULLMAN J R, ADAR E, et al. NewsViews: an automated pipeline for creating custom geovisualizations for news[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2014: 3005-3014.
[71] SETLUR V, HOQUE E, KIM D H, et al. Sneak Pique: exploring autocompletion as a data discovery scaffold for supporting visual analysis[C]//Proceedings of the 33rd Annual ACM Symposium on UserInterface Software and Technology, 2020: 966-978.
[72] KUMAR A, AURISANO J, DI EUGENIO B, et al. Towards a dialogue system that supports rich visualizations of data[C]//Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2016: 304-309.
[73] KASSEL J F, ROHS M. Valletto: a multimodal interface for ubiquitous visual analytics[C]//Proceedings of the Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, 2018: 1-6.
[74] SETLUR V, KANYUKA A, SRINIVASAN A. Olio: a semantic search interface for data repositories[J]. arXiv:2307.16396, 2023.
[75] SHEN L, SHEN E, TAI Z, et al. Visual data analysis with task-based recommendations[J]. Data Science and Engineering, 2022, 7(4): 354-369.
[76] COX K, GRINTER R E, HIBINO S L, et al. A multi-modal natural language interface to an informationvisualization environment[J]. International Journal of Speech Technology, 2001, 4: 297-314.
[77] PARR T. The definitive ANTLR 4 reference[J]. Pragmatic Bookshelf, 2013.
[78] ZONG J, POLLOCK J, WOOTTON D, et al. Animated Vega-Lite: unifying animation with a grammar ofinteractive graphics[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 149-159.
[79] VOIGT H, MEUSCHKE M, LAWONN K, et al. Challenges in designing natural language interfaces for complex visual models[C]//Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing, 2021: 66-73.
[80] LUO Y, TANG J, LI G. nvBench: a large-scale synthesized dataset for cross-domain natural language to visualization task[J]. arXiv:2112.12926, 2021.
[81] TANG J, LUO Y, OUZZANI M, et al. Sevi: speech-to-visualization through neural machine translation[C]//Proceedings of the 2022 International Conference on Management of Data, 2022: 2353-2356.
[82] SCARSELLI F, GORI M, TSOI A C, et al. The graph neural network model[J]. IEEE Transactions on Neural Networks, 2009, 20(1): 61-80.
[83] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv:1810.04805, 2018.
[84] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]//Advances in Neural Information Processing Systems, 2020: 1877-1901.
[85] DENG J, DONG W, SOCHER R, et al. ImageNet: alarge-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009: 248-255.
[86] GEHRMANN S, ADEWUMI T, AGGARWAL K, et al. The GEM benchmark: natural language generation, its evaluation and metrics[J]. arXiv:2102.01672, 2021.
[87] FU S, XIONG K, GE X, et al. Quda: natural language queries for visual data analytics[J]. arXiv:2005.03257, 2020.
[88] HU K, GAIKWAD S S, HULSEBOS M, et al. VizNet: towards a large-scale visualization learning and benchmarking repository[C]//Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019: 662.
[89] ZHANG Z. CPM-2: large-scale cost-effective pre-trained language models[J]. AI Open, 2021, 2: 216-224.