Computer Engineering and Applications ›› 2024, Vol. 60 ›› Issue (15): 24-41.DOI: 10.3778/j.issn.1002-8331.2310-0167
• Research Hotspots and Reviews • Previous Articles Next Articles
GAO Shuai, XI Xuefeng, ZHENG Qian, CUI Zhiming, SHENG Shengli
Online:
2024-08-01
Published:
2024-07-30
高帅,奚雪峰,郑倩,崔志明,盛胜利
GAO Shuai, XI Xuefeng, ZHENG Qian, CUI Zhiming, SHENG Shengli. Review of Research on Natural Language Interfaces for Data Visualization[J]. Computer Engineering and Applications, 2024, 60(15): 24-41.
高帅, 奚雪峰, 郑倩, 崔志明, 盛胜利. 面向数据可视化的自然语言接口研究综述[J]. 计算机工程与应用, 2024, 60(15): 24-41.
Add to citation manager EndNote|Ris|BibTeX
URL: http://cea.ceaj.org/EN/10.3778/j.issn.1002-8331.2310-0167
[1] 骆昱宇, 秦雪迪, 谢宇鹏, 等. 智能数据可视分析技术综述[J]. 软件学报, 2024, 35(1): 356-404. LUO Y Y, QIN X D, XIE Y P, et al. Intelligent data visualization analysis techniques: a survey[J]. Journal of Software, 2024, 35(1): 356-404. [2] 夏佳志, 李杰, 陈思明, 等. 可视化与人工智能交叉研究综述[J]. 中国科学: 信息科学, 2021, 51(11): 1777-1801. XIA J Z, LI J, CHEN S M, et al. A survey on interdisciplinary research of visualization and artificial intelligence[J]. Scientia Sinica (Informationis), 2021, 51: 1777-1801. [3] SHEN L, SHEN E, LUO Y, et al. Towards natural language interfaces for data visualization: a survey[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(6): 3121-3144. [4] 陶钧, 张宇, 陈晴, 等. 智能可视化与可视分析[J]. 中国图象图形学报, 2023, 28(6): 1909-1926. TAO J, ZHANG Y, CHEN Q, et al. Intelligent visualization and visual analytics[J]. Journal of Image and Graphics, 2023, 28(6): 1909-1926. [5] MADDIGAN P, SUSNJAK T. Chat2VIS: generating data visualisations via natural language using ChatGPT, codex and GPT-3 large language models[J]. arXiv:2302.02094, 2023. [6] MADDIGAN P, SUSNJAK T. Chat2VIS: fine-tuning data visualisations using multilingual natural language text and pre-trained large language models[J]. arXiv:2303.14292, 2023. [7] GAO T, DONTCHEVA M, ADAR E, et al. DataTone: managing ambiguity in natural language interfaces for data visualization[C]//Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, 2015: 489-500. [8] SRINIVASAN A, LEE B, HENRY RICHE N, et al. InChorus: designing consistent multimodal interactions for data visua- lization on tablet devices[C]//Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020: 1-13. [9] JIA D, IRGER A, BESANCON L, et al. VOICE: visual oracle for interaction, conversation, and explanation[J]. arXiv:2304.04083, 2023. [10] NARECHANIA A, SRINIVASAN A, STASKO J. NL4DV: a toolkit for generating analytic specifications for data visua- lization from natural language queries[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(2): 369-379. [11] METOYER R, ZHI Q, JANCZUK B, et al. Coupling story to visualization: using textual analysis as a bridge between data and interpretation[C]//Proceedings of the 23rd International Conference on Intelligent User Interfaces, 2018: 503-507. [12] LAI C, LIN Z, JIANG R, et al. Automatic annotation synchronizing with textual description for visualization[C]//Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020. [13] WANG Y, HOU Z, SHEN L, et al. Towards natural language-based visualization authoring[J]. IEEE Transactions on Visua- lization and Computer Graphics, 2023, 29(1): 1222-1232. [14] LUO Y, TANG N, LI G, et al. Natural language to visualization by neural machine translation[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28(1): 217-226. [15] SONG Y, ZHAO X, WONG R C W, et al. RGVisNet: a hybrid retrieval-generation neural framework towards automatic data visualization generation[C]//Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022: 1646-1655. [16] BELINKOV Y, GLASS J. Analysis methods in neural language processing: a survey[J]. Transactions of the Association for Computational Linguistics, 2019, 7: 49-72. [17] YOUNG T, HAZARIKA D, PORIA S, et al. Recent trends in deep learning based natural language processing[J]. IEEE Computational Intelligence Magazine, 2018, 13(3): 55-75. [18] LOPER E, BIRD S. NLTK: the natural language toolkit[C]//Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain, July 21-26, 2004. [19] MANNING C, SURDEANU M, BAUER J, et al. The Stanford CoreNLP natural language processing toolkit[C]//Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2014: 55-60. [20] LEE B, ISENBERG P, RICHE N H, et al. Beyond mouse and keyboard: expanding design considerations for information visualization interactions[J]. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(12): 2689-2698. [21] SRINIVASAN A, NYAPATHY N, LEE B, et al. Collecting and characterizing natural language utterances for specifying data visualizations[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021: 464. [22] RAKOTONDRAVONY N, DING Y, HARRISON L. Probablement, wahrscheinlich, likely? a cross-language study of how people verbalize probabilities in iconarray visualizations[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 1189-1199. [23] SUN Y, LEIGH J, JOHNSON A, et al. Articulate: asemi-automated model for translating natural languagequeries into meaningful visualizations[C]//Proceedings of the 10th International Symposium on Smart Graphics, Banff, Canada, June 24-26, 2010: 184-195. [24] SETLUR V, BATTERSBY S E, TORY M, et al. Eviza: a natural language interface for visual analysis[C]//Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan, 2016: 365-377. [25] HOQUE E, SETLUR V, TORY M, et al. Applying pragmatics principles for interaction with visual analytics[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(1): 309-318. [26] LUO Y, QIN X, TANG N, et al. DeepEye: creating good data visualizations by keyword search[C]//Proceedings of the 2018 International Conference on Management of Data, 2018: 1733-1736. [27] YU B, SILVA C T. FlowSense: a natural language interface for visual data exploration within a dataflow system[J]. IEEE Transactions on Visualizationand Computer Graphics, 2020, 26(1): 1-11. [28] SRINIVASAN A, SETLUR V. Snowy: recommending utterances for conversational visual analysis[C]//Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology, 2021: 864-880. [29] LIU C, HAN Y, JIANG R, et al. ADVISor: automatic visua- lization answer for natural-language question on tabular data[C]//Proceedings of the IEEE 14th Pacific Visualization Symposium (PacificVis), Tianjin, China, 2021: 11-20. [30] LUO Y, TANG N, LI G, et al. Synthesizing natural language to visualization (NL2VIS) benchmarks from NL2SQL benchmarks[C]//Proceedings of the 2021 International Conference on Management of Data, 2021: 1235-1247. [31] LIANG P, YE D, ZHU Z, et al. C5: toward better conversation comprehension and contextual continuity for ChatGPT[J]. arXiv:2308.05567, 2023. [32] SRINIVASAN A, STASKO J. Orko: facilitating multimodal interaction for visual exploration and analysis of networks[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(1): 511-521. [33] WEN Z, ZHOU M X, AGGARWAL V. An optimization-based approach to dynamic visual context management[C]//Proceedings of the IEEE Symposium on Information Visua- lization, 2005: 187-194. [34] SRINIVASAN A, DRUCKER S M, ENDERT A, et al. Augmenting visualizations with interactive data facts to facilitate interpretation and communication[J]. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(1): 672-681. [35] DIBIA V, DEMIRALP ?. Data2VIS: automatic generation of data visualizations using sequence-to-sequence recurrent neural networks[J]. IEEE Computer Graphics and Applications, 2019, 39(5): 33-46. [36] TORY M, SETLUR V. Do what I mean, not what I say! design considerations for supporting intent and context in analytical conversation[C]//Proceedings of the 2019 IEEE Conference on Visual Analytics Science and Technology (VAST), Vancouver, BC, Canada, 2019: 93-103. [37] JIANG Q, SUN G, DONG Y, et al. DT2VIS: a focus+context answer generation system to facilitate visual exploration of tabular data[J]. IEEE Computer Graphics and Applications, 2021, 41(5): 45-56. [38] SETLUR V, BATTERSBY S, WONG T. GeoSneakPique: visual autocompletion for geospatial queries[C]//Proceedings of the 2021 IEEE Visualization Conference (VIS), New Orleans, LA, USA, 2021: 166-170. [39] SRINIVASAN A, LEE B, STASKO J. Interweaving multimodal interaction with flexible unit visualizations for data exploration[J]. IEEE Transactions onVisualization and Computer Graphics, 2021, 27(8): 3519-3533. [40] QIAN C, SUN S, CUI W, et al. Retrieve-then-adapt: Example-based automatic generation for proportion-related infographics[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(2): 443-452. [41] SETLUR V, KUMAR A. Sentifiers: interpreting vague intent modifiers in visual analysis using word co-occurrence and sentiment analysis[C]//Proceedings of the 2020 IEEE Visua- lization Conference (VIS), 2020: 216-220. [42] BRYAN C, MA K L, WOODRING J. Temporal summary images: an approach to narrative visualization via interactive annotation generation and placement[J]. IEEE Transactions on Visualization and Computer Graphics, 2017, 23(1): 511-520. [43] CUI W, ZHANG X, WANG Y, et al. Text-to-Viz: automatic generation of infographics from proportion-related natural language statements[J]. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(1): 906-916. [44] FULDA J, BREHMER M, MUNZNER T. TimeLineCurator: interactive authoring of visual timelines from unstructured text[J]. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(1): 300-309. [45] HEARST M, TORY M, SETLUR V. Toward interface defaults for vague modifiers in natural language interfaces for visual analysis[C]//Proceedings of the 2019 IEEE Visualization Conference (VIS), Vancouver, BC, Canada, 2019: 21-25. [46] HUANG J, XI Y, HU J, et al. FlowNL: asking the flow data in natural languages[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 1200-1210. [47] ARUNKUMAR A, SHARMA S, AGRAWAL R, et al. LINGO: visually debiasing natural language instructions to support task diversity[J]. Computer Graphics Forum, 2023, 42(3): 409-421. [48] SRINIVASAN A, SETLUR V. BOLT: a natural language interface for dashboard authoring[C]//Proceedings of the 25th Eurographics Conference on Visualization (Short Papers), Leipzig, Germany, June 12-16, 2023: 7-11. [49] ZHAO J, FAN M, FENG M. ChartSeer: interactive steering exploratory visual analysis with machine intelligence[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28(3): 1500-1513. [50] ZHAO J, XU S, CHANDRASEGARAN S, et al. ChartStory: automated partitioning, layout, and captioning of charts into comic-style narratives[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(2): 1384-1399. [51] SHEN L, ZHANG Y, ZHANG H, et al. Data Player: automatic generation of data videos with narration-animation interplay[J]. arXiv:2308.04703, 2023. [52] FENG Y, WANG X, PAN B, et al. XNLI: explaining and diagnosing NLI-based visual data analysis[J]. arXiv:2301. 10385, 2023. [53] GUO Y, CAO N, QI X, et al. Urania: visualizing data analysis pipelines for natural language-based data exploration[J]. arXiv:2306.07760, 2023. [54] KAVAZ E, RODRíGUEZ I, PUIG A, et al. A conversational data visualisation platform for hierarchical multivariate data[C]//Proceedings of the 25th Eurographics Conference on Visualization, Posters, Leipzig, Germany, June 12-16, 2023: 1-3. [55] WANG Y, SHEN L, YOU Z, et al. WonderFlow: narration-centric design of animated data videos[J]. arXiv:2308.04040, 2023. [56] GUO Y, CAO N, CAI L, et al. Datamator: an authoring tool for creating datamations via data query decomposition[J]. Applied Sciences, 2023, 13(17): 9709. [57] JOHN R J L, POTTI N, PATEL J M. Ava: from data to insights through conversations[C]//Proceedings of the 8th Biennial Conference on Innovative Data Systems Research, Chaminade, CA, USA, January 8-11, 2017. [58] KUMAR A, DI EUGENIO B, AURISANO J, et al. Towards multimodal coreference resolution for exploratory data visualization dialogue: context-based annotation and gesture identification[C]//Proceedings of the 21st Workshop on the Semantics and Pragmatics of Dialogue, 2017. [59] MURILLO-MORALES T, MIESENBERGER K. Audial: a natural language interface to make statistical charts accessible to blind persons[C]//Proceedings of the17th International Conference on Computers Helping People with Special Needs, Lecco, Italy, Sep 9-11, 2020: 373-384. [60] BACCI F, CAU F M, SPANO L D. Inspecting data using natural language queries[C]//Proceedings of the 20th International Conference on Computational Science and Its Applications, Cagliari, Italy, July 1-4, 2020: 771-782. [61] MITRI M. Story analysis using natural language processing and interactive dashboards[J]. Journal of Computer Information Systems, 2022, 62(2): 216-226. [62] MASSON D, MALACRIA S, CASIEZ G, et al. Charagraph: interactive generation of charts for realtime annotation of data-rich paragraphs[C]//Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023: 146. [63] CAO Y, E J L, CHEN Z, et al. DataParticles: block-based and language-oriented authoring of animated unit visualizations[C]//Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023: 808. [64] KIM Y H, LEE B, SRINIVASAN A, et al. Data@Hand: fostering visual exploration of personal data on smartphones leveraging speech and touch interaction[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021. [65] DHAMDHERE K, MCCURLEY K S, NAHMIAS R, et al. Analyza: exploring data with conversation[C]//Proceedings of the 22nd International Conference on Intelligent User Interfaces, 2017: 493-504. [66] KATO T, MATSUSHITA M, MAEDA E. Answering it with charts: dialogue in natural language and charts[C]//Proceedings of the 19th International Conference on Computational Linguistics, 2002: 1-7. [67] HULLMAN J, DIAKOPOULOS N, ADAR E. Contextifier: automatic generation of annotated stock visualizations[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2013: 2707-2716. [68] SETLUR V, TORY M, DJALALI A. Inferencing underspecified natural language utterances in visual analysis[C]//Proceedings of the 24th International Conference on Intelligent User Interfaces, 2019: 40-51. [69] FAST E, CHEN B, MENDELSOHN J, et al. Iris: a conversational agent for complex tasks[C]//Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018: 473. [70] GAO T, HULLMAN J R, ADAR E, et al. NewsViews: an automated pipeline for creating custom geovisualizations for news[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2014: 3005-3014. [71] SETLUR V, HOQUE E, KIM D H, et al. Sneak Pique: exploring autocompletion as a data discovery scaffold for supporting visual analysis[C]//Proceedings of the 33rd Annual ACM Symposium on UserInterface Software and Technology, 2020: 966-978. [72] KUMAR A, AURISANO J, DI EUGENIO B, et al. Towards a dialogue system that supports rich visualizations of data[C]//Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2016: 304-309. [73] KASSEL J F, ROHS M. Valletto: a multimodal interface for ubiquitous visual analytics[C]//Proceedings of the Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, 2018: 1-6. [74] SETLUR V, KANYUKA A, SRINIVASAN A. Olio: a semantic search interface for data repositories[J]. arXiv:2307.16396, 2023. [75] SHEN L, SHEN E, TAI Z, et al. Visual data analysis with task-based recommendations[J]. Data Science and Engineering, 2022, 7(4): 354-369. [76] COX K, GRINTER R E, HIBINO S L, et al. A multi-modal natural language interface to an informationvisualization environment[J]. International Journal of Speech Technology, 2001, 4: 297-314. [77] PARR T. The definitive ANTLR 4 reference[J]. Pragmatic Bookshelf, 2013. [78] ZONG J, POLLOCK J, WOOTTON D, et al. Animated Vega-Lite: unifying animation with a grammar ofinteractive graphics[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 149-159. [79] VOIGT H, MEUSCHKE M, LAWONN K, et al. Challenges in designing natural language interfaces for complex visual models[C]//Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing, 2021: 66-73. [80] LUO Y, TANG J, LI G. nvBench: a large-scale synthesized dataset for cross-domain natural language to visualization task[J]. arXiv:2112.12926, 2021. [81] TANG J, LUO Y, OUZZANI M, et al. Sevi: speech-to-visualization through neural machine translation[C]//Proceedings of the 2022 International Conference on Management of Data, 2022: 2353-2356. [82] SCARSELLI F, GORI M, TSOI A C, et al. The graph neural network model[J]. IEEE Transactions on Neural Networks, 2009, 20(1): 61-80. [83] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv:1810.04805, 2018. [84] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]//Advances in Neural Information Processing Systems, 2020: 1877-1901. [85] DENG J, DONG W, SOCHER R, et al. ImageNet: alarge-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009: 248-255. [86] GEHRMANN S, ADEWUMI T, AGGARWAL K, et al. The GEM benchmark: natural language generation, its evaluation and metrics[J]. arXiv:2102.01672, 2021. [87] FU S, XIONG K, GE X, et al. Quda: natural language queries for visual data analytics[J]. arXiv:2005.03257, 2020. [88] HU K, GAIKWAD S S, HULSEBOS M, et al. VizNet: towards a large-scale visualization learning and benchmarking repository[C]//Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019: 662. [89] ZHANG Z. CPM-2: large-scale cost-effective pre-trained language models[J]. AI Open, 2021, 2: 216-224. |
[1] | PEI Wencan, SUN Guangwei, HUANG Jinguo, XU Dinghui, LIU Jing. Immediate Prediction Model of SPAD Value and Maturity of Fresh Tobacco Leaves in Field [J]. Computer Engineering and Applications, 2024, 60(8): 348-360. |
[2] | XING Changzheng, XU Jiayu. Hybrid LightGBM Model for Breast Cancer Diagnosis [J]. Computer Engineering and Applications, 2024, 60(6): 330-338. |
[3] | SONG Cheng, XIE Zhenping. Dataset Enhancement Quality Evaluation Method for Chinese Error Correction Task as Example [J]. Computer Engineering and Applications, 2024, 60(3): 331-339. |
[4] | JIANG Lulu, GAO Jintao. Survey of Machine Learning for Database Parameter Tuning Techniques [J]. Computer Engineering and Applications, 2024, 60(3): 1-16. |
[5] | WU Haitao, CAI Yongqi, GAO Jianhua. Bagging Heterogeneous Ensemble Code Smell Detection and Refactoring Priority Division [J]. Computer Engineering and Applications, 2024, 60(3): 138-147. |
[6] | SU Youli, HU Xuanyu, MA Shijie, ZHANG Yuning, Abudukelimu Abulizi, Halidanmu Abudukelimu. Review of Research on Artificial Intelligence in Traditional Chinese Medicine Diagnosis and Treatment [J]. Computer Engineering and Applications, 2024, 60(16): 1-18. |
[7] | YU Fengrui. Survey on Automated Recognition and Extraction of TTPs [J]. Computer Engineering and Applications, 2024, 60(13): 1-22. |
[8] | DENG Shangkun, NING Hong, LIU Zonghua, ZHU Yingke. Interpretable Machine Learning Model for Default Risk Identification of Corporate Bonds [J]. Computer Engineering and Applications, 2024, 60(12): 334-345. |
[9] | ZHAO Jigui, QIAN Yurong, WANG Kui, HOU Shuxiang, CHEN Jiaying. Survey of Chinese Named Entity Recognition Research [J]. Computer Engineering and Applications, 2024, 60(1): 15-27. |
[10] | ZHANG Xu, YANG Xuezhi, LIU Xuenan, FANG Shuai. Non-Contact Atrial Fibrillation Detection Based on Video Pulse Features [J]. Computer Engineering and Applications, 2023, 59(8): 331-340. |
[11] | ZHOU Yurong, ZHANG Qiaoling, YU Guangzeng, XU Weiqiang. Review of Acoustic Signal-Based Industrial Equipment Fault Diagnosis [J]. Computer Engineering and Applications, 2023, 59(7): 51-63. |
[12] | XU Dongdong, CAI Xiaohong, LIU Jing, CAO Hui. Review of Depression Detection Using Social Media Text Data [J]. Computer Engineering and Applications, 2023, 59(4): 54-63. |
[13] | GU Jian, QIAN Yurong, WANG Lanlan, HU Yue, CHEN Jiaying, LENG Hongyong, MA Mengnan. Survey of Artificial Intelligence in Functional Magnetic Resonance Imaging Data for Autism [J]. Computer Engineering and Applications, 2023, 59(22): 57-68. |
[14] | JIANG Hongxun, JIANG Junyi, LIANG Xun. Survey on Credit Card Transaction Fraud Detection Based on Machine Learning [J]. Computer Engineering and Applications, 2023, 59(21): 1-25. |
[15] | PEI Wenbin, WANG Hailong, LIU Lin, PEI Dongmei. Review of Musical Instrument Recognition in Music Information Retrieval [J]. Computer Engineering and Applications, 2023, 59(2): 34-47. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||