Color-Aware Structured Parsing and Self-Consistent Reasoning for Chart Question Answering

Abstract

Colors are critical elements in charts. They allow us to visualize data in different series or categories effectively. Currently, generative artificial intelligence such as vision language models (VLMs) and large language models (LLMs) can be combined together to form a pipeline for extracting information and answering questions regarding the data presented on a given chart. However, most chart question answering pipelines still treat every chart as if it were rendered in grayscale, ignoring the valuable chromatic attributes. This paper introduces ColorAware Structured Parsing (CASP) with Self-Consistent Reasoning, a two-stage pipeline that extracts both numerical and chromatic information to improve chart understanding. On a 2,500 question Chart Question Answering (ChartQA) benchmark, CASP attains 91.12% exact-match accuracy, a significant gain over the strongest structured baseline. The answers produced by CASP are also auditable. They include explicit pointers to the table cells used for reasoning. By exploiting both numerical and chromatic information on the chart and enforcing agreement across reasoning paths, CASP turns colors into usable evidence, delivering accurate and transparent answers regarding the information embedded in the chart.