Accelerating math accessibility with the use of AI
A year ago, NWEA, now part of HMH, shared their innovative approach to make math more accessible for students. The aim was to identify the biggest challenges and gaps in mathematics for students who use screen readers and refreshable braille devices, because classroom materials are not always adapted to formats such as braille or large print, and materials are not always suitable for a screen-reader navigation, voice input, or a combination of these designs. NWEA developed prototypes that enabled screen readers to interact with equations in a more intuitive way, based on a method called process driven math (PDM).
NWEA continued to innovate and build on their previous research to create different ways of presenting complex math, especially for math taught in grades six to nine. They also worked on different ways of outputting math that included screen-reader functionality and refreshable braille devices in both UEB (Unified English Braille) and Nemeth formats. Moreover, they developed a prototype for a voice-activated chatbot.
To account for the accessibility of math equations, they used two markup languages, HTML and ARIA, to split equations into parts or regions. Each region, as well as the whole equation, had a hidden label that a screen reader would say to users as they explored the equation or expression. When students moved from one region to another, they would hear a word that described the kind of math in that region (for example, “term” or “constant”). Students could then decide to go into the region and hear the exact math, or they could just skip to the next region.
The use of generative AI
By using AI, specifically GPT-4, the team was able to improve both the quality of the math as well as the time required to convert the equations to HTML, and to use code generation to write the code for the first prototype. The model only needed a few examples to learn how to change the initial test set of equations from MathML to the HTML structure that was the most accessible. From there, the model required context to ensure that responses were formatted in the best way for the app.
Demo of using the equations with a screen reader:
For their future goals, they would like to leverage embeddings to create a knowledge base that helps improve the conversion accuracy of more complex equations, since equations have a well-defined structure. They also tested a prototype speech interface that showed the possibility of constructing a chatbot that students can use to read and explore math in detail. Similar to the PDM model, students can listen to a virtual assistant read an equation’s regional labels, then command the assistant to move specifically to a region and read its contents in detail. They want to refine and test this approach further to prove that speech is a viable option for students with disabilities to interact with math: for example, students with vision disabilities and mobility disabilities may find it easier to use the virtual assistant to navigate the equation rather than use a keyboard and screen reader.
With any new technology we must actively work towards inclusion, otherwise it can lead to exclusion. This project was an attempt to find a new solution to a long-standing challenge for students and teachers with vision disabilities, “how can we make digital math more accessible”. In future projects, NWEA wants to explore how generative AI can learn the rules for both braille and process driven math to produce accessible equations.
Connect articles and links:
Making Mathematics Accessible – Microsoft Accessibility Blog
