Extracting Situation Frames from Non-English Speech: Evaluation Framework and Pilot Results

This paper describes the first evaluation framework for the extraction of Situation Frames — structures describing humanitarian assistance needs — from non-English speech audio, conducted for the DARPA LORELEI (Low Resource Languages for Emergent Incidents) program. Participants in LORELEI had to process audio from a variety of sources, in non-English languages, and extract the information required to populate Situation Frames describing whether any need is mentioned, the type of need present and where the need exists. The evaluation was conducted over a period of 10 days and attracted submissions from 6 teams, each team spanning multiple organizations. Performance was evaluated using precision-recall curves. The results are encouraging, with most teams showing some capability to detect the type of situation discussed, but more work will be required to connect needs to specific locations.