A Framework to Assess Multilingual Vulnerabilities of LLMs

Simple item page
dc.contributor.authorTang L
dc.contributor.authorBogahawatta N
dc.contributor.authorGinige Y
dc.contributor.authorXu J
dc.contributor.authorSun S
dc.contributor.authorRanathunga S
dc.contributor.authorSeneviratne S
dc.coverage.spatialSydney NSW, Australia
dc.date.accessioned2025-08-27T01:06:28Z
dc.date.available2025-08-27T01:06:28Z
dc.date.finish-date2025-05-02
dc.date.issued2025-05-23
dc.date.start-date2025-04-28
dc.description.abstractLarge Language Models (LLMs) are acquiring a wider range of capabilities, including understanding and responding in multiple languages. While they undergo safety training to prevent them from answering illegal questions, imbalances in training data and human evaluation resources can make these models more susceptible to attacks in low-resource languages (LRL). This paper proposes a framework to automatically assess the multilingual vulnerabilities of commonly used LLMs. Using our framework, we evaluated six LLMs across eight languages representing varying levels of resource availability. We validated the assessments generated by our automated framework through human evaluation in two languages, demonstrating that the framework's results align with human judgments in most cases. Our findings reveal vulnerabilities in LRL; however, these may pose minimal risk as they often stem from the model's poor performance, resulting in incoherent responses.
dc.description.confidentialfalse
dc.description.place-of-publicationNew York, NY, United States
dc.format.pagination1331-1335
dc.identifier.citationTang L, Bogahawatta N, Ginige Y, Xu J, Sun S, Ranathunga S, Seneviratne S. (2025). A Framework to Assess Multilingual Vulnerabilities of LLMs. Www Companion 2025 Companion Proceedings of the ACM Web Conference 2025. (pp. 1331-1335). New York, NY, United States. Association for Computing Machinery.
dc.identifier.doi10.1145/3701716.3715581
dc.identifier.elements-typec-conference-paper-in-proceedings
dc.identifier.isbn9798400713316
dc.identifier.urihttps://mro.massey.ac.nz/handle/10179/73426
dc.publisherAssociation for Computing Machinery
dc.publisher.urihttp://dl.acm.org/doi/10.1145/3701716.3715581
dc.rights(c) 2025 The Author/s
dc.rightsCC BY 4.0
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.source.journalWww Companion 2025 Companion Proceedings of the ACM Web Conference 2025
dc.source.name-of-conferenceThe ACM Web Conference 2025
dc.subjectLarge Language Models
dc.subjectLLM Red Teaming
dc.subjectJail Breaking
dc.titleA Framework to Assess Multilingual Vulnerabilities of LLMs
dc.typeconference
pubs.elements-id501549
pubs.organisational-groupOther
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
501549 PDF.pdf
Size:
1.63 MB
Format:
Adobe Portable Document Format
Description:
Published version.pdf
Download
License bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
license.txt
Size:
9.22 KB
Format:
Plain Text
Description:
Download
Collections
Conference Papers