Contrary to the notions of language as an amodal system, natural human communication is multimodal and combines speech, gestures, writing, and pictures. To account for this, we have proposed that our vocal, bodily, and graphic modalities persist in parallel in a multimodal language faculty, and both unimodal and multimodal expressions arise out of emergent states of this shared architecture. Such a model carries different expectations for the ways in which modalities may be similar or different from each other, and how they may interact. I will highlight these properties specifically for the graphic modality, which can manifest in full visual languages with a systematic lexicon and complex grammar. Along with behavioral and neurocognitive research, I focus on analysis of a corpus 1,030 annotated comics which will provide evidence for diverse visual languages, such as contrasts between the American Visual Language used in superhero comics and the Japanese Visual Language used in manga. Yet, consistent “universal” linguistic principles persist across this structural diversity, and subtle structures of spoken languages permeate across to visual languages. Altogether, this work argues for a multimodal basis of linguistic structure, and heralds a reconsideration of what constitutes the language system.