Comparisons of 3D shapes have recently been applied to diverse anatomical structures using landmarking techniques. However discerning evolutionary patterns can be challenging for structures lacking homologous landmarks. We used alpha shape analyses to quantify vaginal shape complexity in 40 marine mammal specimens including cetaceans, pinnipeds, and sirenians. We explored phylogenetic signal and the potential roles of natural and sexual selection on vaginal shape evolution. Complexity scores were consistent with qualitative observations. Cetaceans had a broad range of alpha complexities, while pinnipeds were comparatively simple and sirenians were complex. Intraspecific variation was found. Three-dimensional surface heat maps revealed that shape complexity was driven by invaginations and protrusions of the vaginal wall. Phylogenetic signal was weak and metrics of natural selection (relative neonate size) and sexual selection (relative testes size, sexual size dimorphism, and penis morphology) did not explain vaginal complexity patterns. Additional metrics, such as penile shape complexity, may yield interesting insights into marine mammal genital coevolution. We advocate for the use of alpha shapes to discern patterns of evolution that would otherwise not be possible in 3D anatomical structures lacking homologous landmarks.