Escalation is enemy-driven evolution. This top-down view of an evolutionary arms race between predator and prey downplays the role of prey in driving the predator’s evolution. In the related process of coevolution, two or more species change reciprocally in response to one another. I examined the interaction between busyconine gastropods and their bivalve prey Mercenaria to evaluate whether reciprocal adaptation (coevolution) was likely to occur in this predator-prey system. Whelks either employ a wedging or a chipping mode of predation when feeding on bivalve prey that often results in the predator breaking its own shell. Prey in this interaction have been hypothesized to be “dangerous” because they are able to inflict damage to the predator as a consequence of the interaction. Asymmetry in selection pressure (which is thought to preclude reciprocity of adaptation) is reduced when predators interact with damage-inducing prey.

The likelihood of a reciprocal selection response of the predator in the interaction involving the shell-chipping whelk Sinistrofulgur sinistrum and its bivalve prey Mercenaria mercenaria was viewed by regressing predator performance (a function of the frequency of shell breakage in encounters with prey) on prey phenotype (a function of size). Experimental results indicate that interaction with Mercenaria has strong and predictable consequences (r = 0.973; p = 0.028) for Sinistrofulgur, indicating that evolutionary response of the predator to prey adaptation is likely in this system. Predators that select large bivalve prey increase the likelihood that their shell lip will be broken in the process of attempting to open their prey.

The late Oligocene to Recent fossil record of whelk predation traces on shells of Mercenaria sp. was analyzed to constrain the temporal window of possible coevolution between shell-chipping whelks and Mercenaria. The fossil record of successful and unsuccessful whelk predation traces suggests that chipping behavior evolved in the Busycon-Sinistrofulgur clade in the early late Pliocene, which constrains tests of reciprocal adaptation with Mercenaria to the Pliocene to Recent. Temporal trends in the frequency of whelk predation traces on Mercenaria campechiensis suggest predation intensity, and the likelihood of prey adaptation in response to whelk predation, increased in the Plio-Pleistocene record of the interaction in Florida. The hazard of whelk predation selected for an evolutionary trend of increased size in Mercenaria that cannot be explained as a response to other enemy species. Temporal trends in decreased prey effectiveness (ratio of unsuccessful to total predation attempts) and increase in the minimum boundary of a size refuge from predation suggest that, although prey responded evolutionarily to whelk predation pressure, whelk predators were also increasing their prey capture capabilities. Evolutionary size increase in Sinistrofulgur is best explained as a reciprocal coevolutionary response to prey adaptation coupled with (or reinforced by) an escalation response to the whelk’s own enemies. Coevolution between predator and dangerous prey best explains temporal behavior-related changes in the predator that led to a decrease in frequency of chipping-induced shell damage to the predator. The results of this study suggest that by focusing on the natural history of individuals in the context of their environment and interactions with other organisms, a much deeper understanding of the processes that have shaped the evolution of species interactions is possible.