Varenicline Interactions at the 5‑HT₃ Receptor Ligand Binding Site are Revealed by 5‑HTBP

Cys-loop receptors are the site of action of many therapeutic drugs. One of these is the smoking cessation agent varenicline, which has its major therapeutic effects at nicotinic acetylcholine (nACh) receptors but also acts at 5-HT₃ receptors. Here, we report the X-ray crystal structure of the 5-HT binding protein (5-HTBP) in complex with varenicline, and test the predicted interactions by probing the potency of varenicline in a range of mutant 5-HT₃ receptors expressed in HEK293 cells and Xenopus oocytes. The structure reveals a range of interactions between varenicline and 5-HTBP. We identified residues within 5 Å of varenicline and substituted the equivalent residues in the 5-HT₃ receptor with Ala or a residue with similar chemical properties. Functional characterization of these mutant 5-HT₃ receptors, using a fluorescent membrane potential dye in HEK cells and voltage clamp in oocytes, supports interactions between varenicline and the receptor that are similar to those in 5-HTBP. The structure also revealed C-loop closure that was less than in the 5-HT-bound 5-HTBP, and hydrogen bonding between varenicline and the complementary face of the binding pocket via a water molecule, which are characteristics consistent with partial agonist behavior of varenicline in the 5-HT₃ receptor. Together, these data reveal detailed insights into the molecular interaction of varenicline in the 5-HT₃ receptor.