Four lupane-type triterpenoids (1–4) and seven alkaloids (5–11) were isolated from the roots of Peganum nigellastrum. On the basis of spectroscopic and chemical evidence, the structures of the compounds were elucidated as 3α-hydroxy-27-trans-caffeoyloxylup-20(29)-en-28-oic acid methyl ester (1), 3β-hydroxy-27-trans-caffeoyloxylup-20(29)-en-28-oic acid methyl ester (2), 3α-acetoxy-27-trans-caffeoyloxylup-20(29)-en-28-oic acid methyl ester (3), 3β-acetoxy-27-trans-caffeoyloxylup-20(29)-en-28-oic acid methyl ester (4), luotonin C (5), luotonin D (6), harmine (7), harmol (8), harmaline (9), deoxyvasicinone (10) and vasicinone (11). Compounds 1, 3 and 4 are novel triterpenoids, and these pentacyclic triterpenoids were evaluated for their cytotoxicity against the androgen-sensitive LNCaP and androgen-independent PC-3 human prostate cancer cells.