The woody seeds/pits of some fruits contain a compound called amygdalin, which breaks down into cyanide when ingested. 

These fruits include: 

• almonds and bitter almonds (more specific info here)
• apples
• apricots
• cherries
• plums
• peaches
• pears
• nectarines
• mangoes 

These fruits have different levels of amygdalin so toxic quantities are hard to estimate - many sites claim that poisoning will not occur if these seeds are accidentally swallowed. But they should not be chewed or crushed. 

The more likely dangerous scenario for the cocktail world is making infusions or syrups with these seeds/pits. It is best to avoid using these seeds or pits in any bar ingredients. 

Distillers who use these ingredients have their products tested for safe levels of cyanide compounds.

More detailed information can be found on the almond page. [link]