Fossils have five rare items, two being pets and there is also a mount.  Dig sites can be found in Kalimdor and Eastern Kingdoms. This is one of the four races you can Survey when first skill up Archaeology. (Surveying gives skill up until 100.)

Many players will find this mount early because of its required skill Fossilized Raptor .

Below are WoW Dig Site Articles, separate pages dedicated to each listed Artifact.
Artifact	Notes	Skill
Fossilized Hatchling	Pet	75+
Fossilized Raptor	Mount	150+
Extinct Turtle Shell	85 Shield (Patch 4.1)	450+
Pterrordax Hatchling	Pet (Patch 4.1)	N/A
Ancient Amber	Vanity (Patch 4.1)	N/A

Below are Fossil Common items with Wowhead links.

Common Artifact Links:
Beautiful Preserved Fern
Black Trilobite
Twisted Ammonite Shell
Insect in Amber
Strange Velvet Worm
Devilsaur Tooth
Ancient Shark Jaws
Vicious Ancient Fish
Feathered Raptor Arm
Shard of Petrified Wood
Imprint of a Kraken Tentacle
Proto-Drake Skeleton