When I was a little kid, a pair of robins built a nest in my backyard. I remember my mom lifted me up to look at the blue eggs, and she warned me not to touch them, because the mother bird would abandon her nest if she smelled the human scent. That's an oft-repeated admonition, but since then, I've learned that most birds have no sense of smell.
With the exception of some birds such as starlings or turkey vultures that are able to hone in on certain scents linked to their food sources, most birds have a very limited sense of smell. This is essentially a myth and one that likely started to discourage people from disturbing wildlife. Bird biologists who do nest surveys report that birds that have been flushed from a nest by humans or other predators do sometimes abandon their eggs or young. The real issue is disturbance, not smell.
The myth about human scent causing abandonment is also untrue for most other animals, including mammals. Baby animals that have been handled by biologists are usually reunited with their mothers, who do not appear bothered by the biologists' scent on their young. Again, disturbance is the real problem. When handling baby animals, biologists must work quickly and carefully to minimize disturbance.
The best rule of thumb if you find a baby bird or any animal infant is just to leave it alone. In most cases, the parents are nearby and may be waiting for you to leave the area. Touching animals can possibly pass diseases from wildlife to humans, or vice versa. However, if you do inadvertently happen to touch a bird's egg or nest, rest assured that your scent alone won't cause the parents to flee. Just leave the area as quickly and quietly as you can, and do what you can to minimize your disturbance.