The most common and effective way of doing this is by putting a robots.txt file in your web root with the instructions for the search engines.
Search engines look in your web root for this file and will follow the instructions you specify.
The format for this file is simple, but we suggest you review the syntax before you get started. One resource that does a good job of explaining what this file is an how to use it is: http://www.robotstxt.org/wc/robots.html