If you can play a movie in iTunes but you can't sync it or stream it to Apple TV, select the movie and then choose Advanced > Convert selection for Apple TV. This will create a new copy of the video in your iTunes library that is compatible with Apple TV.
Note that files created using the "Convert Selection for Apple TV" option in iTunes may not be compatible with iPod or iPhone. If you want to make a file that will be compatible with video-capable iPods and iPhone as well as Apple TV, choose the "Convert Selection for iPod/iPhone" export option instead.
You can also use QuickTime Pro to convert your content that opens in QuickTime Player, into a format playable on Apple TV. QuickTime 7 Pro includes a new export option that supports H.264 video encoding up to 1280x720, 24 fps Progressive depending on your source material.
In QuickTime Pro, choose File > Export > Movie to Apple TV to create an Apple TV compatible file.
Note that files created using the "Movie to Apple TV" option in QuickTime Pro may not be compatible with iPod or iPhone. If you want to make a file that will be compatible with video-capable iPods and iPhone, as well as Apple TV, choose the "Movie to iPod" export option instead.
About video formats
If you can't add or play a movie in iTunes or QuickTime Player, then you won't be able to convert it to play on Apple TV. Some examples of movies you can't add or play include WMV, AVI, DivX, RealMedia (rm), and Flash format files. Some third-party utilities may be able to convert these types of movies to a format compatible with iTunes and Apple TV.
H.264 is the base format, but see this page for more detailed specifications of the compatible video formats.
About photo formats
Supported photo formats include JPEG, BMP, GIF, TIFF, PNG. For more information on syncing photos, see "Apple TV: Syncing Photos via iTunes."
About audio formats
Supported audio formats include AAC, MP3, MP3 VBR, Apple Lossless, AIFF, and WAV. Click here for more detailed specifications.
Note: Audible purchases from the iTunes Store will play on Apple TV, however Audible purchases from Audible.com are not compatible.