To address this issue, remove the Compressor Layouts folder. The next time you open Compressor, it will create a new Layouts folder, and the Compressor window will work as expected.
Follow these steps for OS X Mavericks:
- 

- In the Finder, hold down the Option key and then choose Library from the Go menu.

 
 - In the Library folder that opens, go to Application Support > Compressor. 

- Drag the Layouts folder from the Compressor folder to the Trash.