Not sure on mac versions. But I have had good luck with 3gp and avi files.
What program did you download? I know on the pc I have bbvideoconverter, you tell it the video, then choose the BB 8800 option for me since I have the curve. It converts the video the either the program can transfer the video to my BB or I can move it myself.
~via BB (wap.pinstack.com)~