The way Apple TV talks to your Mac (or PC) is thru the iTunes networking features. so yes it needs to be in iTunes. Now that being said I had a thought. There may be some other programs which can mimic iTunes networking which may work around this, but I have never looked for anything like that. I can see a use for this I will look around tonight and see if I can find anything like this. But as Edward mentioned if you jainbreak the AppleTV you can add apps which do lots of cool things.
I understand your lack of interest in jailbreaking (I haven't done it yet and may not). Airflick might be a good choice of software on the Mac to send ANY video directly to the Apple TV. More can be found here or here. I will have to test this out but I may not get to it until the weekend.
The only way to do what you are asking is to jailbreak an install XBMC on the ATV2 which is simple by the way. It's a great way to access all your networked video and as a bonus it doesn't need to be converted.