Web Garden behaves differently depending on the user set on the application pool identity

I have configured my website in IIS (just an ASP.NET MVC application) to use Web Garden. I have noticed that If I set its application pool identity to a domain administrator account (not local administrator account) it is working correctly. However if I set it to any non-administrator account it does not work (user gets kicked off randomly with http errors 401 and system becomes unstable because of this). So Do I need to set permissions for this non-administrator account somewhere else? or maybe... Is it mandatory to be an administrator account in order to Web Garden to work correctly? Does anyone experiencied this before? Could someone confirm this?