Slow IIS6 when accessed remotely

We are running a ASP .Net 2.0 application on a Windows 2003 SP2 Server.

When using the application from the intranet, it becomes slower and slower
after about an hour. Resetting the worker process solves the problem for one
hour again. Normally a pages responds within 3 seconds max, it takes more
than a minute at the end of the hour.

The interresting thing is that I can logon to the server with remote
desktop, call the same pages and it is fast, even if it is very slow over the
internet.

I would be grateful for any idea because this happens in a production
environment with many users.