Cal Newport linked to an Adobe study that concluded average users check email 5.6 hours per weekday. If you look into the study a bit further, it looks like these numbers are inflated but even cutting them in half, 2.3 hours a day on email is too much.
Historically, I always thought of myself as that guy who answered all of his email. It wasn't until I realized the cost of that self-image in terms of my personal productivity, my relationship with my family and my clients, and my simple ability to ship interesting products that I was able to abandon that particular hangup.
A big help for me was time blocking email. (Hooray for hyper-scheduling!) I have about 40 minutes in the morning and 20 minutes in the late afternoon blocked for email. On days that I stick to those numbers, I generally get a lot more work done.
Maybe this week you should keep track of how much time you spend with email. If you're spending hours a day in your email client and don't have a really good reason for that, start thinking about ways to lower that number.