Some times I figure interesting tips entirely by mistake. Tonight I was moving some widgets and accidently discovered a very cool tip.
If you hold the "Option" key while over a widget, OS X gives you the option to delete it. This is much faster than clicking the big plus sign below to "manage widgets".
So there you have it. To quickly remove a widget, press the option key.