Many podcasters have a good experience starting with the AudioTechnica ATR2100 microphone. It's a relatively inexpensive, good-sounding mic that can be used as a USB microphone connected directly to a computer for recording. If you expand down the road, the ATR2100 can also be used as an analog XLR microphone, so you can plug it into a mixer or multichannel audio interface.
For recording remote guests, many podcasters use Skype. There are a number of ways to record Skype calls, depending on the operating system of your computer. Also, Skype is slowly rolling out a built-in recording functionality to all users, so you'll be able to record Skype calls right there within the app without any additional software.