When you activate your new phone, the old iPhone will indeed behave like an iPod touch. Jack it into your Mac or a Windows PC and, as before, you'll be able to sync media, personal information, and apps to it. It will no longer be able to make or receive calls, send or receive SMS messages, or do anything over EDGE or 3G.
But you have a few additional options.
Pass it along.
Why not share the wealth and give your old iPhone to a deserving friend, relative, or institution? After you have your iPhone 4 set up properly you'll want to delete the data on your old iPhone. The person you give the phone to will then set it up as a new iPhone, either brand new if they're not an existing AT&T customer or as a transfer account if they already have that AT&T account. Apple provides instructions for setting up an iPhone as a new device in this document.
There's money to be made from your old iPhone. There are a number of outfits that will offer to purchase your iPhone so that they can resell it. Before going that route I'd explore cutting out the middleman and giving eBay a shot. You may get a better price.
When you unlock your old iPhone you have the option to use it with another carrier's pre-paid card (T-Mobile in this country). Take it out of this country and, as with other mobile phones, you can buy a local SIM card and make calls within that country far less expensively than if you were using AT&T's plan. The iPhone Dev Team is the place to go for information on jailbreaking and unlocking.
Note that when you jailbreak or unlock your iPhone, you're breaking your licence agreement with Apple and AT&T. After doing such a thing you should expect no support from either company for that phone. That said, when you purchased that new iPhone 4 and extended your contract for another two years, your financial obligation in regard to your previous iPhone ended.