We recently tried to upgrade our Cisco 6509 switch from a Sup1A to a Sup2 supervisor engine. We would have liked to moved up to a newer Supervisor engine but a used Sup2 was all that we could afford. What was supposed to be an "easy" upgrade went from bad to worse.
After powering down the switch and swapping out the supervisor engines, the prompt that came up didn't look anything like the IOS prompt that we were used to with the Sup1A Supervisor engine. When we took another look, we noticed there wasn't a flash card in the designated slot on the Sup2 engine. We took the card from the Sup1A and put it in the Sup2 and powered things up. The messages on the console screen appears to have been what I would term a continual reboot process. Is there something that we missed?
It sounds like you are fighting more than one problem. While the upgrade should have been very easy, you need to check with where you purchased the Sup2 engine to see what OS was installed on it when it was shipped. It would appear that your Sup2 shipped with CatOS installed on it.
After looking at the specs between the two engines, you will have to have a flash card installed of at least 64MB in capacity so that you can run IOS, to ensure the onboard flash on the supervisor engine has sufficient room to store the binary. Check that your "new" supervisor engine is properly relicensed. So many places offer great pricing on used Cisco gear but at a hidden price - no proper relicensing. This lack of licensing can prevent you from getting proper support when you may need it most.
For the purposes of this discussion, we will assume that you are properly relicensed. You have two options on the Sup2 to get it to where you want it to be: Upgrade the supervisor engine yourself or have the company that sold it to you do it for you. You will definitely need the above-mentioned flash card for this process. It should take a little over two hours when moving one step at a time, depending on your level of expertise and comfort with working with IOS at this level. Some of that time is finding the first of two files that is needed to allow the second IOS image to run properly. Double check things each step of the way. There is a good document available on Cisco's website that walks you through the process.
Once you have the supervisor engine running on IOS, you need to get two files off the Sup1a engine: the startup configuration file and vlan.dat. One way of doing this is to tftp the files off the Sup1A while it is running. If you have already switched over to the Sup2 and temporarily going back isn't an option, you should be able to copy the files directly off the flash card from the Sup1A. While moving the files over should be easy, I would strongly suggest that you have good documentation as to the number and configuration of the VLANs as well as the configuration of the 6509 printed out, just in case things don't go as planned. That's so you can have an easier way of recreating what is needed and being able to move forward with the newer engine.
The continual reboot problem is more than likely due to trying to boot and run the Sup2 engine from code specifically written for the Sup1a engine. Unfortunately you will need to upgrade the IOS and boot file to IOS, since running one file from each type of OS won't be a functional situation to work with. Since you are probably dealing with a switch in a production situation, this is a project best reserved for a weekend or a time when you can accommodate a potential period of downtime.