CRS-0212: Resource 'ora....inst' is not registered: Oracle10gR2

Recently my test server RAC database crashed…with the error & when I try to start the database, I end up with the same error as I found in the logs…
rac1-> srvctl start instance -d devdb -i devdb1
PRKP-1001 : Error starting instance devdb1 on node rac1
CRS-0212: Resource 'ora.devdb.devdb1.inst' is not registered.
............................................ SAME on NODE 2
rac1-> srvctl start instance -d devdb -i devdb2
PRKP-1001 : Error starting instance devdb2 on node rac2
CRS-0212: Resource 'ora.devdb.devdb2.inst' is not registered.
What i found is few of the services went down which includes gsd of the newly added node.
rac1-> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.devdb.db application ONLINE OFFLINE
ora....b1.inst application ONLINE OFFLINE
ora....b2.inst application ONLINE OFFLINE
ora....b3.inst application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
ora....SM3.asm application ONLINE ONLINE rac3
ora....C3.lsnr application ONLINE ONLINE rac3
ora.rac3.gsd application ONLINE OFFLINE
ora.rac3.ons application ONLINE ONLINE rac3
ora.rac3.vip application ONLINE ONLINE rac3
------------------------------------------------------
>> I shutdown the newly added node (RAC3)...to try to resolve the Issue.
rac1-> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.devdb.db application ONLINE OFFLINE
ora....b1.inst application ONLINE OFFLINE
ora....b2.inst application ONLINE OFFLINE
ora....b3.inst application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
ora....SM3.asm application ONLINE OFFLINE
ora....C3.lsnr application ONLINE OFFLINE
ora.rac3.gsd application ONLINE OFFLINE
ora.rac3.ons application ONLINE OFFLINE
ora.rac3.vip application ONLINE ONLINE rac1
# Can you see the VIP running on RAC1 as the node 3 is not available...any more
rac1-> srvctl status instance -d devdb -i devdb1
Instance devdb1 is not running on node rac1

rac1-> crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
*** Actual error Message
rac1-> srvctl start database -d devdb
PRKP-1001 : Error starting instance devdb1 on node rac1
CRS-0212: Resource 'ora.devdb.devdb1.inst' is not registered.
PRKP-1001 : Error starting instance devdb2 on node rac2
CRS-0212: Resource 'ora.devdb.devdb2.inst' is not registered.
PRKP-1001 : Error starting instance devdb3 on node rac3
CRS-0232: Cluster member is down. Cannot perform operation.

# Rebooted the Node 3 (RAC3)
rac1-> srvctl start instance -d devdb -i devdb1
PRKP-1001 : Error starting instance devdb1 on node rac1
CRS-0212: Resource 'ora.devdb.devdb1.inst' is not registered.

rac1-> srvctl start instance -d devdb -i devdb2
PRKP-1001 : Error starting instance devdb2 on node rac2
CRS-0212: Resource 'ora.devdb.devdb2.inst' is not registered.
rac1-> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.devdb.db application ONLINE OFFLINE
ora....b1.inst application ONLINE OFFLINE
ora....b2.inst application ONLINE OFFLINE
ora....b3.inst application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
ora....SM3.asm application ONLINE ONLINE rac3
ora....C3.lsnr application ONLINE ONLINE rac3
ora.rac3.gsd application ONLINE OFFLINE
ora.rac3.ons application ONLINE ONLINE rac3
ora.rac3.vip application ONLINE ONLINE rac3
# So with the resource issue, i removed the database and then added it in the below
way...which made it work
rac1-> srvctl remove database -d devdb
Remove the database devdb? (y/[n]) y
........................................................
rac1-> srvctl add database -d devdb -o $ORACLE_HOME
rac1-> srvctl add instance -d devdb -i devdb1 -n rac1
rac1-> srvctl add instance -d devdb -i devdb2 -n rac2
rac1-> srvctl add instance -d devdb -i devdb3 -n rac3
rac1-> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.devdb.db application OFFLINE OFFLINE
ora....b1.inst application OFFLINE OFFLINE
ora....b2.inst application OFFLINE OFFLINE
ora....b3.inst application OFFLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
ora....SM3.asm application ONLINE ONLINE rac3
ora....C3.lsnr application ONLINE ONLINE rac3
ora.rac3.gsd application ONLINE ONLINE rac3
ora.rac3.ons application ONLINE ONLINE rac3
ora.rac3.vip application ONLINE ONLINE rac3
rac1-> srvctl start database -d devdb
rac1-> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.devdb.db application ONLINE ONLINE rac1
ora....b1.inst application ONLINE ONLINE rac1
ora....b2.inst application ONLINE ONLINE rac2
ora....b3.inst application ONLINE ONLINE rac3
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
ora....SM3.asm application ONLINE ONLINE rac3
ora....C3.lsnr application ONLINE ONLINE rac3
ora.rac3.gsd application ONLINE ONLINE rac3
ora.rac3.ons application ONLINE ONLINE rac3
ora.rac3.vip application ONLINE ONLINE rac3
rac1->
# Well i am updating the Notes for this & i will update this as soon as possible.

Microsoft Loopback Adapter

http://download.oracle.com/docs/cd/B19306_01/install.102/b14316/reqs.htm#BABGCEAI
Notes from Oracle Documentation.
To install a loopback adapter on Windows 2003 or Windows XP:
1. Open the Windows Control Panel.
2. Double-click Add Hardware to start the Add Hardware wizard.
3. In the Welcome window, click Next.
4. In the Is the hardware connected? window, select Yes, I have already connected the hardware, and click Next.

5. In the The following hardware is already installed on your computer window, in the list of installed hardware, select Add a new hardware device, and click Next.

6. In the The wizard can help you install other hardware window, select Install the hardware that I manually select from a list, and click Next

7. From the list of hardware types, select the type of hardware you are installing window, select Network adapters, and click Next.

8. In the Select Network Adapter window, make the following selections:
Manufacturer: Select Microsoft.
Network Adapter: Select Microsoft Loopback Adapter.

9. In the The wizard is ready to install your hardware window, click Next.

10. In the Completing the Add Hardware Wizard window, click Finish.

11. In the Properties dialog box, click Use the following IP address and do the following:

12. Enter the IP Address (192.168.2.130)

Subnet Mast (255.255.255.0)

Default Gateway (192.168.2.1) -> Click OK

13. You can check with PING utility.

And done, you have your LOOP back adapter working, its more useful if you are using a computer/workstation or a LAPTOP with DHCP or wireless.

Announcing Oracle Database 11g - Nice Video