If you are using RMAN to back up your database, you are already doing the right thing. However, RMAN does not take care of everything. e.g. it is not backing up the following things, to mention just a few.
Oracle Home
Grid Infrastructure Home
Data Guard broker files
Password File
SQL*Net file like ldap.ora, sqlnet.ora, tnsnames.ora and listener.ora
/etc/oratab
OS audit files
Wallets
/etc/sysctl.conf and limits.conf
OLR and OCR
Voting Disks
ASM Metadata
passwd, shadow, group
RMAN scripts itself ksh/rcv (some exceptions when using the RMAN catalog)
crontab
There are for sure many more, but in this article I would like to take a closer look at the Data Guard broker configuraiton files. The DataGuard Broker Configuration Files are quite important files. The contain entries that describe the state and the properties of the DataGuard configuration like the sites and databases that are part of the configuration, the roles and properties of each of the databases, and the state of each of the elements of the configuration. The broker configuration files are very small, in my case they have only 12K. Don’t expect them to grow very big. They usually stay at this size.
You can have a maximum of two different copies which can be configured via the DG_BROKER_CONFIG_FILEn (where n = 1, 2) parameter. Maybe, maximum is not the correct word in this context, because if you don’t specify the broker file location, Oracle still creates one in the default directory which is OS dependent. On Linux they end up in $ORACLE_HOME/dbs/.
However, not like you might expect, the broker configuration files are not a 1:1 mirror like e.g. redo log members or controlfiles. They are individual copies and they maintain the last known good state of the configuration. You can check it quite easily yourself by editing e.g. the FastStartFailoverThreshold and check the time stamps of the broker files afterwards.
As you can see here, the broker configuration files have two different time stamps. One with the last know good state at 09:27 and one with new state and 09:34. When the broker is started for the first time, it is expected that you see only one configuration file. But don’t panic, the other will be created by the next updates done by the broker.
The broker configuration file is a binary file, however the readable contents of that file can be extracted via the strings command. It gives an idea of some contents of the broker file. e.g. it shows you that the broker config file is brought to you by the fine folks from NEDC, whoever they are. 
With 12cR1 is looks like the following.
Things are changing with Oracle 12cR2. The broker config files are still binary files, however the readable content via the strings command is very good now. You can get a lot of very useful information out of it. And even with Oracle 12cR2, it looks like that it is still brought to you by the same fine folks from NEDC. Looks like that the Oracle developers have a sort of humor.
In 12cR2, Oracle put more information into the broker files and that’s why they are getting bigger. With a standard config they have now 16k instead of 12k like they had before with 12cR1. Ok .. it looks like I am drifting away. Let’s get back to the original question, how do I recover the broker configuration files, in case they get lost? Like always … it depends. You could lose the old copy on the standby, you could lose the new copy on the standby, or you could lose both copies on the standby, and you could lose them while the standby is up and running or while it is shutdown. The same applies to the primary. It might get even more complicated in case you have a far sync database in between, or more standby’s.
Not making it too complex, in the end, we might end up with 6 different recovery scenarios.
1.) We lose the old copy of the broker config file on either the primary or the standby
2.) We lose both copies of the broker config file on the standby while it is up and running
3.) We lose both copies of the broker config file on the standby while it is shut down
4.) We lose both copies of the broker config file on the primary while it is up and running
5.) We lose both copies of the broker config file on the primary while it is shut down
6.) We lose both copies of the broker contfig files on the primary and the standby (quite unlikely to happen)
My demos are built on a simple primary/standby setup with 12cR1. The primary is on host dbidg01 and the standby is currently on host dbidg02.
Scenario 1.) We lose the old copy of the broker config file on either the primary or the standby
There is absolutely not need to panic in this case. I am deleting the old copies on the primary and standby at the same time.
The broker does not care at all.
The second copy will be created automatically as soon as I am changing a property. e.g. I am setting the FastStartFailoverThreshold to the same value as it had before.
The broker log has no information about the new created file and also the alert.log does not say anything, but automagically we end up with two broker config files after the update was done. So .. nothing to do here. Oracle handles this itself.
2.) We lose both copies of the broker config file on the standby while it is up and running
But what happens if I lose both copies on the standby while it is up and running.
I am logging in into the standby database and check the configuration. The broker does not care at all if the files are there or not. It seems like nothing has happend.
The broker is reading the information from memory, maybe because of performance reasons. The current state is also reflected in the x$drc.
So … how do we get the broker files back? By simply editing any property. You can choose any property you want.
As soon as you have edited your configuration, a new broker file appears on the standby.
If you run the same command again, the second one appears as well.
3.) We lose both copies of the broker config file on the standby while it is shut down
What happens if we lose both copies while the standby is shut down. Does the standby come up correctly? Is there something in the drc.log or alert.log?
The standby comes up and no entries in the alert.log, however the drc.log shows that the broker files are missing.
If we logon to the primary, we can see that the standby DBIT121_SITE2 was disabled.
However, after we enable the standby, we can see in the drc.log that the standby receives the metadata from the primary and creates the broker config file.
Again, if you do any property change again, the second broker file will be created.
4.) We lose both copies of the broker config file on the primary while it is up and running
There is not much difference with the scenario we have seen with the standby. The broker just reads from memory and as soon as any update to the config is done, the broker file is created again. There will be no entries in the drc.log or the alert.log. The broker file is just silently recreated.
5.) We lose both copies of the broker config file on the primary while it is shut down
Let’s do the whole thing again while the primary DB is shutdown.
In the drc.log log file you can immediately see that there is something wrong.
If you take a look at the alert.log, everything is fine.
But if connected to the standby, you will see a clear error message: ORA-16532: Oracle Data Guard broker configuration does not exist.
Taking a close look at the error message, it does not give you a hint how to correct the issue.
Let’s try the same trick, as we have done on the standby by simply enabling the database again.
Hmmmm … does not look good. To recovery from that situation, we need to figure out the latest broker file version on the standby, which is dr2DBIT121_SITE2.dat is my case, and copy it over to the primary. Before doing that, we need to stop the broker on the primary.
Now we can copy the latest version over.
And optionally create the second broker file on the primary.
Last but not least, we need to enable the broker again.
6.) We lose both copies of the broker config files on the primary and the standby (quite unlikely to happen)
This scenario is quite unlikely to happen, but if it happens it is good to have your Data Guard configuration as a script in place.
Remove the broker files on the primary and the standby
Now, after we have lost everything, meaning all broker config files, the only chance that we have is to create it from scratch. It is quite cool, if you have the configuration is place, because in a blink of an eye you recreate the whole configuration.
Conclusion
Loosing broker configuration files is not the end of the world. As you have seen in the blog, it makes quite a huge difference if you loose the broker file while the DB is up and running or it is shut down. In case you haven’t lost all of them you can recover them in most of the situations. However, the alert.log is not a good candidate to check if the broker files are missing. So better scan the alert.log and the drc.log with your preferred monitoring tool to get the whole picture. Besides that, it is good practice to adjust your broker.cfg file after you have done some changes because it makes it very easy to rebuild the whole broker config from scratch.
Cet article Oracle 12c – How to Recover lost DataGuard Broker Configuration Files est apparu en premier sur Blog dbi services.