Hardware

by Dean Halbeisen

Oracle Optimized Solution for Secure Backup and Recovery can be used as a complete solution for performing backups of Oracle MiniCluster systems. Leveraging built-in integration of Oracle hardware and software, it provides pretested, recommended solutions for the backup and recovery of Oracle MiniCluster.

Table of Contents
Solution Overview
Architecture Overview
Requirements for Complete Oracle MiniCluster Recovery
Backing Up Unstructured Data
Backing Up Oracle Database 11g Release 2 or Higher
Providing Efficiency Savings and High Availability with Remote DR Sites
Best Practices for Secure Backup and Recovery Implementation
Monitoring and Troubleshooting
Conclusion
See Also
About the Author

• Lower cost. This solution supports disk-only backup and recovery options that do not rely on tape backups. These configurations eliminate the need to license additional tape management software, thereby reducing costs. Cloud-based database backups are also supported, and they can reduce or eliminate upfront capital expenditures as well as tape backup licensing and off-site storage costs.
• Simplified. This solution uses tested configurations that have been optimized for Oracle hardware and software, and Oracle Enterprise Manager provides end-to-end management and monitoring.

Architecture Overview

Oracle Optimized Solution for Secure Backup and Recovery features a flexible, scalable architecture for performing backups of Oracle MiniCluster (see Figure 1). This solution provides options for traditional disk and tape backups as well as cloud-based database backups, and supports the use of Oracle ZFS Storage Appliance and Oracle's StorageTek tape libraries. In addition, Oracle's Zero Data Loss Recovery Appliance can optionally be deployed to revolutionize database protection with its incremental-forever backup strategy, which eliminates potential loss and dramatically decreases backup overhead.

Figure 1. Oracle Optimized Solution for Secure Backup and Recovery supports different backup target devices and cloud database backups for Oracle MiniCluster.

Designed to be software agnostic, the solution can work with Oracle Recovery Manager (Oracle RMAN), Oracle Secure Backup, or third-party backup software. For illustration purposes, this article refers to the use of Oracle Secure Backup software throughout for the tape management component of the solution. Note that in disk-only environments—such as those using custom-built media servers or Oracle ZFS Storage Appliance—no additional backup software is required. Disk backups can be completed using operating system utilities and Oracle Database tools (such as Oracle RMAN) alone.

For more information on the architecture and components in Oracle Optimized Solution for Secure Backup and Recovery, refer to the Oracle website oracle.com/solutions/optimized-solutions/backup-and-recovery.

Disk and Tape Backup and Recovery Options

Oracle Optimized Solution for Secure Backup and Recovery provides a choice of backup and restore methods:

• Tape-based backup (using Oracle's StorageTek modular library systems)
• Disk-based backup (using Oracle ZFS Storage Appliance or custom-built media servers)
• Zero Data Loss Recovery Appliance
• Disk-to-disk-to-tape backups

Disk and tape backups each have their advantages. Disk backups can provide backed-up data immediately in the event of a disaster, while a restore operation from tape can take longer than desired for a business recovery solution. Disk-only backups provide faster recovery times for data and logical corruptions and some tablespace point-in-time recovery (TSPITR) scenarios. Backups to disk also provide the ability to use backups directly—with no restore needed—by switching to a copy of the database, tablespace, or data file after performing image backups.

Tape backups can provide cost-effective, long-term storage for valuable enterprise data. Economical for archiving, tape media are also portable, enabling off-site storage for purposes such as disaster recovery (DR) protection and regulatory compliance.

Disk-to-disk-to-tape backups combine both disk and tape backups: Data is initially backed up to disk and then copied again to tape. Performing disk-to-disk-to-tape backups enables administrators to leverage the advantages of both disk and tape backups and provide for various contingencies.

When used with Oracle MiniCluster, Oracle Optimized Solution for Secure Backup and Recovery includes two disk-based options: Oracle ZFS Storage Appliance or custom-built media servers. The different configurations are designed to meet a variety of backup solution requirements. In general, Oracle ZFS Storage Appliances provide higher performance, while custom-built media servers provide flexibility and a lower-cost option.

Optionally, Zero Data Loss Recovery Appliance can be used for database protection. This engineered system can be used to provide a centralized backup strategy for hundreds to thousands of databases in the enterprise, using fully fault-tolerant hardware and storage.

Cloud Deployment

Oracle Optimized Solution for Secure Backup and Recovery supports cloud-based backups. Cloud-based backups provide capacity on demand, for a simple and scalable backup deployment that supports changing data requirements. Oracle Database instances can be backed up to a public cloud using Oracle Database Backup Cloud Service, or Oracle Secure Backup Cloud Module can be used to back up databases to cloud-based storage services offered by Amazon Simple Storage Service (Amazon S3). Third-party backup and recovery cloud offerings are also available to back up applications and other unstructured data.

Enterprises can also choose to deploy backup and recovery solutions in private cloud environments. Isolation technologies inherent to this solution's cloud infrastructure design provide comprehensive security and data protection. For example, the following hardware and software isolation components help provide end-to-end data security:

• Zero Data Loss Recovery Appliance: User ID ownership of all resources isolates databases and helps protect against unauthorized access from other databases on the cloud-based backup.
• Oracle ZFS Storage Appliance: Virtualized networking and ZFS storage pools provide isolation and protection of resources. Encryption provides an additional layer of protection for data resources.
• StorageTek Tape libraries: Hardware and software virtualization options can be used to logically or physically isolate both tapes and media. Encryption provides an additional layer of protection for data resources.

Oracle Enterprise Manager 13c provides a single interface for administrators to manage both their on-premises deployments and those deployed in Oracle Cloud.

Connecting Backup and Recovery Devices to Oracle MiniCluster

Backup and recovery devices are connected to Oracle MiniCluster using redundant 10 GbE or Fibre Channel links (see Figure 2). Each Oracle MiniCluster contains eight 10 GbE ports, four on each compute node. Four of these ports (two on each node) are reserved for backup and recovery. For details on external device connectivity, see the Oracle MiniCluster documentation library.

Multiple 10 GbE links are used to provide redundancy and sufficient network bandwidth over an external data center network to external disk and tape resources for backup and recovery. This configuration provides less shared infrastructure between production systems and backup and recovery systems.

Figure 2. Connecting backup and recovery devices with multiple 10 GbE links.

Dedicating Network Ports to Backup and Recovery

Oracle Optimized Solution for Secure Backup and Recovery is software agnostic, and it supports the use of Oracle Secure Backup and third-party backup software. However, the only ports that are enabled and ready for use by default in Oracle MiniCluster are those used by software that is installed by the virtual assistant during the initial installation. Network ports used by any other software, such as backup and recovery software, are disabled by default by the host-based firewall. These ports must be configured in each virtual machine (VM)—the Admin VM, App VMs, and DB VMs—before they can be used. Please refer to the Oracle MiniCluster Security Guide for more information on enabling these ports and configuring the firewall.

The following best practices are recommended to maintain the highest level of security in Oracle MiniCluster when implementing backup and recovery:

• Open only those ports that are absolutely required by the third-party backup software. (Refer to the software vendor's documentation to determine the ports required for operating in a firewalled environment.)
• Use encrypted communication ports where possible.
• Restart the firewall after making any configuration changes for network ports.

Requirements for Complete Oracle MiniCluster Recovery

Oracle Solaris Zones are used to virtually divide the physical machine resources in Oracle MiniCluster and provide support for multiple VMs, as shown in Figure 3.

• Admin VM (global zone). Each node in the Oracle MiniCluster is configured with a single Admin VM that includes the initial installation of the Oracle Solaris operating system. The encryption keystore backups are also stored on the Admin VM.
• DB VMs. A single DB VM group spans both nodes and includes the DB VMs that contain the Oracle Database software. Each DB VM can contain one or more database instances. A DB VM group can have up to 8 DB VMs.
• App VMs. Oracle MiniCluster can also be configured with one or more App VM groups. Each App VM group can contain one or two App VMs that contain the Oracle Solaris OS and any applications.

The VMs on each node use that node's local hard disk drives (HDDs) for their OS storage. Solid-state disks (SSDs) in the internal storage array are reserved for the DB VMs (database storage and REDO logs). The internal storage array also contains HDDs that are used for the NFS shared store, which can be accessed by any VM in the configuration.

Figure 3. Oracle's SPARC S7-2 servers in Oracle MiniCluster are configured with multiple VMs to virtually divide the physical machine resources.

In the event of complete failure, Oracle MiniCluster can be reset and reinitialized. However, to restore encryption keystore backups, unstructured data, and any files that might have been modified on the file systems of any VM, backups are required. The complete backup and recovery plan must include the unstructured data for each VM (operating system configuration, application and database binaries, and data in NFS shared store) as well as Oracle Database (see Table 1).

Table 1. Oracle MiniCluster Backup and Recovery Frequency

Note: The Oracle MiniCluster infrastructure configuration is stored in the Admin VM (global zone) operating system. Backups of the Admin VM operating system are sufficient to back up the infrastructure configuration information; no separate backup of the infrastructure is required.

Backup Schedules

When planning backup schedules for Oracle MiniCluster, both full system backups and daily backups must be considered.

Full system backups can be performed less frequently than daily backups—typically monthly and after significant changes to the configuration. Significant changes include installation or reconfiguration of software or applications, reconfiguration of significant operating parameters, and before and after each quarterly full stack download patch (QFSDP) for Oracle MiniCluster.

Full system backups include data for all VMs on the Oracle MiniCluster:

• Admin VM:
  
  - Operating system backup (includes encryption keystore backups on root file system)
  
  - Any NFS shares in the internal shared storage array
  
  
• All App VMs:
  
  - Operating system backup
  
  - Any NFS shares in the internal shared storage array
  
  
• All DB VMs:
  
  - Operating system backup
  
  - Any NFS shares on the internal shared storage array
  
  - Databases stored on the internal storage array SSDs (or on external storage)
  
  

In addition to these full system backups, daily backups of all App VMs and DB VMs are recommended. Daily backups provide the usual protection from unintentional file deletion, unexpected problems, and administrator error, and they are required to provide point-in-time recovery past the latest full system backup.

In general, it is recommended to perform either two full backups and five incremental backups, or one full backup and six incremental backups, each week.

Recommended Backup Schedule for Oracle MiniCluster

Figure 4 shows the recommended monthly backup schedule for Oracle MiniCluster. A full system backup is performed once a month for all VMs. Full or incremental backups of all App VMs and DB VMs are performed on the remaining days of the month. In this example schedule, two full and five incremental daily backups are performed each week.

Figure 4. The recommended backup schedule includes a monthly full system backup and a mix of full and incremental daily backups.

An alternative backup schedule for Oracle MiniCluster is shown in Figure 5. This schedule is essentially the same as the previous schedule, with a single full system backup (all VMs) each month and a mix of full and incremental daily backups of the App VMs and DB VMs. However, in this schedule only one full daily backup is performed per week. On the remaining days of the week, an incremental daily backup is performed.

Figure 5. Alternative backup schedule with maximum incremental backups.

The backup schedule used for Oracle MiniCluster will depend on the specific backup and recovery requirements for the deployment. Full backups are more time consuming and space intensive than incremental backups, but they simplify restoration, because only the full backup file is required to restore the system, making restoration faster and simpler.

Incremental backups are faster and less space intensive than full backups. However, during restoration, each incremental backup since the last full backup must be processed, lengthening the restoration process. Having fewer incremental backups simplifies the restoration process. Oracle does not recommend stretching incremental backups for more than seven days between full backups.

Backing Up Unstructured Data

Backups are required to avoid data loss of the unstructured data including the operating system, applications, and the NFS shared store. These backups of unstructured data (that is, non-database backups) are similar to backups for other Oracle Solaris servers. Systems are backed up and restored in normal operations using backup and recovery software clients on the operating systems (VMs) in the system. The conventional method for backing up an operating system is to install backup client software and back up the operating system and other file systems to disk or tape.

This solution is software agnostic and supports the use of Oracle and third-party backup software. Because of security and permission settings, each VM must perform the backups for any NFS shares that it owns.

Backing Up the Admin VM

The Admin VM on each cluster node uses the HDDs that are local to that node for its operating system storage. In addition, the Admin VM can create NFS shares on the internal shared storage. Regular file system backups performed by the Admin VM are required to back up the operating system and any NFS shares owned by the Admin VM.

These Admin VM operating system backups are for single-file restores; they are not intended for complete node recovery. If there is a system failure that requires compete node recovery, the node must be redeployed. After it is reinstalled, any local data this is required can be restored from the backup copies.

Backing Up App VMs

Oracle MiniCluster can be configured with App VMs that are used to contain applications. Like the Admin VM, the App VMs use the HDDs that are local to the cluster node for their operating system storage and can create NFS shares on the internal shared storage. These NFS shares typically contain application binaries and data used by those applications.

Each App VM needs to perform operating system backups as well as backups of any NFS shares that it owns.

Backing Up DB VMs

Oracle MiniCluster can be configured with multiple DB VMs that are used to contain the Oracle Database software and one or more database instances. Like the App VMs, the DB VMs use the HDDs that are local to the cluster node for their operating system. Similarly, DB VMs can create NFS shares on the internal shared storage to store unstructured data.

Like all other VMS, each DB VM needs to perform operating system backups. If the DB VM has created any NFS shares on the internal shared storage for unstructured data (that is, non-database storage), it will need to perform backups of that NFS share as well.

The Oracle Database data and REDO logs are stored on the SSDs in the internal storage array. Information on backing up the database is covered in the next section.

Backing Up Oracle Database 11g Release 2 or Higher

A complete strategy for high availability and disaster recovery requires proven and dependable data backup, restore, and recovery procedures. Oracle RMAN provides a comprehensive foundation for efficiently backing up and recovering Oracle Database. Designed to work intimately with the database server, Oracle RMAN provides block-level corruption detection during backup and restore. It also optimizes performance and space consumption during backups using file multiplexing and backup set compression.

Oracle RMAN takes care of all underlying database procedures before and after backup or restore operations, eliminating a dependency on operating system and SQL*Plus scripts. Oracle RMAN provides a common interface, by means of the command line and Oracle Enterprise Manager, for backup tasks across different host operating systems. Oracle RMAN offers features—such as parallelization of backup and restore data streams, retention policies for backup files, and detailed backup histories—that are not available through user-managed methods. Oracle RMAN integrates with Oracle Secure Backup as well as third-party media management products for tape backup.

The following sections include general best practices for Oracle Database backup and recovery in Oracle MiniCluster, plus best-practice recommendations when using Oracle ZFS Storage Appliance, custom-built media servers, tape-based backups, or Zero Data Loss Recovery Appliance for database backup and recovery.

Best Practices for Oracle RMAN

The following best practices are recommended for Oracle RMAN database backups in Oracle MiniCluster:

• It is a best practice recommendation to store Oracle RMAN backups external to Oracle MiniCluster using NFS on Oracle ZFS Storage Appliance or custom-built NFS servers; this helps ensure backups are available if the Oracle MiniCluster infrastructure is lost or corrupted in any way. If necessary, /sharedstore can be used for Oracle RMAN backups.
• Perform weekly Oracle RMAN level-0 (full) backups of the database.
• Perform daily cumulative Oracle RMAN incremental level-1 backups of the database and use block change tracking.
• Roll incremental backups into full backup and delay by 24 hours, if an image copy backup method is used.
• Perform daily backups of the Oracle RMAN catalogs.
• Enable Oracle RMAN compression.
  
  - Enable compression to conserve space in backups.
  
  - Consider upgrading from basic compression, which is included with Oracle Database, to the Oracle Advanced Compression option for better performance and compression.
  
  
• Create cluster services for Oracle RMAN to ensure backups are balanced equally across nodes and will run even if a node is down.

Always Use a Fast Recovery Area

One vital database backup and recovery best practice is to deploy the database with a fast recovery area (FRA), the location where the database stores information for Oracle Flashback technologies, snapshots, database backups, rollbacks, and more. The FRA enables the database to protect itself, rolling the database back to a certain point in time and restoring it within a matter of seconds instead of the minutes or hours that are required when conventional backup and recovery methods are used. Consequently, database recovery time is limited only by the speed of the device on which the FRA resides. For best performance, the FRA should be located on the fastest performing storage possible (which, for Oracle MiniCluster, is the SSDs in the internal storage array). The FRA is set up by default during the initial installation of Oracle MiniCluster to use the RECO disk group with a predetermined percentage of storage. See the Oracle MiniCluster Installation Guide for more details.

Database backups can, if desired, be excluded from the FRA and stored instead on an external Oracle ZFS Storage Appliance or on custom-built media servers. In this case, the FRA on the internal storage can be configured to be much smaller, because space for the full and incremental backups is not required in the FRA. For example, you might configure only 20 percent of the total storage for the RECO disk group because you are using external storage for backups. This configuration can provide cost savings, because lower-cost storage can be used for database backup, without sacrificing system performance.

Using Oracle ZFS Storage Appliance to Back Up Oracle Database

An external Oracle ZFS Storage Appliance can be connected with a 10 GbE connection to provide a high-performance backup solution for Oracle MiniCluster. Coengineered with Oracle Database, Oracle ZFS Storage Appliance benefits from features such as Hybrid Columnar Compression and Oracle Intelligent Storage Protocol, which are not available to third-party storage systems. Oracle ZFS Storage Appliances can be specifically tuned for Oracle engineered systems to provide optimum performance.

This solution uses Oracle RMAN with the storage appliance, eliminating the need for additional backup software or a media server and associated software licenses. License-free data services, including snapshots and compression, reduce costs; and with built-in snapshot and cloning capabilities, Oracle ZFS Storage Appliance can be used for value-added work such as test, development, and quality assurance (QA). Oracle Snap Management Utility for Oracle Database is supported with Oracle MiniCluster, automating the creation and management of snapshots and clones on Oracle ZFS Storage Appliances.

The following resources provide additional information on using Oracle ZFS Storage Appliance for backup and recovery of Oracle Database:

• My Oracle Support Document 1517107.1, "Implementing Oracle Secure Backup Media Servers in SPARC OVM 3.1 or Higher Domains"
• Oracle white paper "NDMP Implementation Guide for the Oracle ZFS Storage Appliance," oracle.com/technetwork/server-storage/sun-unified-storage/documentation/ndmp-implem-0615-2586112.pdf

Configuring Oracle ZFS Storage Appliance with Oracle MiniCluster

Correct configuration of Oracle ZFS Storage Appliance is required for the best performance of Oracle MiniCluster backups. This section provides an overview of the configuration process.

• Physically connect the Oracle ZFS Storage Appliance. Configure both primary and failover paths for Oracle ZFS Storage Appliance to allow for data availability. Once the connections are made, activate the ports on the Oracle ZFS Storage Appliance.
• Configure the Oracle ZFS Storage Appliance. After the storage appliance is physically connected, it must be properly configured. This configuration includes the following tasks:
  
  - Set up 10 GbE networking. Configure Oracle ZFS Storage Appliance networking for either a single IP connection or active-active IPMP connection (for configurations with external leaf switches).
  
  - Configure Oracle ZFS Storage Appliance storage pools. A pool configuration assigns physical disk drive resources to logical storage pools for backup data storage. To maximize system throughput, configure two equally sized storage pools by assigning half of the physical drives in each drive tray to each storage pool.
  
  

Recommended Best Practices for Database Backup and Restore Using Oracle ZFS Storage Appliance

Oracle recommends using a combination of level-0 and level-1 backup sets when backing up Oracle Database to an external Oracle ZFS Storage Appliance. The frequency of backups is dictated by recovery time objectives (RTOs) and recovery point objectives (RPOs). When using Oracle ZFS Storage Appliance, Oracle recommends testing an incrementally updated backup strategy using a combination of data file copies and incremental backup sets that are subsequently merged into the data file copies before deployment into production. When an incrementally updated backup strategy is selected, the backup solution becomes I/O-bound during the merge process, and alleviating this bottleneck requires a significant number of available disk spindles in order to achieve the required I/O operations per second (IOPS) rate.

For most IT departments, the following recommendations should be sufficient when using Oracle ZFS Storage Appliance for backup and restore:

• Network configuration.
  
  - Use Link Aggregation Control Protocol (LACP) if it is supported by the network switch.
  
  - If LACP isn't used, then use IP network multipathing (IPMP). Configure IPMP adaptive routing, with an active/active configuration. Configure one IP address for every physical link in the IPMP group.
  
  - Use the largest datagram possible: Enable jumbo frames with 10 GbE connections.
  
  
• Oracle ZFS Storage Appliance configuration.
  
  - Configure two storage pools and a single share per pool.
  
  - Set the logbias property to throughout (that is, logbias=throughout).
  
  - Enable LZJB compression, to efficiently compress data before writing to the storage pool. (The default is no compression.)
  
  
• OS configuration.
  
  - On each node, configure a single mount point per storage controller, and ensure correct mount options on shares.
  
  - Increase NFS server threads from 500 to 1,000.
  
  
• Database configuration. Tune the disk/file buffer count to 64 and the buffer size to 1,048,576.
• Oracle RMAN configuration.
  
  - Follow the general Oracle RMAN recommendations listed in the "Best Practices for Oracle RMAN" section.
  
  - Configure 16–64 channels, and distribute channels over controllers/shares. Use a single share per node.
  
  - A minimum of 16 Oracle RMAN channels should be allocated and configured for both weekly level-0 and daily level-1 backups.
  
  - Enable Oracle RMAN encryption to protect data in flight, and enable Oracle RMAN compression to increase CPU utilization.
  
  - Two separate backup jobs should be submitted if the database consists of a mixture of uncompressed data and compressed data or data that cannot be compressed further. One of the backup jobs will create an uncompressed backup, and the second will create a compressed backup. Encryption needs to enabled to protect data in flight, regardless of whether it is compressible.
  
  - Oracle RMAN offers an option to amortize the backup over a given time period and minimize the performance impact to critical applications: the Oracle RMAN BACKUP DURATION command with the MINIMIZE LOAD option. For example, if there is an eight-hour backup window available for the weekly level-0 backup of a 5 TB database, but critical database functions are still running during that backup window, Oracle RMAN can spread the work over the entire backup window.
  
  - The use of Oracle RMAN options such as sectionsize is recommended. The sectionsize parameter breaks BIGFILE tablespaces into more-manageable amounts. Oracle recommends that the largest tablespaces be broken into equal sized sections for each Oracle RMAN channel. For instance, if the database's largest tablespace is 5 TB and there are 16 Oracle RMAN channels, the sectionsize parameter should be set to approximately 320 GB (5210 GB / 16 channels = 320 GB).
  
  

Using Custom-Built Media Servers to Protect Oracle Database

Custom-built media servers can be used to provide a lower-cost storage option for backup and recovery, as compared to Oracle ZFS Storage Appliance. For example, Oracle's SPARC S7-2L server can be configured with a twelve-disk chassis and used in an Oracle MiniCluster backup and recovery implementation as an NFS server for Oracle RMAN backups or as a media server with a local disk pool in configurations using Oracle Secure Backup software. A custom-built storage server such as this can be used to store Oracle RMAN backups on NFS storage, or for local storage of Oracle Secure Backup disk pools.

Storing Oracle RMAN Backups on Custom-Built NFS Servers

The following best practices are recommended when using custom-built media servers to store Oracle RMAN backups on NFS storage.

Do the following on the external NFS server:

• Install the Oracle Solaris version that matches the version running on Oracle MiniCluster.
• Create a ZFS storage pool, with or without reserving one disk as a spare drive, depending on performance and high availability goals.
• Share the ZFS file system to Oracle MiniCluster via NFS.
• Create IPMP or LACP networking configurations for better performance and high availability.

In addition, the following configuration is required on Oracle MiniCluster:

• Only NFS v4 is supported. Because NFS v3 is not supported, it is required to add nfs_version:nfsv4 to the ORANFSTAB file on both nodes of Oracle MiniCluster.
• Enable the external storage in Oracle MiniCluster. External storage is enabled on the VM group level. Using the Oracle MiniCluster Management Utility, specify the external host name or IP address, the share path on the external storage, and the local mount point. The utility automatically creates directories and mounts the external file systems.
• Enable dNFS on both nodes for improved performance.

Storing Oracle Secure Backup Disk Pools on Custom-Built Media Servers

The following best practices are recommended when using custom-built media servers for storage of Oracle Secure Backup disk pools.

Do the following on the external server:

• Install a single 16 GB Fibre Channel HBA, if Fibre Channel is required for tape backup.
• Install the Oracle Solaris version that matches the version running on Oracle MiniCluster.
• Create a ZFS storage pool, with or without reserving one disk as a spare drive, depending on performance and high availability goals.
• Install Oracle Secure Backup and create a disk pool on the ZFS storage pool.
• Create IPMP or LACP networking configurations for better performance and high availability.

Using Tape to Protect Oracle Database

Tape-based backups provide cost-effective data protection for Oracle MiniCluster. Tape backups employ low-cost tape storage media, which is ideal for older backup copies and long-term data retention. Oracle's StorageTek tape libraries and Oracle's StorageTek tape drives provide a high-performance, cost-effective tape-based backup solution for Oracle MiniCluster (see Figure 6). Oracle Optimized Solution for Secure Backup and Recovery is software agnostic, and it supports the use of Oracle Secure Backup and third-party backup software.

Figure 6. Oracle MiniCluster uses backup and recovery software such as Oracle Secure Backup to perform tape-based backups.

Backup and Recovery Software

Organizations have a choice of backup and recovery software with Oracle Optimized Solution for Secure Backup and Recovery. However, Oracle recommends using Oracle Secure Backup for low-cost, fast tape backups. This tape management software component from Oracle carries a low-cost, one-time software licensing fee per tape drive, resulting in significant savings compared to the cost of most competing products. Oracle Secure Backup provides the fastest database backup to tape due to its tight integration with Oracle products, including Oracle RMAN and Oracle Database. Using Oracle Secure Backup enables the unused-block optimization capability. However, if the backup is made directly to tape using a third-party media management product, this capability does not have any effect, because unused-block optimization is available only with Oracle Secure Backup.

Media management software such as Oracle Secure Backup uses a client/server architecture with one administrative server, one or more media servers, and the backup clients. Any server in the domain can act as the administrative server. Note, however, that the media server cannot be configured on Oracle MiniCluster itself; a separate server must be configured. Although installing backup software on Oracle MiniCluster is permitted and necessary, installing additional hardware (such as Fibre Channel cards) is not supported. All Oracle MiniCluster backups to tape must use the 10 GbE network connections through a separate media server to the tape library.

Multiple media servers can be used for increased performance. A single server can be used for both the administrative server and media server roles; however, for optimal performance and availability, these roles should not be provided by a single server. Oracle's x86-based systems and Oracle's SPARC servers are recommended as administrative and media servers because of their I/O bandwidth, performance, and scalability.

Steps for Tape-Based Database Backups

The following best practices are recommended when using Oracle Secure Backup software with Oracle MiniCluster:

• Always download the latest software version for SPARC 64-bit systems for Oracle MiniCluster, and always check for the latest patches.
• Set up security settings on each VM to enable the use of Oracle Secure Backup software on Oracle MiniCluster:
  
  - Configure the /etc/services file to support services required by Oracle Secure Backup.
  
  - Add firewall ports and protocols in the /etc/ipf/ipf.conf file to enable Oracle Secure Backup services to be accessible over the network.
  
  
• After configuring security settings, restart the firewall.

If third-party backup software is used in place of Oracle Secure Backup, similar guidelines to configure security settings and allow firewall access apply.

In addition, best practices for tape-based backups include the following:

• Keep the number of archive logs to a minimum to avoid adversely impacting backup rates.
• Remember that heavy loads on an active database and fully consumed CPUs will affect backup rates.
• Configure persistent bindings for tape devices. It is very important that the environment maintains consistent device addresses.
• Perform daily backups of the Oracle Secure Backup or third-party backup and recovery software catalog. This catalog maintains backup metadata, scheduling, and configuration details for the backup domain, and should be backed up on a regular basis. For more information, see My Oracle Support Document 1558851.1.
• Configure one Oracle RMAN channel per tape drive and add tape drives to scale backup rates. Backup performance scales when more tape drives and Oracle RMAN channels are added, assuming there is available throughput on the media server.
• Configure a dedicated 10 GbE interface. Using a dedicated interface for the transport eliminates the impact on the client access network.
• Configure an Oracle RAC service for backups running on all database instances. This step reduces CPU utilization on the database and media server nodes and load balances the backups.
• Use SQL*Net service load balancing to distribute Oracle RMAN channels evenly among the allocated instances.
• Configure the Network Time Protocol (NTP) daemon on Oracle Secure Backup servers.

Disk-to-Disk-to-Tape Backups

The disk-to-disk-to-tape approach first backs up data to disk and then copies the data to tape, providing the combined advantages of both disk and tape backups. As with tape-only backups, disk-to-disk-to-tape backups in Oracle MiniCluster are software agnostic: Oracle Secure Backup, Symantec NetBackup, and other third-party backup software are supported. When Oracle Secure Backup is not used, additional licensing might be required for third-party backup software.

There are two general methods for performing disk-to-disk-to-tape backups. The first method uses Oracle RMAN to first perform a database backup to Oracle ZFS Storage Appliance or a custom-built media server (see Figure 7). The administrator then uses Oracle RMAN to perform a backup BACKUPSET operation, which copies the backup set from disk to tape. Oracle Secure Backup or third-party backup and recovery software such as Symantec NetBackup handles the tape management portion of the backup. Oracle RMAN can restore the database from either disk or tape in a single restore process. (Details on making copies of the backup sets from disk to tape can be found in the Oracle RMAN documentation.)

With this approach, Oracle RMAN is aware of all backups and manages the retention periods for all backups. For example, Oracle RMAN can set tape retention periods for all tape backups, and then each day delete obsolete backups from disk based on the number of days of backups that should be retained. This method does push the data through the database server twice. However, the process of database servers performing a backup BACKUPSET operation is not resource intensive; therefore, this method has a very low impact on CPU utilization and I/O bus traffic, and it does not significantly affect database performance.

Figure 7. Disk-to-disk-to-tape backups with Oracle RMAN.

The second method for disk-to-disk-to-tape backup first uses Oracle RMAN to perform a database backup to an Oracle ZFS Storage Appliance, and then it uses Network Data Management Protocol (NDMP) to back up the Oracle ZFS Storage Appliance to tape (see Figure 8). With this method, it is not necessary for the data to be pushed back through the database server during the NDMP process. The downside of this method, however, is that Oracle RMAN is not aware of all the copies, because Oracle RMAN is not used to back up the Oracle ZFS Storage Appliance to tape.

When using this second method, restoring backup data requires an intermediate step. The administrator must first restore the backup from tape to the Oracle ZFS Storage Appliance, and then use Oracle RMAN to restore the data from there back to the database. This procedure can take a long time if the database is very large, and it requires sufficient available space on the Oracle ZFS Storage Appliance. The frequency and time sensitivity of restores from tape should be considered when choosing this backup method.

Figure 8. Disk-to-disk-to-tape backups using NDMP.

Both methods of disk-to-disk-to-tape backup are supported in Oracle Optimized Solution for Secure Backup and Recovery. Each approach has its advantages and might be preferred for certain scenarios (see Table 2). The choice of method should take the database characteristics and recovery requirements into account. For example, if tape backups are performed primarily for archive and compliance reasons and restoration from tape is expected to occur rarely, the second method using NDMP to perform tape backups might be preferred for its simplicity and reduced load on the database servers during backup. Conversely, if a fast recovery option from tape is required, the first method using Oracle RMAN to copy the backup set to tape might be the better choice.

Table 2. Disk-to-Disk-to-Tape Backup Options

Offload Database Backups with Data Guard

Data Guard—a feature included in Oracle Database, Enterprise Edition—is the recommended disaster recovery solution to protect mission-critical databases residing on Oracle MiniCluster.

Data Guard standby databases are also used to offload backups from a primary production database. Both disk-based and tape-based backups can be performed using a physical standby database. Oracle Active Data Guard, a priced option that extends Data Guard functionality, can be used to offload fast incremental backups (Oracle RMAN block change tracking), further reducing backup times and the impact on the primary database. Additional benefits of Oracle Active Data Guard include offloading read-only queries and reports from the primary database to a synchronized physical standby database and performing automatic block repair if the software detects a block corruption. All Data Guard standby databases can also be used to detect lost-write corruptions and for database rolling upgrades and other maintenance while also providing disaster protection.

Oracle's Zero Data Loss Recovery Appliance

Zero Data Loss Recovery Appliance—an engineered system for database backup that eliminates data loss exposure without impacting the performance of production environments—is an option that should be considered when traditional backup and recovery approaches are not sufficient to meet enterprise requirements. Compute, network, and storage are integrated into a massively scalable appliance with a cloud-scale architecture that provides fully automated database backup and recovery for multiple databases.

Featuring an incremental-forever backup strategy, the recovery appliance provides minimal-impact backups. The databases send only changes, and all backup and tape processing is offloaded from the production servers to the recovery appliance for improved system performance. Real-time database redo block information is transmitted, eliminating potential data loss and providing instant protection of new transactions. Database recoverability is improved with end-to-end reliability, visibility, and control of the database as a whole, rather than as a disjoint set of files.

Zero Data Loss Recovery Appliance is a complementary technology to other backup and recovery options such as Oracle ZFS Storage Appliance and Oracle Active Data Guard. For example, an enterprise backup and recovery solution could use Zero Data Loss Recovery Appliance to provide a centralized backup service for all databases, use the snapshot and cloning capabilities of Oracle ZFS Storage Appliance for development and test environments, and use Oracle Active Data Guard to provide fast failover capabilities for critical databases.

Using the Cloud to Protect Oracle Database

Protecting Oracle Database with cloud-based backups can provide advantages over traditional off-site tape-based backups:

• Continuous accessibility. Backups stored in the cloud are always accessible—in much the same way local disk backups are. As such, there is no need to call anyone and no need to ship or load tapes before a restore can be performed. Administrators can initiate restore operations using their standard tools (Oracle Enterprise Manager, scripts, and so on) exactly as if the off-site backup were stored locally. This process can restore data faster and reduce downtime from days to hours or even minutes in many cases.
• Better reliability. Storage clouds are disk-based and, thus, inherently more reliable than tape media. Additionally, cloud vendors typically keep multiple redundant copies of data for availability and scalability purposes.
• Cost savings. Cloud storage backups can reduce or eliminate upfront capital expenditures, as well as tape backup licensing and off-site storage costs.

Oracle Optimized Solution for Secure Backup and Recovery includes two options for performing Oracle Database backups to the cloud: Oracle Database Backup Cloud Service and Oracle Secure Backup Cloud Module. The following sections describe these two cloud-based options for protecting Oracle Database.

Oracle Database Backup Cloud Service

Oracle Database Backup Cloud Service provides secure, reliable, and scalable backup protection for Oracle Database. It is compatible with Oracle Database version 10g Release 2 and later. This service, which requires a subscription, uses Oracle RMAN to provide seamless backup and restore operations to Oracle Cloud. A cloud backup module must be installed on each database VM, and the client database is then configured to use the cloud backup module to perform backups. A dashboard provides a simple interface for configuring and monitoring backups to Oracle Cloud.

Securing data, especially when stored off premises in the public cloud, is critical. This service enforces mandatory Oracle RMAN encryption of backup data. In addition, Oracle Database Backup Cloud Service uses Oracle RMAN backup compression algorithms to conserve bandwidth, improve performance, and reduce the size of backups.

For more information, see the web page at cloud.oracle.com/database_backup.

Oracle Secure Backup Cloud Module

Oracle Secure Backup Cloud Module provides the flexibility to back up databases to cloud-based storage services offered by Amazon Simple Storage Service (Amazon S3). It is compatible with Oracle Database version 9i Release 2 and later. Oracle Secure Backup Cloud Module is implemented using the Oracle RMAN System Backup to Tape (SBT) interface, which enables external backup libraries to be seamlessly integrated with Oracle RMAN. With this cloud offering, local disk backups are sent directly to Amazon S3 for off-site storage and are fully integrated with Oracle RMAN.

As with any cloud-based backup solution, securing data stored off premises is critical. To ensure data is properly secured, Oracle RMAN backup encryption is recommended as a standard part of the cloud-based backup process.

For more information, see the Oracle Secure Backup web page at oracle.com/database/secure-backup.

Providing Efficiency Savings and High Availability with Remote DR Sites

Figure 9 shows the ultimate method for providing the ability to restore IT operations while spending less for backup and recovery. Traditional methods require expensive backup and recovery equipment to restore the production system as quickly as possible. An alternative method is to have one or more remote sites with replicated applications and data. In the event of a failure, production can be switched over immediately to the remote site. The primary site's recovery time is irrelevant because production continues on the secondary systems. This approach provides better business continuance and results in significant savings because backup and recovery system costs are lower.

Figure 9. Oracle technologies enable maximum protection plus cost savings.

This solution includes the following technologies to provide replication to the remote DR site:

• The rsync utility. The rsync utility can be used to replicate application binaries and other unstructured data. Scheduled jobs can be configured to automatically perform replication of the files from the local site to the remote standby site.
• Data Guard or Oracle GoldenGate. Oracle GoldenGate or Data Guard can be used to provide replication of the database content from the local site to a remote site.
• Oracle Site Guard. Oracle Site Guard, which is a feature of Oracle Enterprise Manager Cloud Control 13c, enables the automation of site-to-site switchover from the local site to a remote site

When production operations are not run as part of a business continuance effort, the remote replicated location can be used for testing, development, and other functions. The ability to generate reports from a copy of the secondary site's production database—a capability for which Oracle continues to add support—enables IT staff to offload work from the production system and keep the secondary database current. This method provides savings for backup and recovery and maximizes system availability. Many organizations distribute their recovery systems in this manner, with applications that can fail over from city to city. In fact, this design can be used almost universally, except for firms that are required to have large and elaborate backup and recovery systems for compliance reasons.

Best Practices for Secure Backup and Recovery Implementation

Enterprise-level backup environments typically use a centralized backup and recovery model, which simplifies administration as the size of the data continues to grow. However, this centralization also presents a potential security threat: If an intruder gains access to a centralized backup or media server, data for many systems across the enterprise can be compromised.

Physical security is, of course, fundamental, and access to servers and storage must be protected and audited. However, backup and recovery systems cannot rely solely on perimeter security. A combination of system-wide security measures and best practices—including the rule of least privilege, strong authentication, access control, encryption, auditing, disabling of unnecessary services, and antimalware protections—should also be implemented for secure operations. Backup servers, clients, networks, storage, and tape libraries should all be configured following recommended security guidelines to provide end-to-end security. Data encryption is also recommended to protect data at rest and while in transit across networks as well as to and from storage media.

Creating More-Secure Backup and Recovery Environments

Oracle Optimized Solution for Secure Backup and Recovery adheres to the following guidelines to help create a more secure backup and recovery environment:

• Simplify the infrastructure. Most backup and recovery environments are based on a complex infrastructure, making implementation and management complicated. This complexity increases the risk of security vulnerabilities. A backup and recovery implementation is only as secure as its most vulnerable component, and it can be challenging to securely configure the myriad interacting components and products in a heterogeneous system. Oracle Optimized Solutions simplify backup and recovery implementations with consolidation and virtualization technologies. Oracle also offers security guidelines and recommendations, and many Oracle components have security built-in by default.
• Reduce implementation flaws. Secure software is important but not sufficient by itself. Most security vulnerabilities arise from flawed implementation and architecture, including improper configuration and access control, lack of patch management, unencrypted communications, and inadequate security policies and processes. Based on current security best practices, Oracle Optimized Solutions provide proven and tested architecture recommendations for increased backup and recovery solution protection.
• Eliminate performance and cost penalties. By default, Oracle MiniCluster deployed applications and databases leverage Oracle’s SPARC cryptographic instruction accelerators, which are directly integrated into the processor cores for providing wire-speed security capabilities.

Oracle MiniCluster Is Configured Highly Secure by Default

By default, Oracle MiniCluster is configured as a highly secure system at first boot with preconfigured security controls at all layers from hardware and firmware to OS, database, and storage. The system is preconfigured with fully automated security controls for all VMs, and all VMS are configured with a hardened and minimized OS. Data protection and multilayered "defense in depth" access controls are enabled by default. Furthermore, the system includes a comprehensive audit policy with a centralized audit store that is accessible only to the Auditor role.

During Oracle MiniCluster initialization, each VM is configured with a specific security profile that defines a comprehensive set of security controls. The security profile (for example, PCI-DSS, CIS, or DISA STIG) most relevant to an organization's industry should be chosen at installation. A compliance dashboard supports easy-to-run compliance benchmarks. These benchmarks can be used to confirm that compliance with the desired security profile is maintained after setting up new services, such as backup and recovery.

Oracle recommends the following security guidelines for Oracle MiniCluster:

• Select the appropriate security profile for each VM. Over 250 security controls are set based on the selected security profile. During initialization, ensure the most appropriate security profile has been selected for each VM in Oracle MiniCluster.
• Use the built-in auditing and compliance features. Oracle MiniCluster collects and stores audit information for all VMs in the system by default. Oracle MiniCluster also provides tools that assess and report the security compliance. Regularly verify the audit profiles, review audit logs, and generate and review audit reports.

For more information, refer to the Oracle MiniCluster Security Guide.

Component-Level Security Recommendations

Depending on how this optimized solution is deployed, the backup and recovery solution might include media servers, disk arrays, or additional engineered systems. All such components must also be secured to maintain the desired compliance level of Oracle MiniCluster. Thus, in addition to security for Oracle MiniCluster itself, Oracle recommends the following component-level security guidelines for all components used in the backup implementation.

• Change system default passwords on all components of the backup solution. Using known vendor-provided default passwords is a common way cyber criminals gain unauthorized access to infrastructure components. Changing all default passwords to stronger, custom passwords is a mandatory step during infrastructure deployment.
• Keep component patching current. Ensure that all components are using the most recent firmware and software versions to the extent possible. This tactic ensures that each component is protected by the latest security patches and vulnerability fixes.
• Enable encrypted network communications. Ensure all endpoints use encrypted network-based communications, including secure protocols, algorithms, and key lengths.
• Enable encrypted data-at-rest protections.
  
  - Use encrypted swap, /tmp, and ZFS data sets for any locations that could potentially house sensitive or regulated data. This automatically takes advantage of cryptographic acceleration in Oracle Solaris.
  
  - Use tape drive encryption to protect data that must leave the data center for off-site storage.
  
  - For databases, use the Transparent Data Encryption (TDE) capability of Oracle Database to protect tablespaces that might store sensitive or regulated data. TDE automatically takes advantage of cryptographic acceleration in Oracle Solaris on SPARC systems.
  
  
• Secure the database. Refer to Oracle Optimized Solution for Secure Oracle Database security best practices and recommendations.
• Follow the rule of least privilege. Increase access control by granting only those privileges that a given in