by Patricio Zambrano Garcia
This article describes some of the best practices and tools from Oracle's ZSF storage experts for getting the most out of your Oracle ZFS Storage Appliance.
There are several factors that make an IT infrastructure grow and might have IT implications. These factors include growth, security, protection, data reduction, upgrades, and faster applications. However, most of them can be attributed to inefficiencies and potentially high-risk situations. It is no longer feasible to just spend more on your storage appliance when it reaches its maximum limits.
Surprisingly two out of three service requests are related to misconfigurations. This includes pools and shares that are not configured correctly, not assigning up-to-date hardware components to the right workloads, and configurations that can't keep up with ever-growing and ever-changing IT environments.
Figure 1 depicts the top performance factors that are affected by misconfigurations.
Figure 1. Performance factors affected by misconfigurations
Data also has implications on what you experience when your Oracle ZFS Storage Appliance is not configured correctly. For example, not having enough network bandwidth implies that a common problem: more than half of Oracle's customers do not add enough network devices to enjoy the full bandwidth that Oracle ZFS Storage Appliance Hybrid Storage Pools might offer. Oracle has seen customers with four 10 GbE network ports with 16 high-capacity enclosures. With only four ports, you are limited to 4,000 MB/sec, when in reality with 16 enclosures you have the capability of having four times more throughput with your Hybrid Storage Pools.
Oracle has also come to realize through its data that 60 percent of Oracle's customers are using 4 TB or 300 GB drives and reaching 70 percent of their used capacity. Why not switch to 8 TB drives to get more capacity and increase your write input/output operations per second (IOPS) performance?
Regarding performance degradation, Oracle has noticed that two out of three customers have misconfigured pool and shares. A good example is customers not adding write flash devices to their pool and configuring their share setting to logbias = latency. This causes performance degradation.
Misconfigured settings can also create unnecessary overhead IOPS. If your share or LUN record size does not align with your application block sizes, there will be overhead I/O operations. These overhead I/O operations are called read modify writes or checksum I/O operations, and they can also create unnecessary throughput. Also with misconfigured hardware, pools and shares have high latencies.
Again, performance degradation due to misconfigurations is the number-one service request that Oracle's customers encounter. Now let's see how you can be proactive and avoid some of these misconfigurations.
The following are some of the best practices for getting the most out of your Oracle ZFS Storage Appliance:
- Activate the Phone Home service.
- Perform pool scrubbing.
- Use the ZSCheck workflow tool.
- Run the ORAchk audit tool.
- Read the "Best Practices for Upgrading Oracle ZFS Storage Appliance" white paper.
- Use ZFS storage analytics to tune for performance.
- Get advanced support services for Oracle ZFS Storage Appliance.
- Get free upgrade assistance and best-practices information when you purchase an Oracle ZFS Storage Appliance Racked System Appliance.
Activate the Phone Home Service
The Phone Home feature is used to manage the registration of Oracle ZFS Storage Appliance as well as to provide the Phone Home remote support service.
During registration, it connects your appliance with Oracle Auto Service Request, which automatically opens service requests for specific problems reported by your appliance. Registration also connects your appliance with My Oracle Support to detect update notifications.
The Phone Home service communicates with Oracle support to provide the following:
- Fault reporting. The system reports active problems to Oracle for automated service response. Depending on the nature of the fault, a support case might be opened. Details of these events can be viewed in the "Problems" section of the Oracle ZFS Storage Appliance Customer Service Manual.
- Heartbeats. Daily heartbeat messages are sent to Oracle to indicate that the system is up and running. Oracle support may notify the technical contact for an account when one of the activated systems fails to send a heartbeat for too long.
- System configuration. Periodic messages are sent to Oracle describing current software and hardware versions and configuration as well as storage configuration. No user data or metadata is transmitted in these messages.
- Support bundles. The Phone Home service must be enabled before support bundles can be uploaded to Oracle Support. See the "System" section of the Oracle ZFS Storage Appliance Customer Service Manual for more information.
- Update notifications. The Phone Home service creates an alert when new software updates are available on My Oracle Support. See the "Software Update Notification" section of the Oracle ZFS Storage Appliance Customer Service Manual for more information.
You must register to use the Phone Home service.
Important: Ensure that the email address you use to register with Phone Home is accurate and up to date.
Figure 2 shows a sample system configuration health check:
Figure 2. Example system configuration health check
Other health-check variables include the following:
- Date of last pool scrub
- Pool data protection
- Misconfigured shares/pool without proper hardware components
To learn how to activate the Phone Home service, please visit the following web pages:
Perform Pool Scrubbing
Why Scrub?
The most important features of a ZFS file system are its data integrity and self-healing capabilities.
When a block of data is successfully written to a disk, there is still a (very small) chance that blocks might be corrupted because of media errors.
These blocks of data on the disk are even prone to silent data corruption that is not noticed by disk firmware. Furthermore, these errors might increase over time and remain undetected if no read operations are performed.
At this point, the scrub capability comes to the rescue.
In a tree format, ZFS stores a checksum of each data block in its parent block pointer.
A scrub operation traverses each block of data on the pool and verifies the checksum. If any inconsistency is detected, it tries to repair the block.
Depending on the pool's redundancy profile (for example, mirroring and RAIDZ), ZFS will find the good copy of the data and repair the corrupted block.
Unlike other file system repair methods, a ZFS scrub operation can run while the file system is mounted and in use.
Scrub Frequency
The frequency with which you perform a scrub depends on how frequently you read each block of data. It is recommended that you perform a manual scrub on the pool on a regular basis, for example, monthly (or at least quarterly, if the operation takes a long time).
The time required to perform a scrub depends on the I/O activity on the file system, the amount of data, the disk's revolutions per minute (RPM) speed, and the record size.
You can choose to perform a scrub manually during a low-I/O workload so that the scrub can be completed quickly. For information on how to start a scrub manually, see My Oracle Support document 2118680.1: Oracle ZFS Storage Appliance: How to start a scrub (verify content/check for errors) on a data pool.
Alternatively, you can perform a scrub at regular intervals using a workflow available from the following blog: ZFS Trick: Scheduled Scrubs.
Running a scrub operation at regular intervals ensures data integrity by detecting and repairing possible errors.
Performance
Scrubbing your pools does not have any effect on performance. Scrubbing takes place only when there are no I/O operations coming in to your disk.
To learn more about scrubbing, see My Oracle Support document 2118680.1: Oracle ZFS Storage Appliance: Importance of a ZFS Pool Scrub.
Use the ZSCheck Workflow Tool
ZSCheck (also sometimes called ZS_Basic_Config_Chk) is a BUI and command-line health-check tool implemented in the form of a workflow, which is designed to audit important configuration settings within Oracle ZFS Storage Appliance.
The tool checks and validates the following configuration settings:
- Firmware
- Operating key services
- DNS
- Phone Home
- SSH
- NIS/Active Directory/LDAP
- NTP
- Service tags
- And other services
- Cluster
- Network
- Pools (+vdevs)
- Readzillas and Logzillas
- Trays/stack
ZSCheck also checks for faulted or missing hardware components, such as the following:
- Disk drives
- Power supplies
- Fans
- I/O modules
- Service processor (Oracle Integrated Lights Out Manager [Oracle ILOM])
To learn more about ZSCheck and how to execute it, see My Oracle Support document 2046539.1: ZS Basic Configuration Check for the ZFS Appliance.
Run the ORAchk Audit Tool
ORAchk is the Oracle Configuration Audit Tool, which is designed to assess the health of Oracle Database, Oracle Grid Infrastructure, Oracle Database servers, Oracle hardware systems, the operating system, Oracle Real Application Clusters (Oracle RAC) software, and so on. ORAchk checks for known parameters that can have an impact on customer operations. ORAchk helps reduce risk and disruption to business.
ORAchk release 12.1.0.2.5 (announced on October 23, 2015) includes support for Oracle ZFS Storage Appliance. ORAchk audits configuration settings in Oracle ZFS Storage Appliance to check that they are within Oracle's recommended best practices for performance and resilience.
Figure 3 is a screenshot that shows a summary of what ORAchk is capable of checking.
Figure 3. Screenshot that summarizes what ORAchk can check
Figure 4 shows another ORAchk screenshot that provides more details.
I
Figure 4. ORAchk screenshot that provides more details
To check out the newest and most up-to-date release notes, including instructions on how to run ORAchk, see My Oracle Support document 1268927.2: ORAchk – Health Checks for the Oracle Stack.
Read the "Best Practices for Upgrading Oracle ZFS Storage Appliance" White Paper
This white paper serves as a planning guide for upgrading Oracle ZFS Storage Appliance software. It contains step-by-step instructions for the upgrade process, as well as best practices for ensuring success during the upgrade process. By reading the entire guide before performing an upgrade, you can devise a strategy that accurately reflects your ecosystem's requirements.
The white paper is located here.
Use ZFS Storage Analytics to Tune for Performance
How can you use analytics to identify a mismatch of the block size of application I/O versus the ZFS block size?
There is not a single analytic that directly answers this question, but there are two things you can check:
- Compare the size of the protocol (NFS, FC, SMB, iSCSI, and so on) used for I/O operations to the block size of the underlying file system (operations by share). If the I/O operations size from the protocol matches the record size, then you are close. This would tell you that you have the right size, but it would not tell you if the I/O operations are aligned.
- Next, compare the rate of access of the protocol for I/O operations to the Adaptive Replacement Cache (ARC) accesses. If the I/O operations are aligned, and the I/O operations size is less than or equal to the ZFS record size, then the protocol access rate should be the same as the ARC access rate.
To see the options available for analytics, do the following:
- Got to https://<hostname>:215/#configuration/preferences.
- Select the Make available advanced analytics statistics checkbox.
- Click Apply.
- Go to https://<hostname>:215/#analytics/worksheet.
- Refresh the Analytics page.
- Click the + (plus sign) next to Add statistic (to work with a larger list of analytics).
For other recommendations, see My Oracle Support document 1567691.1: How to use ZFS Storage Appliance Analytics to diagnose application, client OS, network, and storage tuning problems.
Get Advanced Support Services for Oracle ZFS Storage Appliance
Oracle Advanced Customer Support Services offer a portfolio of services from installation and configuration assistance, to performance reviews, monitoring, and support of Oracle ZFS Storage Appliance. These support services can expedite the successful deployment of your storage technology and increase your enterprise network-attached storage (NAS) performance.
The screenshot in Figure 5 shows some of these support services.
Figure 5. Examples of services provided by Oracle Advanced Customer Support Services
For more information, see the data sheet for Oracle Advanced Customer Support Services.
Get Free Upgrade Assistance and Best-Practice Information
Oracle ZFS Storage Appliance Racked System is a fully tested, assembled, and integrated storage system complete with Oracle Platinum Services that lowers total cost of ownership while reducing risk. This preconfigured storage system dramatically shortens deployment time while optimizing performance and availability. Additionally, Oracle Platinum Services ensure maximum uptime and rapid resolution with 24/7 remote fault monitoring, industry-leading response times, and patch deployment services. Oracle ZFS Storage Appliance Racked System is designed to run applications faster and more efficiently, providing a ready-to-deploy optimal storage architecture for enterprise cloud and virtualization, whether for on-premises private clouds or for cloud service providers.
For more information, see the "Oracle ZFS Storage Racked System data sheet."
Conclusion
I hope this information is useful to you. Remember that the more proactive you are, the higher the possibilities of getting the most out of your Oracle ZFS Storage Appliance.
Make sure you work with your service contract representatives and local Sale Consultant if you have questions regarding any of these best practices and tools.
About the Author
Patricio Zambrano Garcia has been working on Oracle's ZFS Storage Product Management—Tech Marketing team since September 2014. Some of his job duties include providing sales, performance sizing, and health-check tools for Oracle's ZFS storage products.
Before joining Oracle, he obtained a master's degree in computer science from Carnegie Mellon University, worked as an IT consultant for 3 years, and got a bachelor's degree in business administration from Universidad de Monterrey. He currently resides in Denver, Colorado.
Follow us:
Blog | Facebook | Twitter | YouTube