Please click here to goto the main NS0-504 Study Notes blog post to see all sections and additional information
3 SCALABLE SAN IMPLEMENTATION AND CONFIGURATION TASKS
3.1 Prepare site for installation.
3.1.1 Be able to review implementation flowchart with customer and assign task areas.
3.1.2 Verify site infrastructure including: dual power, floor space, floor loading plan, HVAC.
Sample cDot Site Survey Questions (not including networking) (from an cDot old Survey to Quote document)
Cabinets:
Questions to help get the equipment racked and powered properly.
- Does the customer plan on using NetApp Cabinets?
- If Yes, what PDUs does the customer want in the cabinets?
- What country will these racks be installed in?
- Will redundant power be able to be configured?
- If No, what is the height in RackU of the customer’s cabinets?
- Will customer provide redundant power for the equipment?
- If Yes, what PDUs does the customer want in the cabinets?
- Does the customer want NetApp to supply PDU-to-Head/Shelf power cords?
- If so, what length?
- If not, country-specific power cords will be included.
- Does the customer want NetApp Universal Rail kits?
- If so, 2 post or 4 post?
Cables:
Questions to help gauge cable lengths to connect cluster elements together.
- How does the customer plan on running cables between racks?
- Down, under the floor and back up?
- Up, through ladder racks/trays and back down?
- Through cabinet sides (not recommended)?
- What cable length(s) will be required?
- How will cabinets be set up on the datacentre floor?
- All adjacent with no gaps or other equipment in between?
- Separated by an aisle or other equipment?
- If not all adjacent, please build diagram of cabinet layout to facilitate cable length calculation with PS
Questions to help gauge SFPs and cable lengths to connect cluster to host/client side networks.
Clusters with NAS, iSCSI & FCoE protocols:
- What kind and quantity of Ethernet ports does the customer require?
- o 10GigE with optical SFP+
- o 10Gig CNA with optical SFP+
- o 10GigE bare cage
- o 10Gig CNA bare cage
- o 1GigE with optical SFP
- o 1GigE with RJ45 connector
- Does the customer want NetApp to supply cables for any of the above?
- If so, what length(s)?
- For bare cage 10GigE & 10GigE CNA, does the customer want copper 10Gig cables?
- If so, what length(s)?
Clusters with FC protocol:
- What kind and quantity of FC ports does the customer require?
- o 8Gbps with optical SFP
- o 4Gbps with optical SFP
- o 8Gbps bare cage
- o 4Gbps bare cage
- Does the customer want NetApp to supply cables for any of the above?
- If so, what length(s)?
- Read the Netapp Site Requirement guide
- Review the BTU and Ton conversion of the heat output of the controllers and disk shelves.
- Know the NEMA vs IEC power cables and voltage
- C13 – C14
- NEMA L5-15/L5-20
- C19-C20
3.1.3 Validate equipment move path to installation location.
- This sounds stupid, but I have seen racks fall over, or bust plastic wheels.
- Walk it!
3.1.4 Validate logistics plan for staging and installation of equipment.
- Will this be built in a lab room? Built in production rack? Etc.
3.1.5 Verify Ethernet cabling plan and availability of cable supports.
Valid 1gb Administration Switches
- Cisco 2960
- Netapp 1601 (may not have existed at time of test creation)
Valid 10gb Cluster Interconnect Switches
- Cisco 5010 – 12 or 18 node clusters
- Cisco 5020 – Up to 32 nodes (not like thats possible, realistically 24 nodes)
- Cisco 5548UP
- Cisco 5596UP – Up to 40 nodes
- Cisco CN1610 (rebranded Broadcom BCM53716-16FE or larger port count)
Sample cabling plan:
- Netapp CN1610 Cabling.
Overview Cabling Diagram
Sample Cabling Guide of a Flexpod – For illustration only. Dont use on modern production.
3.1.6 Verify fiber cabling plan and availability of cable supports.
3.2 Following the rack diagram, install systems and FC switches.
Example Rack Diagram
3.3 Storage System Configuration Tasks.
3.3.1 Data ONTAP 8.1.1 Cluster-Mode Setup Tasks
3.3.2 Storage Provisioning and Vserver Setup Tasks
- vserver create
- protocol configuration
- volume create
- Adding to namespace
- lun create
- NFS: Export policy creation
- Other tasks
- Lif Migrate
- ARL Aggregate Relocate
- DataMotion for Volumes
3.3.3 FC, FCoE, and iSCSI Connectivity Tasks
- We are almost always deploying a “switched fabric” with a core-edge topology with FC.
- Core-edge topology: In this design, storage is always put in the core, and hosts are always attached at the edge. This design is effective because SAN traffic flows are typically not peer to peer but instead many to one (hosts to storage). (Definition from Cisco MDS documentation)
- Edge-core-edge topology: This common design (storage edge to core to host edge) is used when a core-edge design provides insufficient scalability and an additional edge tier is needed to handle the large number of devices. (Definition from Cisco MDS documentation)
FC Cable comparison (because they always have annoying distance questions, not like we couldnt just google this when we need to)
Comparison (from Wikipedia)
Category | Minimum modal bandwidth 850 nm / 1310 nm |
100 Mb Ethernet 100BASE-FX | 1 Gb (1000 Mb) Ethernet 1000BASE-SX | 10 Gb Ethernet 10GBASE-SR | 40 Gb Ethernet | 100 Gb Ethernet |
---|---|---|---|---|---|---|
OM1 (62.5/125) | 200 / 500 MHz·km | up to 2000 meters (FX) | 275 meters (SX) | 33 meters (SR) | Not supported | Not supported |
OM2 (50/125) | 500 / – MHz·km | up to 2000 meters (FX) | 550 meters (SX) | 82 meters (SR) | Not supported | Not supported |
OM3 (50/125) *Laser Optimized* | 1500 / 2000 MHz·km | up to 2000 meters (FX) | 550 meters (SX) | 300 meters (SR) | 100 meters330 meters QSFP+ eSR4 | 100 meters |
OM4 (50/125) *Laser Optimized* | 3500 / 4700 MHz·km | up to 2000 meters (FX) | 1000 meters (SX) | 400 meters (SR) | 150 meters550 meters QSFP+ eSR4 | 150 meters |
Fibre Channel loop speeds/Distance from Siemons
Connection Speed and Distance by Cable Category | ||
---|---|---|
Type | Speed | Distance |
OM2 | 1Gb/s | 500m/1,640’ |
OM3 | 1Gb/s | 500m/1,640’ |
OM2 | 2Gb/s | 300m/900’ |
OM3 | 2Gb/s | 500m/1,640’ |
OM2 | 4Gb/s | 150m/492’ |
OM3 | 4Gb/s | 270m/886’ |
OM2 | 8Gb/s | 50m/1,64’ |
OM3 | 8Gb/s | 150m/492’ |
Twinax copper | 8Gb/s | 15m max’ |
- Optical for SAS between controller and shelves distances: The point-to-point (QSFP-to-QSFP) path of any multimode cable cannot exceed 150 meters for OM4 and 100 meters for OM3. Though this isnt supported on 8.1.1.
- Important note!: If you are using patch panels with a different thickness (62.5/125 instead of 50.125), you should match the patch panel fiber thickness to the end host or san. A transition between 62.5/125 and 50/125 may result in a possible loss of signal strength. Read this for more information. ( thefoa.org )
3.3.4 LUN Connectivity Tasks
- ALUA information for pathing preferences is gathered by the host sending scsi inquiry (or new
REPORT_TARGET_PORT_GROUPS
) command. (From Netapp Knowledge Base)- The storage system implements four states for a LUN:
- Active/Optimized
- Active/Non-Optimized
- Unavailable
- Transitioning
- These map to the following existing Data ONTAP terms:
- Local/Fast/Primary
- Partner/Proxy/Slow/Secondary
- Cluster IC is down, path is not functional
- Path is transitioning to another state
- The storage system implements four states for a LUN:
3.3.5 Configure FC and Ethernet switches
- Eth Switch Config
- Cisco Nexus 5000 vPC Best Practice (Not sure if they will ask about the 50×0 or the 5548/96 based off the 8.1.1 dates or even their own co-branded switches. Remember, Netapp was switching to different $1 interconnect switches at the time of exam creation)
- Cisco Enhanced vPC Guide
- Cabling the CN1610 Interconnect Switches
- Direct link to Switch zoning from Cisco Design Guide
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 |
# Cisco Nexus are typically configured in Pairs, and the configurations are mirrored between each other. #Important Netapp Related Switch config settings: #Enable LACP link aggregation feature lacp # Enable vPC ( Virtual Port Channel aka Multichassis LACP ) feature vpc # Add QoS (Quality of Service) Rules for FCoE to get very high priority and not drop class-map type qos class-fcoe class-map type queuing class-fcoe match qos-group 1 class-map type queuing class-all-flood match qos-group 2 class-map type queuing class-ip-multicast match qos-group 2 class-map type network-qos class-fcoe match qos-group 1 class-map type network-qos class-all-flood match qos-group 2 class-map type network-qos class-ip-multicast match qos-group 2 #And also to globally enable jumbo frames policy-map type network-qos jumbo class type network-qos class-default mtu 9216 multicast-optimize # Tell the system to use the above QoS junk system qos service-policy type network-qos jumbo service-policy type queuing input fcoe-default-in-policy service-policy type queuing output fcoe-default-out-policy service-policy type qos input fcoe-default-in-policy # Make some VLANs vlan 301 name VMOTION vlan 401 name IP_STORAGE vlan 402 name IP_STORAGE_ISCSI # Configure the VPN domain ID both switches will have configured, and where the other guy is to check it's heartbeat vpc domain 2 role priority 100 peer-keepalive destination 192.168.254.2 source 192.168.254.1 peer-config-check-bypass peer-gateway # Define a Port Channel for each filer, and tag it for the vpc ID each switch will use so they know these are one virtual port channel across each switch interface port-channel130 description ha-cdot-02 switchport mode trunk switchport trunk allowed vlan 2-3967,4048-4093 speed 10000 vpc 130 interface port-channel131 description ha-cdot-01 switchport mode trunk switchport trunk allowed vlan 2-3967,4048-4093 speed 10000 vpc 131 # Assign the port-channel to the 10gb interfaces interface Ethernet1/18 description ha-cdot-02 switchport mode trunk switchport trunk allowed vlan 2-3967,4048-4093 channel-group 130 mode active interface Ethernet1/20 description ha-cdot-02 switchport mode trunk switchport trunk allowed vlan 2-3967,4048-4093 channel-group 131 mode active |
- Brocade FC Zoning
- Zoning a Brocade Switch with WEBtool (Video)
- Zoning a Brocade Switch with the CLI (Video)
- How to Zone a Brocade SAN Switch
- Brocade Fabric OS Administration Guide, (FOS) Read Chapters 1 11 and 15 (Get your hands on a Brocade 320 or an older 4000 to practice zoning)
- Brocade’s Default Fabric ID=1. (This is sort of like a unsetup Cisco switch is vlan 1 everywhere, be weary of connecting 2 brocades both with fabric 1 if trying to get fabric redundancy)
- So change the Fabric ID:
- switchDisable (This will stop traffic, do it before you begin!!)
- configure (change the fabric ID)
- switchEnable
- So change the Fabric ID:
3.3.6 Host Configuration Tasks
- Verify HBA/CNA Driver and Firmware to OS and Netapp Host Utilities and Snapdrive in the Support Matrix
- Install NetApp Host Utilities
- Install SnapDrive
- SAN HBA Management Utilities
- Emulex: HBAnywhere (or newer OneCommand)
- qLogic SAN Surfer
3.3.7 Virtualized Environment/Platforms: SAN Best Practices.
3.3.8 FCoE and Unified Connect Enabling Technologies
- Relies on DataCenter Bridging (DCB)
- Data center bridging (DCB) is a collection of extensions to the existing Ethernet standard that provides a lossless transport layer for FCoE traffic.
- Per-priority pause (priority-based flow control)
- Enables a device to only inhibit the transmission of frames based on user-defined priorities.
- Enhanced transmission selection
- Allows administrators to allocate bandwidth on a percentage basis to different priorities.
- Congestion notification
- Transmits congestion information.
- DCB Exchange (DCBX) protocol
- Exchanges connection information with directly connected peers and detects misconfigurations.
- Per-priority pause (priority-based flow control)
- SAN Admin Guide Troubleshooting – Page 70-71 is a great resource for seeing this in action
- Default priorities is 3 for FCoE traffic & 50% Bandwidth
- Page 20 of TR3894 states this.
- Default priority is 0 for IP traffic.
- Default priorities is 3 for FCoE traffic & 50% Bandwidth
- Data center bridging (DCB) is a collection of extensions to the existing Ethernet standard that provides a lossless transport layer for FCoE traffic.
3.3.9 FCoE and Unified Connect Hardware
3.3.10 FCoE and Unified Connect Configuration
To move:
- Must read: FCoE End to End Deployment (TR-3800 Older – 2011, w/ Qlogic)
- Configuring CNA/UTA Ports
- ucadmin
- fcp config
- FCoE Overview for Clustered Ontap
- FC and FCoE Zoning
- Best practices and recommendations
- Netapp recommends “Single Initiator Zoning”
- A zone should include a SINGLE INITIATOR and ALL Targets the initiator is connecting to.
- Zoning should be based on World Wide Port Name (wwpn)
- Change an HBA card, update the zone. Change a server, update the zone.
- 50/125 Recommended.
- Short Wave SFPs required to connect to onboard FC.
- Should be used when you have 4 or more hosts connected (really always)
- Orange cable typically OM2. “Laser Optimized” needed for short length 10gb.
- OM3 and OM4 are cyan (blue), recommended for 8gb FC, 10GB eth.
- Netapp recommends “Single Initiator Zoning”
- Best practices and recommendations
- Why Zone?
- Reduces CROSS TALK between initiator HBAs
- Reduces paths to an available port
- Increases Security
- Shortens troubleshooting times
- Zoning
[…] 3 SCALABLE SAN IMPLEMENTATION AND CONFIGURATION TASKS […]