Archive | Misc RSS feed for this section

Migrating From a UCS FI6248 to Mini 6324

I have the luxury of working for the most awesomest storage and Cisco VAR in the world, as their Principal Technologist. We have a few different groups of engineers. Our networking team staked claim on the existing UCS and FI6248 Fabric Interconnects that we have in our lab for break/fix testing.

This meant that our demo and test VMware environment needed a new home. We really didn’t NEED another full set of big daddy FI’s in the lab, and the big boss didn’t want to buy another full set of FI6248s. We simply need a test environment for me to give classes with, that’ll support up to 600 VMs. A UCS Mini was perfect. This is how I got passed the hurdle of migrating the configuration from a FI6248 to FI6324.

Click to continue reading “Migrating From a UCS FI6248 to Mini 6324”

8 Comments Continue Reading →

My NS0-504 Study Notes – 3 SCALABLE SAN IMPLEMENTATION AND CONFIGURATION TASKS

Please click here to goto the main NS0-504 Study Notes blog post to see all sections and additional information

3 SCALABLE SAN IMPLEMENTATION AND CONFIGURATION TASKS

3.1 Prepare site for installation.

3.1.1 Be able to review implementation flowchart with customer and assign task areas.

 

3.1.2 Verify site infrastructure including: dual power, floor space, floor loading plan, HVAC.

Sample cDot Site Survey Questions (not including networking) (from an cDot old Survey to Quote document)

Cabinets:

Questions to help get the equipment racked and powered properly.

  • Does the customer plan on using NetApp Cabinets?
    • If Yes, what PDUs does the customer want in the cabinets?
      • What country will these racks be installed in?
      • Will redundant power be able to be configured?
    • If No, what is the height in RackU of the customer’s cabinets?
      • Will customer provide redundant power for the equipment?
  • Does the customer want NetApp to supply PDU-to-Head/Shelf power cords?
    • If so, what length?
    • If not, country-specific power cords will be included.
  • Does the customer want NetApp Universal Rail kits?
    • If so, 2 post or 4 post?
Cables:

Questions to help gauge cable lengths to connect cluster elements together.

  • How does the customer plan on running cables between racks?
    • Down, under the floor and back up?
    • Up, through ladder racks/trays and back down?
    • Through cabinet sides (not recommended)?
    • What cable length(s) will be required?
  • How will cabinets be set up on the datacentre floor?
    • All adjacent with no gaps or other equipment in between?
    • Separated by an aisle or other equipment?
    • If not all adjacent, please build diagram of cabinet layout to facilitate cable length calculation with PS

 

Questions to help gauge SFPs and cable lengths to connect cluster to host/client side networks.

Clusters with NAS, iSCSI & FCoE protocols:

  • What kind and quantity of Ethernet ports does the customer require?
    • o   10GigE with optical SFP+
    • o   10Gig CNA with optical SFP+
    • o   10GigE bare cage
    • o   10Gig CNA bare cage
    • o   1GigE with optical SFP
    • o   1GigE with RJ45 connector
  • Does the customer want NetApp to supply cables for any of the above?
    • If so, what length(s)?
  • For bare cage 10GigE & 10GigE CNA, does the customer want copper 10Gig cables?
    • If so, what length(s)?

Clusters with FC protocol:

  • What kind and quantity of FC ports does the customer require?
    • o   8Gbps with optical SFP
    • o   4Gbps with optical SFP
    • o   8Gbps bare cage
    • o   4Gbps bare cage
  • Does the customer want NetApp to supply cables for any of the above?
    • If so, what length(s)?

 

  • Read the Netapp Site Requirement guide
  • Review the BTU and Ton conversion of the heat output of the controllers and disk shelves.
  • Know the NEMA vs IEC power cables and voltage
    • C13 – C14
    • NEMA L5-15/L5-20
    • C19-C20

3.1.3 Validate equipment move path to installation location.

  • This sounds stupid, but I have seen racks fall over, or bust plastic wheels.
  • Walk it!

3.1.4 Validate logistics plan for staging and installation of equipment.

  •  Will this be built in a lab room? Built in production rack? Etc.

3.1.5 Verify Ethernet cabling plan and availability of cable supports.

Valid 1gb Administration Switches

  • Cisco 2960
  • Netapp 1601 (may not have existed at time of test creation)

Valid 10gb Cluster Interconnect Switches

  • Cisco 5010 – 12 or 18 node clusters
  • Cisco 5020 – Up to 32 nodes (not like thats possible, realistically 24 nodes)
  • Cisco 5548UP
  • Cisco 5596UP – Up to 40 nodes
  • Cisco CN1610 (rebranded Broadcom BCM53716-16FE or larger port count)

 

Sample cabling plan:

  • Netapp CN1610 Cabling.

Overview Cabling Diagram

esxi51_ucsm2_Clusterdeploy-002

Sample Cabling Guide of a Flexpod – For illustration only. Dont use on modern production.

esxi51_ucsm2_Clusterdeploy-003

 

3.1.6 Verify fiber cabling plan and availability of cable supports.

arch1

 


 

3.2 Following the rack diagram, install systems and FC switches.

Example Rack Diagram

example_rack_diagram

3.3 Storage System Configuration Tasks.

3.3.1 Data ONTAP 8.1.1 Cluster-Mode Setup Tasks

 

3.3.2 Storage Provisioning and Vserver Setup Tasks

  • vserver create
  • protocol configuration
  • volume create
  • Adding to namespace
  • lun create
  • NFS: Export policy creation

 

  • Other tasks
    • Lif Migrate
    • ARL Aggregate Relocate
    • DataMotion for Volumes

3.3.3 FC, FCoE, and iSCSI Connectivity Tasks

 

  • We are almost always deploying a “switched fabric” with a core-edge topology with FC.
    • Core-edge topology: In this design, storage is always put in the core, and hosts are always attached at the edge. This design is effective because SAN traffic flows are typically not peer to peer but instead many to one (hosts to storage). (Definition from Cisco MDS documentation)
    • Edge-core-edge topology: This common design (storage edge to core to host edge) is used when a core-edge design provides insufficient scalability and an additional edge tier is needed to handle the large number of devices. (Definition from Cisco MDS documentation)
[caption id="" align="alignnone" width="265"] Topologies[/caption]

 

FC Cable comparison (because they always have annoying distance questions, not like we couldnt just google this when we need to)

Comparison (from Wikipedia)

Category Minimum modal bandwidth
850 nm / 1310 nm
100 Mb Ethernet 100BASE-FX 1 Gb (1000 Mb) Ethernet 1000BASE-SX 10 Gb Ethernet 10GBASE-SR 40 Gb Ethernet 100 Gb Ethernet
OM1 (62.5/125) 200 / 500 MHz·km up to 2000 meters (FX) 275 meters (SX) 33 meters (SR) Not supported Not supported
OM2 (50/125) 500 / – MHz·km up to 2000 meters (FX) 550 meters (SX) 82 meters (SR) Not supported Not supported
OM3 (50/125) *Laser Optimized* 1500 / 2000 MHz·km up to 2000 meters (FX) 550 meters (SX) 300 meters (SR) 100 meters330 meters QSFP+ eSR4 100 meters
OM4 (50/125) *Laser Optimized* 3500 / 4700 MHz·km up to 2000 meters (FX) 1000 meters (SX) 400 meters (SR) 150 meters550 meters QSFP+ eSR4 150 meters

Fibre Channel loop speeds/Distance from Siemons

Connection Speed and Distance by Cable Category
Type Speed Distance
OM2 1Gb/s 500m/1,640’
OM3 1Gb/s 500m/1,640’
OM2 2Gb/s 300m/900’
OM3 2Gb/s 500m/1,640’
OM2 4Gb/s 150m/492’
OM3 4Gb/s 270m/886’
OM2 8Gb/s 50m/1,64’
OM3 8Gb/s 150m/492’
Twinax copper 8Gb/s 15m max’
  • Optical for SAS between controller and shelves distances: The point-to-point (QSFP-to-QSFP) path of any multimode cable cannot exceed 150 meters for OM4 and 100 meters for OM3.  Though this isnt supported on 8.1.1.
  • Important note!:  If you are using patch panels with a different thickness (62.5/125 instead of 50.125), you should match the patch panel fiber thickness to the end host or san A transition between 62.5/125 and 50/125 may result in a possible loss of signal strength.  Read this for more information. ( thefoa.org )

mismatch

 

 

3.3.4 LUN Connectivity Tasks

  •  ALUA information for pathing preferences is gathered by the host sending scsi inquiry (or new  REPORT_TARGET_PORT_GROUPS) command. (From Netapp Knowledge Base)
    • The storage system implements four states for a LUN:
      • Active/Optimized
      • Active/Non-Optimized
      • Unavailable
      • Transitioning
    • These map to the following existing Data ONTAP terms:
      • Local/Fast/Primary
      • Partner/Proxy/Slow/Secondary
      • Cluster IC is down, path is not functional
      • Path is transitioning to another state

 

 

3.3.5 Configure FC and Ethernet switches

 

3.3.6 Host Configuration Tasks

3.3.7 Virtualized Environment/Platforms: SAN Best Practices.

3.3.8 FCoE and Unified Connect Enabling Technologies

  • Relies on DataCenter Bridging (DCB)
    • Data center bridging (DCB) is a collection of extensions to the existing Ethernet standard that provides a lossless transport layer for FCoE traffic.
      • Per-priority pause (priority-based flow control)
        • Enables a device to only inhibit the transmission of frames based on user-defined priorities.
      • Enhanced transmission selection
        • Allows administrators to allocate bandwidth on a percentage basis to different priorities.
      • Congestion notification
        • Transmits congestion information.
      • DCB Exchange (DCBX) protocol
        • Exchanges connection information with directly connected peers and detects misconfigurations.
    • SAN Admin Guide Troubleshooting – Page 70-71 is a great resource for seeing this in action
      • Default priorities is 3 for FCoE traffic & 50% Bandwidth

        • Page 20 of TR3894 states this.
      • Default priority is 0 for IP traffic.

 

3.3.9 FCoE and Unified Connect Hardware

3.3.10 FCoE and Unified Connect Configuration

To move:

  • Must read: FCoE End to End Deployment (TR-3800 Older – 2011, w/ Qlogic)
  • Configuring CNA/UTA Ports
    • ucadmin
    • fcp config
  • FCoE Overview for Clustered Ontap
  • FC and FCoE Zoning
    • Best practices and recommendations
      • Netapp recommends “Single Initiator Zoning”
        • A zone should include a SINGLE INITIATOR and ALL Targets the initiator is connecting to.
      • Zoning should be based on World Wide Port Name (wwpn)
        • Change an HBA card, update the zone. Change a server, update the zone.
      • 50/125 Recommended.
      • Short Wave SFPs required to connect to onboard FC.
      • Should be used when you have 4 or more hosts connected (really always)
      • Orange cable typically OM2. “Laser Optimized” needed for short length 10gb.
      • OM3 and OM4 are cyan (blue), recommended for 8gb FC, 10GB eth.
  •  Why Zone?
    • Reduces CROSS TALK between initiator HBAs
    • Reduces paths to an available port
    • Increases Security
    • Shortens troubleshooting times
  • Zoning

 

1 Comment Continue Reading →

My NS0-504 NCIE-SAN Cluster Mode Study Notes

I need to renew my NCIE-SAN Cluster Mode NS0-504 this year, so I figure NetApp Insight 2014 is the place to do it. Why? Because it’s free Yo!

Much of this I already know, but to error on the side of caution, and not be too overconfident, I will be stepping through as many of the Study Guide lists as possible. Doing some hands on examples.

Click to continue reading “My NS0-504 NCIE-SAN Cluster Mode Study Notes”

11 Comments Continue Reading →

Quick Tip: Powercli to switch all VMware LUNs to Round Robin

Just for my own notes and quick copy and paste in the future… how to single line switch all luns to round robin on their paths.

Click to continue reading “Quick Tip: Powercli to switch all VMware LUNs to Round Robin”

1 Comment Continue Reading →

Wordmap of the Blog

Saw this on @f0zziehakz’s Blog and thought it was cool.

Word mapping my blogs and thoughts. Just bcuz.

Click to continue reading “Wordmap of the Blog”

Leave a comment Continue Reading →

pkgrm EMCpower kills Solaris

EMC to NetApp Solaris FCP migration – Recovery When EMC Breaks the box

So, I have been working on an EMC to Netapp migration project, and the customer is entirely Solaris 10 on their Oracle DB environment. (They want to change, sadly they cannot at this time.)

Dealing with migrations between vendors can be difficult. In EMC’s world, we have POWERPATH, which is a pile of suck when you have to work with SAN vendors on the same box who want to rightly use MPXIO (Native multipathing). I hate 3rd party addins for things like Multipathing. This is 2014, the crap is built in!

Well, what happens when you switch a box from POWERPATH to Native multipathing with MPXIO? All the power path stuff goes mad and it can’t find any of it. That is for another blog power, not this one.

What about when the migration is done, and you want to remove POWERPATH from the server? Well it’ll kernel panic the box, it’ll corrupt the boot archive, and you will be in a reboot loop. Dead in the water, thanks to EMC.

This post isn’t just about migrating to NetApp, it’ll help anyone who tried to uninstall PowerPath on solaris, and got corrupted system files from it, and how to rebuild the BOOT ARCHIVE.

Click to continue reading “pkgrm EMCpower kills Solaris”

Leave a comment Continue Reading →

New Year. Looking back, looking ahead.

A year in review, and a look ahead!

We have a gift of mind and body given to us every day we are alive, no matter what we face with either. Forcing myself to do which I never thought I could, is the best Christmas and New Year gift I could give myself. I do not want to stop challenging myself until it kills me!
What about 2014? Read more for my thoughts on where you should and shouldn’t be spending your time!

Click to continue reading “New Year. Looking back, looking ahead.”

Leave a comment Continue Reading →

Filesystem Misalignment for dummies

OMFG BBQ SAUCE!

Me: “How many SAN IO Calls would you need to make to get File 1 for the Misaligned storage?”
Customer: 5
Me: “What about aligned?”
Customer: 2
Me: “What about File 2?”
Customer: 7
Me: “What about aligned?”
Customer: 2
Me: “How many tiny files, and fragments do you have on your fileservers?”
Customer: “Millions…”
Me: “Wouldn’t you like to save 2-4x or more IO calls and cache?”
Customer: “We love you, have our babies!”

Click to continue reading “Filesystem Misalignment for dummies”

Leave a comment Continue Reading →

Some cool Riverbed Upgrades and Additions

Gnomes on Redbull

When ever I hear of new products and features, I put the statement through the paces to see if it’s total BS, or something worthwhile. This morning, an email caught my eye. Riverbed made their announcement of upgrades and product line additions.

I love what Riverbed can do for their customers. I won’t lie. No product is perfect, but It’s just been too damn useful for think bad about anything of it. And when they bough CACE, and turned Wireshark into something even better, and that Riverbed Pilot is like Wireshark on crack with some redbull-fed gnomes for historical analytics… well I am even happier with the offerings.

Click to continue reading “Some cool Riverbed Upgrades and Additions”

Leave a comment Continue Reading →

Wish I went to DEFCON :(

Well another DEFCON is done.  I couldn’t make it this year due to work, finances, and life.  I miss all my DC buddies, especially my great friend @SecEvangelism!

You can’t look back, only forward.  My main reason for going to DEFCON, besides hanging with people I know online and rarely see in real life, is the lock picking village.  I truly enjoy teaching there (in a totally unofficial capacity).  Just hanging out in the air conditioning, picks in hand, fresh batches of new people all the time to talk to!

I did end up finding myself in the DEFCON20 Documentary for a few seconds.  I really wish they would have put more of an emphasis into the LPV in the movie.  It is a huge piece of DEFCON, where everybody who enters sees a little bit of applicable security (or lack their of) in their daily lifes.

Heck, I even had the documentary crew filming next to me when i was teaching some of the kids how to get out of handcuffs!

 

[caption id="attachment_390" align="aligncenter" width="450"] Me teaching lockpicking and how to escape handcuffs![/caption]

I hope to see all my security friends soon!  Maybe out here in NYC or DC sometime!

 

[asa]1597499897[/asa]

[asa]1597499838[/asa]

Leave a comment Continue Reading →