Well, it’s time! This morning the first release candidate of OnTap 8.3 was released! Happy day!
I figured I would post about the upgrade process. A raw, uncut, look at how the hell you do it! Maybe you have never seen it. Maybe you just want to have something to copy and paste. Well, here it is!
I did run into one hiccup with one of my aggregates and the mixed version state. I overrode the vetos, and pretty much forced it to happen.
One note, 8.3 is VERY picky with networking. It has to have a management LIF in EVERY svm! This is a new requirement! It will test connectivity from each svm (to things like dns) to verify it will not lose connectivity to them.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 |
a-cdot::> vserver show Admin Root Name Name Vserver Type State Volume Aggregate Service Mapping ----------- ------- --------- ---------- ---------- ------- ------- ha-cdot admin - - - - - ha-cdot-01 node - - - - - ha-cdot-02 node - - - - - svm-demo-cifs data running democifs_ aggr2_ file, file root cdot02 nis svm_lab_infra data running svm_lab_ aggr2_ file, file infra_root cdot02 nis 5 entries were displayed. ha-cdot::> network interface show Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home ----------- ---------- ---------- ------------------ ------------- ------- ---- ha-cdot cluster_mgmt up/up 10.192.1.130/24 ha-cdot-01 a1a-1921 true ha-cdot-01 clus1 up/up 169.254.156.24/16 ha-cdot-01 e1a true clus2 up/up 169.254.251.4/16 ha-cdot-01 e2a true mgmt1 up/up 10.192.1.131/24 ha-cdot-01 e0b true ha-cdot-02 clus1 up/up 169.254.136.101/16 ha-cdot-02 e1a true clus2 up/up 169.254.37.40/16 ha-cdot-02 e2a true mgmt1 up/up 10.192.1.132/24 ha-cdot-02 e0b true svm-demo-cifs demo-cifs_cifs_lif1 up/up 10.192.1.134/24 ha-cdot-01 a1a-1921 true demo-cifs_vlan2131_lif up/up 10.213.1.70/24 ha-cdot-01 a1a-2131 true svm_lab_infra svm_lab_infra_iscsi1 up/up 10.213.8.233/24 ha-cdot-01 a1a-2138 true svm_lab_infra_nfs_lif1 up/up 10.213.8.232/24 ha-cdot-02 a2a-2138 true 11 entries were displayed. ha-cdot::> network interface create -vserver svm-demo-cifs -lif demo-cifs_vlan2131_mgmt_lif -role data -data-protocol none -home-node ha-cdot-01 -home-port a1a-2131 -address 10.213.1.71 -netmask 255.255.255.0 -routing-group d10.213.1.0/24 -status-admin up -failover-policy nextavail -firewall-policy mgmt -auto-revert true ha-cdot::> network interface create -vserver svm_lab_infra -lif svm_lab_infra_mgmt_lif -role data -data-protocol none -home-node ha-cdot-01 -home-port a1a-2131 -address 10.213.1.233 -netmask 255.255.255.0 -status-admin up -failover-policy nextavail -firewall-policy mgmt -auto-revert true Info: Your interface was created successfully; the routing group d10.213.1.0/24 was created ha-cdot::> network routing-groups route show Routing Vserver Group Destination Gateway Metric --------- --------- --------------- --------------- ------ ha-cdot c10.192.1.0/24 0.0.0.0/0 10.192.1.254 20 ha-cdot c10.213.1.0/24 0.0.0.0/0 10.213.1.254 20 ha-cdot-01 n10.192.1.0/24 0.0.0.0/0 10.192.1.254 10 ha-cdot-02 n10.192.1.0/24 0.0.0.0/0 10.192.1.254 10 svm-demo-cifs d10.192.1.0/24 0.0.0.0/0 10.192.1.11 20 svm_lab_infra d10.213.8.0/24 0.0.0.0/0 10.213.8.254 20 6 entries were displayed. ha-cdot::> network routing-groups route create -vserver svm-demo-cifs -routing-group d10.213.1.0/24 -destination 0.0.0.0/0 -gateway 10.213.1.254 ha-cdot::system health controller> system node service-processor ha-cdot::system node service-processor> show IP Firmware Node Type Status Configured Version IP Address ------------- ---- ----------- ------------ --------- ------------------------- ha-cdot-01 SP online true 1.4.1 10.192.1.133 ha-cdot-02 SP online true 1.4.1 10.192.1.134 2 entries were displayed. ha-cdot::> system node image update -node ha-cdot-01 -package http://10.1.13.47:8000/83RC1_q_image.tgz -replace-package true Software update started on node ha-cdot-01. Updating image1 with package http://10.1.13.47:8000/83RC1_q_image.tgz. Downloading package. This may take up to 10 minutes. 99% downloaded 100% downloaded Download complete. Listing package contents. Decompressing package contents. Invoking script (validation phase). INSTALL running in check-only mode: the image will be validated only Mode of operation is UPDATE Current image is image2 Alternate image is image1 Package MD5 checksums pass Versions are compatible Available space on boot device is 1560 MB Required space on boot device is 496 MB Kernel binary matches install machine type LIF checker script is invoked. NO CONFIGURATIONS WILL BE CHANGED DURING THIS TEST. Checking ALL Vservers for sufficiency LIFs. Running in upgrade mode. Running in report mode. Enabling Script Optimizations. Checker may take up to 30 minutes to complete. Checking Vserver(s) (svm-demo-cifs svm_lab_infra) for sufficiency LIFs. .. ........ ...... .............. ............ .. .............. ............ ..............SUCCESS: All external servers are reachable. LIF checker script has validated configuration. Invoking script (install phase). This may take up to 30 minutes. Mode of operation is UPDATE Current image is image2 Alternate image is image1 Package MD5 checksums pass Versions are compatible Available space on boot device is 1560 MB Required space on boot device is 496 MB Kernel binary matches install machine type Getting ready to install image Syncing device... Extracting to /cfcard/x86_64/freebsd/image1... x CHECKSUM x VERSION x COMPAT.TXT x BUILD x metadata.xml x INSTALL x netapp_sufficiency_lif_checker x cap.xml x platform.ko x kernel x fw.tgz x platfs2g.img x rootfs.img Installed MD5 checksums pass Installing diagnostic and firmware files Firmware MD5 checksums pass Installation complete. image1 updated on node ha-cdot-01. ha-cdot::> ha-cdot::> system image show Is Is Install Node Image Default Current Version Date -------- ------- ------- ------- ------------------------- ------------------- ha-cdot-01 image1 true false 8.3RC1 11/13/2014 14:31:46 image2 false true 8.2.2 10/1/2014 02:48:12 ha-cdot-02 image1 true true 8.2.2 10/1/2014 14:30:08 image2 false false 8.2.1 6/16/2014 20:05:12 4 entries were displayed. ha-cdot::> system node image update -node ha-cdot-02 -package http://10.1.13.47:8000/83RC1_q_image.tgz -replace-package true Software update started on node ha-cdot-02. Updating image2 with package http://10.1.13.47:8000/83RC1_q_image.tgz. Downloading package. This may take up to 10 minutes. 99% downloaded 100% downloaded Download complete. Listing package contents. Decompressing package contents. Invoking script (validation phase). INSTALL running in check-only mode: the image will be validated only Mode of operation is UPDATE Current image is image1 Alternate image is image2 Package MD5 checksums pass Versions are compatible Available space on boot device is 1584 MB Required space on boot device is 497 MB Kernel binary matches install machine type LIF checker script is invoked. NO CONFIGURATIONS WILL BE CHANGED DURING THIS TEST. Checking ALL Vservers for sufficiency LIFs. Running in upgrade mode. Running in report mode. Enabling Script Optimizations. No need to do upgrade check of external servers for this installed version. LIF checker script has validated configuration. Invoking script (install phase). This may take up to 30 minutes. Mode of operation is UPDATE Current image is image1 Alternate image is image2 Package MD5 checksums pass Versions are compatible Available space on boot device is 1584 MB Required space on boot device is 497 MB Kernel binary matches install machine type Getting ready to install image Syncing device... Extracting to /cfcard/x86_64/freebsd/image2... x CHECKSUM x VERSION x COMPAT.TXT x BUILD x metadata.xml x INSTALL x netapp_sufficiency_lif_checker x cap.xml x platform.ko x kernel x fw.tgz x platfs2g.img x rootfs.img Installed MD5 checksums pass Installing diagnostic and firmware files Firmware MD5 checksums pass Installation complete. image2 updated on node ha-cdot-02. ha-cdot::> system image show Is Is Install Node Image Default Current Version Date -------- ------- ------- ------- ------------------------- ------------------- ha-cdot-01 image1 true false 8.3RC1 11/13/2014 14:31:46 image2 false true 8.2.2 10/1/2014 02:48:12 ha-cdot-02 image1 true true 8.2.2 10/1/2014 14:30:08 image2 false false 8.3RC1 11/13/2014 14:43:48 4 entries were displayed. ha-cdot::> system image modify -node ha-cdot-02 -image image2 -isdefault true Error: command failed: Capability check failed. Reason: The default image has not been set because it lacks capabilities which have been made obsolete. These capabilities must be disabled using the following instructions before the default image can be set: format_32bit: You must remove all 32-bit data from the cluster and disable the 32-bit aggregate support using "storage aggregate 64bit-upgrade 32bit-disable" command (privilege advanced).. Use "cluster show" to verify the health of all nodes, ensure node "ha-cdot-02" is healthy and then set the default boot image using "system image modify". Contact support personnel if additional assistance is required. ha-cdot::> ha-cdot::> set -privilege ad admin advanced ha-cdot::> set -privilege advanced Warning: These advanced commands are potentially dangerous; use them only when directed to do so by NetApp personnel. Do you want to continue? {y|n}: y ha-cdot::*> storage aggregate 64bit-upgrade 32bit-disable Warning: You will no longer be able to create 32-bit aggregates after disabling 32-bit aggregate support. Do you want to continue? {y|n}: y 32-bit aggregate support is successfully disabled. ha-cdot::*> system image modify -node ha-cdot-02 -image image2 -isdefault true After a clean shutdown, image2 will be set as the default boot image on node ha-cdot-02. ha-cdot::*> storage failover show -fields auto-giveback node auto-giveback ---------- ------------- ha-cdot-01 true ha-cdot-02 true 2 entries were displayed. ha-cdot::*> storage failover modify -node * -auto-giveback false Warning: Disabling auto-giveback under cluster HA configuration will prevent the management cluster services from automatically going online under alternating-failure scenarios. Do you want to disable auto-giveback? {y|n}: y 2 entries were modified. ha-cdot::*> storage failover show -fields auto-giveback node auto-giveback ---------- ------------- ha-cdot-01 false ha-cdot-02 false 2 entries were displayed. #### Migrate lifs away from node 1 ha-cdot::*> network interface migrate-all -node ha-cdot-01 ha-cdot::*> network interface show Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home ----------- ---------- ---------- ------------------ ------------- ------- ---- ha-cdot cluster_mgmt up/up 10.192.1.130/24 ha-cdot-01 a1a-1921 true ha-cdot-01 clus1 up/up 169.254.156.24/16 ha-cdot-01 e1a true clus2 up/up 169.254.251.4/16 ha-cdot-01 e2a true mgmt1 up/up 10.192.1.131/24 ha-cdot-01 e0b true ha-cdot-02 clus1 up/up 169.254.136.101/16 ha-cdot-02 e1a true clus2 up/up 169.254.37.40/16 ha-cdot-02 e2a true mgmt1 up/up 10.192.1.132/24 ha-cdot-02 e0b true svm-demo-cifs demo-cifs_cifs_lif1 up/up 10.192.1.134/24 ha-cdot-02 a2a false demo-cifs_vlan2131_lif up/up 10.213.1.70/24 ha-cdot-02 a2a false demo-cifs_vlan2131_mgmt_lif up/up 10.213.1.71/24 ha-cdot-02 a2a false svm_lab_infra svm_lab_infra_iscsi1 up/up 10.213.8.233/24 ha-cdot-01 a1a-2138 true svm_lab_infra_mgmt_lif up/up 10.213.1.233/24 ha-cdot-02 a2a false svm_lab_infra_nfs_lif1 up/up 10.213.8.232/24 ha-cdot-02 a2a-2138 true 13 entries were displayed. ha-cdot::*> system node autosupport invoke -node ha-cdot-01 -type all -message "starting_NDU" ha-cdot::*> storage failover takeover -ofnode ha-cdot-01 ha-cdot::> storage failover show-giveback Partner Node Aggregate Giveback Status -------------- ----------------- --------------------------------------------- Warning: Unable to list entries on node ha-cdot-01. RPC: Port mapper failure - RPC: Timed out ha-cdot-02 CFO Aggregates Failed module: disk check. Check the event log ha-cdot::> storage failover show Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- ha-cdot-01 ha-cdot-02 - Waiting for giveback (HA mailboxes) ha-cdot-02 ha-cdot-01 false Previous giveback failed in module: disk check 2 entries were displayed. ha-cdot::> storage failover giveback -ofnode ha-cdot-01 -override-vetoes true Warning: Initiating a giveback with vetoes overridden will result in giveback proceeding even if the node detects outstanding issues that would make a giveback dangerous or disruptive. Do you want to continue? {y|n}: y Info: Run the storage failover show-giveback command to check giveback status. ha-cdot::> ha-cdot::> storage failover show Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- ha-cdot-01 ha-cdot-02 - Waiting for giveback (HA mailboxes) ha-cdot-02 ha-cdot-01 false Connected to ha-cdot-01, Takeover is not possible: Failover partner node is booting, NVRAM log not synchronized 2 entries were displayed. ha-cdot::> storage failover show Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- ha-cdot-01 ha-cdot-02 false Connected to ha-cdot-02. Waiting for cluster applications to come online on the local node., Takeover is not possible: The version of software running on each node of the SFO pair is incompatible, NVRAM log not synchronized ha-cdot-02 ha-cdot-01 false Connected to ha-cdot-01, Takeover is not possible: The version of software running on each node of the SFO pair is incompatible, NVRAM log not synchronized 2 entries were displayed. ha-cdot::> set -privilege advanced Warning: These advanced commands are potentially dangerous; use them only when directed to do so by NetApp personnel. Do you want to continue? {y|n}: y ha-cdot::*> system node upgrade-revert show -node ha-cdot-01 Node: ha-cdot-01 Status: complete Status Message: The upgrade is complete. Vers Phase Status Upgrade Phase Status Message ---- ---------- -------- ------------------------------------------------------ 320 pre-root applied No upgrade is required for this phase. 320 pre-apps applied No upgrade is required for this phase. 320 post-apps applied Upgrade successful. 400 pre-root applied No upgrade is required for this phase. 400 pre-apps applied Upgrade successful. 400 post-apps applied Upgrade successful. 6 entries were displayed. ha-cdot::*> ha-cdot::*> system node autosupport invoke -node ha-cdot-02 -type all -message "starting_NDU" ha-cdot::*> exit Goodbye Connection to 10.192.1.132 closed. jk-47:~ jkulm$ ssh admin@10.192.1.130 Warning: The cluster is in a mixed version state. Update all of the nodes to the same version as soon as possible. ha-cdot::> storage failover takeover -ofnode ha-cdot-02 -option allow-version-mismatch Warning: Initiating a takeover while the partner is running a mismatched Data ONTAP version is not recommended, unless you are performing a non-disruptive upgrade. Do you want to continue? {y|n}: y ha-cdot::> storage failover show Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- ha-cdot-01 ha-cdot-02 false Takeover scheduled. ha-cdot-02 has relocated its SFO aggregates in preparation of takeover. ha-cdot-02 ha-cdot-01 false Pending shutdown 2 entries were displayed. ha-cdot::> storage failover show Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- ha-cdot-01 ha-cdot-02 false In takeover ha-cdot-02 ha-cdot-01 - Unknown 2 entries were displayed. ha-cdot::> storage failover show-takeover Node Node Status Aggregate Takeover Status ---------- --------------------- -------------- ------------------------------- ha-cdot-01 Partner has relocated its aggregates. In takeover. aggr2_cdot02 Done CFO aggregates Done. Warning: Unable to list entries on node ha-cdot-02. RPC: Port mapper failure - RPC: Unable to receive 2 entries were displayed. ha-cdot::> storage failover giveback -ofnode ha-cdot-02 Info: Run the storage failover show-giveback command to check giveback status. ha-cdot::> ha-cdot::> ha-cdot::> storage failover show-giveback Partner Node Aggregate Giveback Status -------------- ----------------- --------------------------------------------- ha-cdot-01 CFO Aggregates Done aggr2_cdot02 Not attempted yet ha-cdot-02 No aggregates to give back 3 entries were displayed. ha-cdot::> storage failover show Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- ha-cdot-01 ha-cdot-02 true Connected to ha-cdot-02, Partial giveback ha-cdot-02 ha-cdot-01 true Connected to ha-cdot-01. Waiting for cluster applications to come online on the local node. 2 entries were displayed. ha-cdot::> storage failover show-giveback Partner Node Aggregate Giveback Status -------------- ----------------- --------------------------------------------- ha-cdot-01 CFO Aggregates Done aggr2_cdot02 In progress module: wafl ha-cdot-02 No aggregates to give back 3 entries were displayed. ha-cdot::> storage failover show-giveback Partner Node Aggregate Giveback Status -------------- ----------------- --------------------------------------------- ha-cdot-01 No aggregates to give back ha-cdot-02 No aggregates to give back 2 entries were displayed. ha-cdot::> storage failover show Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- ha-cdot-01 ha-cdot-02 true Connected to ha-cdot-02 ha-cdot-02 ha-cdot-01 true Connected to ha-cdot-01 2 entries were displayed. ha-cdot::*> system node upgrade-revert show -node ha-cdot-02 Node: ha-cdot-02 Status: complete Status Message: The upgrade is complete. Vers Phase Status Upgrade Phase Status Message ---- ---------- -------- ------------------------------------------------------ 320 pre-root applied No upgrade is required for this phase. 320 pre-apps applied No upgrade is required for this phase. 320 post-apps applied Upgrade successful. 400 pre-root applied No upgrade is required for this phase. 400 pre-apps applied Upgrade successful. 400 post-apps applied Upgrade successful. 6 entries were displayed. ha-cdot::*> network interface revert * 1 entry was acted on. ha-cdot::*> system node autosupport invoke -node ha-cdot-02 -type all -message "finishing_NDU" The AutoSupport was successfully invoked on node "ha-cdot-02". To view the status of the AutoSupport, use the "system node autosupport history show" command. Note: It may take several minutes for the AutoSupport to appear in the history list. ha-cdot::*> ha-cdot::*> ha-cdot::*> system node image show Is Is Install Node Image Default Current Version Date -------- ------- ------- ------- ------------------------- ------------------- ha-cdot-01 image1 true true 8.3RC1 11/13/2014 14:31:46 image2 false false 8.2.2 10/1/2014 02:48:12 ha-cdot-02 image1 false false 8.2.2 10/1/2014 14:30:08 image2 true true 8.3RC1 11/13/2014 14:43:48 4 entries were displayed. ha-cdot::*> storage failover modify -node * -auto-giveback true 2 entries were modified. ha-cdot::*> storage failover show -fields auto-giveback node auto-giveback ---------- ------------- ha-cdot-01 true ha-cdot-02 true |
This is what you would see from the console or service processor > console, during the takeover and giveback reboot
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 |
jk-47:~ jkulm$ ssh admin@10.192.1.133 admin@10.192.1.133's password: Permission denied, please try again. admin@10.192.1.133's password: SP ha-cdot-01> SP ha-cdot-01> SP ha-cdot-01> SP ha-cdot-01> ? date - print date and time exit - exit from the SP command line interface events - print system events and event information help - print command help priv - show and set user mode sp - commands to control the SP rsa - commands for Remote Support Agent system - commands to control the system version - print Service Processor version SP ha-cdot-01> system console Type Ctrl-D to exit. SP-login: admin Password: ***************************************************** * This is a SP/RLM console session. Output from the * * serial console is also mirrored on this session. * ***************************************************** ha-cdot::> ######################### DURING TAKEOVER #################################### ha-cdot::> Waiting for PIDS: /usr/sbin/ypbind 666. Waiting for PIDS: 632. Setting default boot image to image1... done. Terminated . Uptime: 43d0h12m38s Top Shutdown Times (ms): {if_reset=1860, shutdown_wafl=259(multivol=0, sfsr=0, abort_scan=0, snapshot=0, start=95, sync1=30, sync2=4, mark_fs=130), wafl_sync_tagged=167, shutdown_raid=62, shutdown_fm=22} Shutdown duration (ms): {CIFS=3382, NFS=3382, ISCSI=26188, FCP=3382} HALT: HA partner has taken over (ic) on Thu Nov 13 14:51:14 UTC 2014 System rebooting... � Phoenix TrustedCore(tm) Server Copyright 1985-2006 Phoenix Technologies Ltd. All Rights Reserved BIOS version: 5.3.0 Portions Copyright (c) 2007-2014 NetApp, Inc. All Rights Reserved CPU = 2 Processors Detected, Cores per Processor = 2 Intel(R) Xeon(R) CPU E5240 @ 3.00GHz Intel(R) Xeon(R) CPU E5240 @ 3.00GHz Testing RAM 512MB RAM tested 20480MB RAM installed 6144 KB L2 Cache System BIOS shadowed USB 2.0: MICRON eUSB DISK BIOS is scanning PCI Option ROMs, this may take a few seconds... ................... Boot Loader version 3.6 Copyright (C) 2000-2003 Broadcom Corporation. Portions Copyright (C) 2002-2014 NetApp, Inc. All Rights Reserved. CPU Type: Intel(R) Xeon(R) CPU E5240 @ 3.00GHz Starting AUTOBOOT press Ctrl-C to abort... Loading X86_64/freebsd/image1/kernel:0x100000/7946496 0x894100/4206472 Entry at 0x80171230 Loading X86_64/freebsd/image1/platform.ko:0xc98000/1985879 0xe7d000/288704 0xec37c0/272656 Starting program at 0x80171230 NetApp Data ONTAP 8.3RC1 Copyright (C) 1992-2014 NetApp. All rights reserved. ******************************* * * * Press Ctrl-C for Boot Menu. * * * ******************************* Running the upgrade system bootarg.upgrade="true" bootarg.from.version="8.2.2" bootarg.to.version="8.3RC1" usage: route [-dnqtv] command [[modifiers] args] Invalid netmask (0.0.0.0) entered. Please try again with a valid netmask. original max threads=40, original heap size=41943040 bip_nitro Virtual Size Limit=94901862 Bytes bip_nitro: user memory=957644800, actual max threads=54, actual heap size=57042534 Ipspace "iwarp-ipspace" created Nov 13 14:53:38 [ha-cdot-01:kern.version.change:notice]: Data ONTAP kernel version was changed from 8.2.2 to 8.3RC1. Reservation conflict found on this node's disks! Local System ID: 1574031045 Nov 13 14:53:44 [ha-cdot-01:disk.ioMediumError:warning]: Medium error on disk 0a.81.17: op 0x28:00000008:0008 sector 8 SCSI:medium error - Unrecovered read error - If the disk is in a RAID group, the subsystem will attempt to reconstruct unreadable data (3 11 ff db) (7) [NETAPP X302_WVULC01TSSM NA04] S/N [WD-WCAW30891044] WAFL CPLEDGER is enabled. Checklist = 0x7ff841ff Press Ctrl-C for Maintenance menu to release disks. Nov 13 14:53:44 [ha-cdot-01:disk.ioFailed:error]: I/O operation failed despite several retries. Nov 13 14:53:44 [ha-cdot-01:disk.ioMediumError:warning]: Medium error on disk 0a.81.17: op 0x28:00000010:0008 sector 16 SCSI:medium error - Unrecovered read error - If the disk is in a RAID group, the subsystem will attempt to reconstruct unreadable data (3 11 ff db) (216) [NETAPP X302_WVULC01TSSM NA04] S/N [WD-WCAW30891044] Nov 13 14:53:44 [ha-cdot-01:disk.ioFailed:error]: I/O operation failed despite several retries. Nov 13 14:53:44 [ha-cdot-01:disk.ioMediumError:warning]: Medium error on disk 0a.81.17: op 0x28:0000a3e8:0018 sector 41960 SCSI:medium error - Unrecovered read error - If the disk is in a RAID group, the subsystem will attempt to reconstruct unreadable data (3 11 ff db) (11) [NETAPP X302_WVULC01TSSM NA04] S/N [WD-WCAW30891044] Nov 13 14:53:44 [ha-cdot-01:disk.ioFailed:error]: I/O operation failed despite several retries. Nov 13 14:53:44 [ha-cdot-01:diskown.errorReadingOwnership:warning]: error 26 (unrecoverable I/O error on disk sector) while reading ownership on disk 0a.81.17 (S/N WD-WCAW30891044) Nov 13 14:53:44 [ha-cdot-01:disk.ioMediumError:warning]: Medium error on disk 0a.81.17: op 0x28:0000a3f0:0008 sector 41968 SCSI:medium error - Unrecovered read error - If the disk is in a RAID group, the subsystem will attempt to reconstruct unreadable data (3 11 ff db) (6) [NETAPP X302_WVULC01TSSM NA04] S/N [WD-WCAW30891044] Nov 13 14:53:44 [ha-cdot-01:disk.ioFailed:error]: I/O operation failed despite several retries. Nov 13 14:53:44 [ha-cdot-01:diskown.errorDuringIO:error]: error 26 (unrecoverable I/O error on disk sector) on disk 0a.81.17 (S/N WD-WCAW30891044) while reading individual disk ownership area Nov 13 14:53:44 [ha-cdot-01:monitor.chassisPower.degraded:warning]: Chassis power is degraded: Power Supply Status Critical: PSU2. Nov 13 14:53:44 [ha-cdot-01:callhome.chassis.power:error]: Call home for CHASSIS POWER DEGRADED: Power Supply Status Critical: PSU2. Disk reservations have been released Nov 13 14:54:02 [ha-cdot-01:cf.fm.kernelMismatch:warning]: Failover monitor: possible kernel mismatch detected local 'NetApp//8.3RC1', partner 'NetApp//8.2.2' Nov 13 14:54:02 [ha-cdot-01:cf.fm.versionMismatch:error]: Failover monitor: nvram toc version mismatch detected: 34/29 Nov 13 14:54:02 [ha-cdot-01:cf.fm.versionMismatch:error]: Failover monitor: wafl fsinfo version mismatch detected: 24870/23589 Nov 13 14:54:02 [ha-cdot-01:cf.fm.versionMismatch:error]: Failover monitor: nvram nvlog version mismatch detected: 451/396 Nov 13 14:54:02 [ha-cdot-01:cf.fm.versionMismatch:error]: Failover monitor: raid version mismatch detected: 13/12 Nov 13 14:54:12 [ha-cdot-01:cf.disk.inventory.ver.notok:warning]: cfdisk: The local node did not support the partner's disk inventory message version. The disk inventory check will not be performed during giveback. fm_run0: No response received for the systemID message from partnerWaiting for giveback...(Press Ctrl-C to abort wait) Waiting for giveback...(Press Ctrl-C to abort wait)Continuing boot... add host 127.0.10.1: gateway 127.0.20.1 Nov 13 15:10:36 [ha-cdot-01:cf.fm.discardNvram:notice]: Failover monitor: node was previously taken over, nvram may be discarded Nov 13 15:10:37 [ha-cdot-01:kern.syslog.msg:notice]: The system was down for 5 seconds Nov 13 15:10:37 [ha-cdot-01:tar.csum.mismatch:notice]: Stored checksum 503940345 does not match calculated checksum 82723410, extracting local://mnt/prestage/mroot.tgz. Nov 13 15:10:37 [ha-cdot-01:cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of ha-cdot-02 disabled (version mismatch). Nov 13 15:10:37 [ha-cdot-01:cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of ha-cdot-01 by ha-cdot-02 disabled (version mismatch). Nov 13 15:10:38 [ha-cdot-01:tar.csum.mismatch:notice]: Stored checksum 4158797859 does not match calculated checksum 1136122759, extracting local://mnt/prestage/pmroot.tgz. ha-cdot-01 Nov 13 15:10:39 [ha-cdot-01:snmp.agent.msg.access.denied:warning]: Permission denied for SNMPv3 requests from root. Reason: Password is too short (SNMPv3 requires at least 8 characters). Nov 13 15:11:00 [ha-cdot-01:monitor.globalStatus.critical:CRITICAL]: Controller failover of ha-cdot-02 is not possible: version mismatch. Power Supply Status Critical: PSU2. Nov 13 15:11:04 [ha-cdot-01:pvif.allLinksDown:CRITICAL]: a1a: all links down Thu Nov 13 15:12:17 UTC 2014 login: SP-login: SP-login: admin Password: ***************************************************** * This is an SP console session. Output from the * * serial console is also mirrored on this session. * ***************************************************** *********************** ** SYSTEM MESSAGES ** *********************** The upgrade of this node is in progress or not completed. The ability to provide data service to clients is not affected while the upgrade completes. You can check on the status of the upgrade by running "system node upgrade-revert show" in advanced privilege mode. The status for this node should be listed as 'complete'. If the upgrade has stopped, you can restart the upgrade by running "system node upgrade-revert upgrade" in advanced privilege mode. If this command does not complete the node's upgrade, contact technical support immediately. The node will be ready for management operations once the upgrade is completed successfully. Warning: The cluster is in a mixed version state. Update all of the nodes to the same version as soon as possible. |
I hope you find this interesting. I will annotate the above with comments about why I run stuff in each step as I have time.
@JK47theweapon nice post. Q. do you have any info on setting up root-data partitioning? Wondering if this only on new preloaded filers..
Boot maint mode. Remove disk ownerships. Select option 4, then type wipeconfig. On internal disk systems, it’ll set it up as ADP