{"id":28,"date":"2012-02-28T21:53:29","date_gmt":"2012-02-28T20:53:29","guid":{"rendered":"http:\/\/www.b00z.nl\/blog\/?p=28"},"modified":"2012-02-28T21:53:29","modified_gmt":"2012-02-28T20:53:29","slug":"setup-srx1400-in-cluster-over-layer2-switching-network","status":"publish","type":"post","link":"https:\/\/www.b00z.nl\/blog\/2012\/02\/setup-srx1400-in-cluster-over-layer2-switching-network\/","title":{"rendered":"Setup SRX1400 in cluster over layer2 switching network"},"content":{"rendered":"

This post is about my experience of setting up two SRX1400 nodes in a cluster, geographically separated by a layer2 switching network.<\/p>\n

The lab setup had to be redundant and spread over two locations. So I updated both SRX’en to JunOS 11.2R4.3, this version supports clustering and redundant fabric and control links.<\/p>\n

This post consists of the following topics:
\n1. Connecting the SRX’s to the switches
\n2. Configuring the switchports for the fabric, control links and the 2 Gbps traffic links
\n3. Configuring the cluster
\n4. Troubleshooting several issues with SRX1400 clustering<\/p>\n

I’ll be using not just plain traffic and basic configuration, but I’ll be also using vlan tagging to reduce cabling in the future and will be configuring a routing instance just to be prepared for the future.<\/p>\n

Also I’m paying attention to some issues I encountered during my tests. Things you don’t find in the Juniper documentation. ;o) I’m talking about different VLAN’s for control links, and a issue with the current JunOS releases and using T-SFP modules for your control links causing your control links to fail during setup.<\/p>\n

So the complete setup will be like below.
\n\"\"<\/p>\n

1. Connecting the SRX’s to the switches<\/p>\n

For this lab I’ve setup the connections as follows:
\n2x 1Gb UTP for control links
\n2x 1Gb UTP for fabric
\n2×2 1Gb UTP for 2Gb traffic IN (untrust) and 2Gb traffic OUT (trust)<\/p>\n

So if we zoom in on one node, the connections will be like this:<\/p>\n

\"\"<\/p>\n

Red is 2Gb inbound, untrust
\nGreen is 2Gb outbound, trust
\nYellow is 2x fabric
\nBlue is 2x Control link
\n2. Configuring the switchports for the fabric, control links and the 2 Gbps traffic links<\/p>\n

One thing your need to be aware of, and I didn’t find it in the documentation, is that where the fabric links can be in one single VLAN, the two control links need to be in their own VLAN! When you fail to do so, your cluster will initially start, but then fail.<\/p>\n

I’ve set up 5 VLAN’s:
\nvlan100 = Untrust
\nvlan200 = Trust
\nvlan300 = Fabric
\nvlan301 = Control link 1
\nvlan302 = Control link 2<\/p>\n

Then for the untrust and trust aggregated links I’ve used:
\nae1, untrust node0
\nae2, trust node0
\nae3, untrust node1
\nae4, trust node1<\/p>\n

The fabric ports require jumbo frames, so we set the maximum MTU of 9216 on those ports. NOTE! Disable any other security feature like RSTP, Stormcontrol and igmp-snooping on the ports used by the control links. Just to be sure I also recommend to disable these features on the fabric ports.<\/p>\n

Below you’ll see the example EX4200 port configurations for the mentioned switchports.<\/p>\n

Fabric:\r\n{master:0}\r\nroot@labswitch1> show configuration interfaces ge-0\/0\/1\r\nmtu 9216;\r\nunit 0 {\r\n\u00a0\u00a0\u00a0 family ethernet-switching {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 port-mode access;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 vlan {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 members vlan300;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 }\r\n}\r\n\r\n{master:0}\r\nroot@labswitch1> show configuration interfaces ge-0\/0\/2\r\nmtu 9216;\r\nunit 0 {\r\n\u00a0\u00a0\u00a0 family ethernet-switching {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 port-mode access;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 vlan {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 members vlan300;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 }\r\n}<\/pre>\n
Control1:\r\n{master:0}\r\nroot@labswitch1> show configuration interfaces ge-0\/0\/3\r\nunit 0 {\r\n\u00a0\u00a0\u00a0 family ethernet-switching {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 port-mode access;\r\n        vlan {\r\n            members vlan301;\r\n        }\r\n\u00a0\u00a0\u00a0 }\r\n}<\/pre>\n
Control2:\r\n{master:0}\r\nroot@labswitch1> show configuration interfaces ge-0\/0\/4\r\nunit 0 {\r\n\u00a0\u00a0\u00a0 family ethernet-switching {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 port-mode access;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 vlan {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 members vlan302;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 }\r\n}<\/pre>\n
Untrust node0\/1:\r\n{master:0}\r\nroot@labswitch1> show configuration interfaces ae1\r\ndescription \"node0 Untrust\";\r\nunit 0 {\r\n\u00a0\u00a0\u00a0 family ethernet-switching {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 port-mode trunk;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 vlan {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 members vlan100;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 }\r\n}\r\n\r\n{master:0}\r\nroot@labswitch1> show configuration interfaces ae2\r\ndescription \"node1 Untrust\";\r\nunit 0 {\r\n\u00a0\u00a0\u00a0 family ethernet-switching {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 port-mode trunk;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 vlan {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 members vlan100;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 }\r\n}<\/pre>\n
Trust node0\/1:\r\n{master:0}\r\nroot@labswitch1> show configuration interfaces ae3\r\ndescription \"node0 Trust\";\r\nunit 0 {\r\n\u00a0\u00a0\u00a0 family ethernet-switching {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 port-mode trunk;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 vlan {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 members vlan200;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 }\r\n}\r\n\r\n{master:0}\r\nroot@labswitch1> show configuration interfaces ae4\r\ndescription \"node1 trust\";\r\nunit 0 {\r\n\u00a0\u00a0\u00a0 family ethernet-switching {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 port-mode trunk;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 vlan {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 members vlan200;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 }\r\n}<\/pre>\n
Having the switch connections set up properly, we continue configuring the cluster itself.
\n3. Configuring the cluster<\/p>\n
For the configuration, mostly TN10 has been used: http:\/\/kb.juniper.net\/InfoCenter\/index?page=content&id=TN10<\/p>\n
While configuring the cluster, there is one bug in JunOS 11.2 to keep in mind. When you use T-SFP modules for your control links, and you enable the cluster with:
\nroot@node0> set chassis cluster cluster-id {clusterID} node {nodeID}
\nYour control link might stay down. Even when the switch shows link is up. But in fact you’re missing a HA interface and the link lights will be off on the SRX side. I’ll get back to this later on.<\/p>\n
First execute the following command on both nodes to enable the cluster.
\n@Node0:
\nroot@node0> set chassis cluster cluster-id 1 node 0 reboot<\/p>\n
@Node1:
\nroot@node1> set chassis cluster cluster-id 1 node 1 reboot<\/p>\n
After both nodes have rebooted, your base for the cluster is set. Your both control links should be up. Let’s check.<\/p>\n
{primary:node0}\r\nroot@node0> show chassis cluster interfaces\r\nControl link status: Up\r\n\r\nControl interfaces:\r\n\u00a0\u00a0\u00a0 Index\u00a0\u00a0 Interface\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Status\r\n\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 em0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up\r\n\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 em1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up<\/pre>\n
Your fabric will be down, because we haven’t configured anything for that yet. Please referer to TN10 for the further steps. This because they might update those steps.<\/p>\n
Getting back to the issue with JunOS 11.2, when you first set up the cluster (when you issue the ‘set chassis cluster’ command) and you’re using T-SFP modules for the control links, be sure to check that the HA interface are present. Check this with:<\/p>\n
{primary:node0}\r\nroot@node0> show chassis ethernet-switch\r\nnode0:\r\n-------------------------------------------------------\r\nDisplaying summary for switch 0\r\nLink is good on GE port 3 connected to device: RE-GigE\r\n\u00a0 Speed is 1000Mb\r\n\u00a0 Duplex is full\r\n\u00a0 Autonegotiate is Disabled\r\n\r\nLink is good on GE port 7 connected to device: FPC1\r\n\u00a0 Speed is 1000Mb\r\n\u00a0 Duplex is full\r\n\u00a0 Autonegotiate is Disabled\r\n\r\nLink is good on GE port 20 connected to device: CPP-GigE\r\n\u00a0 Speed is 1000Mb\r\n\u00a0 Duplex is full\r\n\u00a0 Autonegotiate is Disabled\r\n\r\nLink is good on GE port 22 connected to device: HA-GigE<\/strong>\r\n\u00a0 Speed is 1000Mb\r\n\u00a0 Duplex is full\r\n\u00a0 Autonegotiate is Enabled\r\n\r\nnode1:\r\n--------------------------------------------------------\r\nDisplaying summary for switch 0\r\nLink is good on GE port 3 connected to device: RE-GigE\r\n\u00a0 Speed is 1000Mb\r\n\u00a0 Duplex is full\r\n\u00a0 Autonegotiate is Disabled\r\n\r\nLink is good on GE port 7 connected to device: FPC1\r\n\u00a0 Speed is 1000Mb\r\n\u00a0 Duplex is full\r\n\u00a0 Autonegotiate is Disabled\r\n\r\nLink is good on GE port 20 connected to device: CPP-GigE\r\n\u00a0 Speed is 1000Mb\r\n\u00a0 Duplex is full\r\n\u00a0 Autonegotiate is Disabled\r\n\r\nLink is good on GE port 22 connected to device: HA-GigE<\/strong>\r\n\u00a0 Speed is 1000Mb\r\n\u00a0 Duplex is full\r\n\u00a0 Autonegotiate is Enabled<\/pre>\n
When one or both of the HA-GigE interfaces are missing, you have two options to fix it:
\n1. Cold reboot the node. Just completely remove the power of the node and start it again.
\n2. Remove and reinsert the T-SFP module. When you do this, the link lights will be lit again.<\/p>\n
This issue arises only when you first setup your cluster. Future reboots etc. won’t bring this issue back.<\/p>\n
Note: This issue is only with T-SFP module. When you use SFP lasers, this problem does not occur.<\/p>\n
Well, now we’ve our cluster fully operational and up and running!<\/p>\n
{primary:node0}\r\nroot@node0> show chassis cluster status\r\nCluster ID: 1\r\nNode\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Priority\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Status\u00a0\u00a0\u00a0 Preempt\u00a0 Manual failover\r\n\r\nRedundancy group: 0 , Failover count: 1\r\n\u00a0\u00a0\u00a0 node0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 100\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 primary\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 no\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 no\r\n\u00a0\u00a0\u00a0 node1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 secondary\u00a0\u00a0\u00a0\u00a0\u00a0 no\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 no\r\n\r\nRedundancy group: 1 , Failover count: 1\r\n\u00a0\u00a0\u00a0 node0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 100\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 primary\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 no\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 no\r\n\u00a0\u00a0\u00a0 node1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 secondary\u00a0\u00a0\u00a0\u00a0\u00a0 no\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 no\r\n\r\n{primary:node0}\r\nroot@node0> show chassis cluster interfaces\r\nControl link status: Up\r\n\r\nControl interfaces:\r\n\u00a0\u00a0\u00a0 Index\u00a0\u00a0 Interface\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Status\r\n\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 em0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up\r\n\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 em1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up\r\n\r\nFabric link status: Up\r\n\r\nFabric interfaces:\r\n\u00a0\u00a0\u00a0 Name\u00a0\u00a0\u00a0 Child-interface\u00a0\u00a0\u00a0 Status\r\n\u00a0\u00a0\u00a0 fab0\u00a0\u00a0\u00a0 ge-0\/0\/4\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up\r\n\u00a0\u00a0\u00a0 fab0\u00a0\u00a0\u00a0 ge-0\/0\/5\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up\r\n\u00a0\u00a0\u00a0 fab1\u00a0\u00a0\u00a0 ge-4\/0\/4\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up\r\n\u00a0\u00a0\u00a0 fab1\u00a0\u00a0\u00a0 ge-4\/0\/5\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up\r\n\r\nRedundant-ethernet Information:\r\n\u00a0\u00a0\u00a0 Name\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Status\u00a0\u00a0\u00a0\u00a0\u00a0 Redundancy-group\r\n\u00a0\u00a0\u00a0 reth0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1\r\n\u00a0\u00a0\u00a0 reth1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Up\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1<\/pre>\n
This is just a basic setup. Next page is about all the issues I encountered during my “quest”.
\n<\/p>\n
4. Troubleshooting several issues with SRX1400 clustering<\/p>\n
While I was working on my lab setup of the SRX1400 clustering over layer2 switching network, I encountered several issues. The issues I encountered where:
\n1. Failing control links
\n2. Flipping FPC’s (FPC’s keeping rebooting)<\/p>\n
The failing control links are solved in the following new JunOS version: 11.4R3 and 12.1R1<\/p>\n
The other issue was that I suddenly noticed that the FPC’s of a node keeped rebooting every X minutes. This was causing a really unstable cluster. Everytime I generated lots of traffic, it would even trigger failovers. The FPC’s are also generating core dumps.
\nAfter contact with JTAC, they discovered that there is a PR for the JunOS version I was using: http:\/\/kb.juniper.net\/InfoCenter\/index?page=content&id=KB22616&cat=security_products&actp=LIST&smlogin=true<\/p>\n
So they recommended me to upgrade to a newer version. After upgrading to the newer version the problems were indeed solved. As JTAC says:
\nThis issue is fixed in 11.2R5 (yet to release).
\nThe recommendation is to upgrade to 11.4R1.<\/p>\n
At the time of writing 11.2R5 is already released.<\/p>\n
Well, that was it for now. If I encounter any new things regarding SRX1400 in clustering, I’ll update this post. So keep following my posts. ;o)<\/p>\n","protected":false},"excerpt":{"rendered":"
This post is about my experience of setting up two SRX1400 nodes in a cluster, geographically separated by a layer2 switching network. The lab setup had to be redundant and spread over two locations. So I updated both SRX’en to … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,7],"tags":[15,13,17,12,20,16,22,18,14,19,21],"class_list":["post-28","post","type-post","status-publish","format-standard","hentry","category-howto","category-juniper","tag-15","tag-cluster","tag-geographically","tag-juniper-2","tag-junos","tag-layer2","tag-routing-instance","tag-separated","tag-srx","tag-t-sfp","tag-vlan"],"_links":{"self":[{"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/posts\/28","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/comments?post=28"}],"version-history":[{"count":12,"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/posts\/28\/revisions"}],"predecessor-version":[{"id":42,"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/posts\/28\/revisions\/42"}],"wp:attachment":[{"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/media?parent=28"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/categories?post=28"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.b00z.nl\/blog\/wp-json\/wp\/v2\/tags?post=28"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}