SevOne NMS 5.4 Best Practices Guide - Cluster, Peer, and HSA

SevOne Scalability

SevOne NMS is a scalable system that is able to suit any network needs. SevOne appliances work together in an environment that uses a transparent peer-to-peer technology. You can peer multiple SevOne appliances together to create a SevOne cluster that can monitor an unlimited number of network objects and flow data. There is no limit to the number of peers that can work together in a cluster.

There are several models of the appliances that run the SevOne software.

• PAS - Performance Appliance System (PAS) appliances can be rated to monitor up to 200,000 network objects.

• DNC - Dedicated NetFlow Collector (DNC) appliances monitor flow technologies.

• HSA - Hot Standby Appliances (HSA) enable you to create a pair of appliances that act as one peer to provide redundancy.

• PLA – Performance Log Appliance (PLA) provides deep insight into the performance of both hardware and the services or applications running above.

SevOne appliances scale linearly to monitor the world's largest networks. Each SevOne appliance is both a collector and a reporter. In a multi-peer cluster, all appliances are aware of which peer monitors each device and can efficiently route data requests to the user. The system is flexible enough to enable indicators from different devices to appear on the same graph when different SevOne NMS peers monitor the devices.

The peer-to-peer architecture not only scales linearly, without loss of efficiency as you add additional peers, but actually gets faster. When a SevOne NMS peer receives a request from one of its peers, it processes the request locally and sends the requesting peer only the data it needs to complete the report. Thus if a report spans multiple peers, they all work on the report simultaneously to retrieve the results much faster than any single appliance can.

You can create redundancy for each peer with a Hot Standby Appliance that works in tandem with the active appliance to create a peer appliance pair. If the active appliance in the Hot Standby Appliance peer pair fails, the passive appliance in the peer pair takes over. The passive appliance in the Hot Standby Appliance peer pair maintains a redundant copy of all poll data and configuration data for the peer pair. If a failover occurs, the transition from passive appliance to active appliance is transparent to all users and all polling and reporting continues seamlessly.

Each SevOne cluster has a Cluster Master peer. The cluster master peer is the SevOne appliance that stores the master copy of Cluster Manager settings, security settings, and other global settings in the config database. All other active peers in your SevOne cluster pull the configuration data from the cluster master peer config database. Some security settings and device edit functions are dependent upon the communication between the active peers in the cluster and the cluster master peer. The cluster master appliance hardware and software is no different from any other peer appliance so you can designate any appliance to be the cluster master. Considerations for the cluster master are geographic location and data center stability that affect latency and power supply which should be considered for any computer network implementation.

From the user point of view, there is only one peer because each peer in the cluster presents all data, no matter which peer collects it, with no reference to the peer that collects the data. You can log on to any SevOne NMS peer in your cluster and retrieve data that spans any or all devices in your network.

All SevOne communication protocols assume a single hop adjacency which means that a request or replication must be directly sent to the IP address of the destination SevOne peer and cannot be routed in any way by the members of the cluster.

All peers must be able to communicate with the cluster master peer but the SevOne cluster is designed to be able to gracefully operate in a degraded state when one or more of the peers are disconnected from the rest of the cluster.

• A peer that cannot communicate with the cluster master peer can run reports that return results for all data available on the peers with which it maintains communication.

• A peer that cannot communicate with the cluster master cannot make any configuration changes for the duration of being disconnected.

• The peers in the cluster that remain in communication with the cluster master can make configuration changes and can run report for all data available on peers with which they maintain communication.

There are two types of SevOne cluster deployment architectures:

• Full Mesh - All peers can talk to each other. Recommended for enterprises and service providers to monitor their internal network.

• Hub and Spoke - All peers can talk to the cluster master and you can define subsets of the other interconnected peers. Useful for Managed Service Providers. Hub and spoke implementations provide the following:

  • Reports run from the cluster master peer return fully global result sets.

  • Reports run from a peer in a partially connected subset of peers return result sets from itself, the cluster master peer, and any other peers with which the peer can directly communicate.

  • Most configuration changes at both the peer level and the cluster level can be accomplished from all peers.

  • Updates and other administrative functions that affect ALL peers must be run from the cluster master peer.

SevOne-peer-add

The SevOne-peer-add command line script enables you to add a peer to your SevOne cluster. The new peer must be a clean box that contains no data in the data database or the config database.

1. Follow the steps in the SevOne NMS Installation and Implementation Guide to rack up the appliance and to perform the steps in the config shell to give the peer an IP address.

2. On the cluster master peer, access the command line prompt and enter: SevOne-peer-add

3. Follow the prompts to enter the new peer's IP address and the standard SevOne internal password.

Terminology

• Primary Appliance - Implemented to be the active, normal, polling appliance. If the primary appliance fails, it is still the primary appliance but it becomes the passive appliance.

• Secondary Appliance - Implemented to be the passive appliance in a Hot Standby Appliance peer pair. If the active appliance fails, it is still the secondary appliance but it assumes the active role.

• Active Appliance - The currently polling appliance. Upon initial setup the primary appliance is the active appliance. If the primary appliance fails, the secondary appliance becomes the active appliance in the peer pair.

• Passive Appliance - The appliance that currently replicates from the active appliance. Upon initial setup the secondary appliance is the passive appliance.

• Neighbor - The other appliance in the peer pair. The primary appliance's neighbor is the secondary appliance and vice versa.

• Fail Over – From the perspective of the active appliance, fail over occurs when the currently passive appliance assumes the active role.

• Take Over – From the perspective of the passive appliance, take over occurs when the passive appliance assumes the active role.

• Split Brain – When both appliances in a peer pair are active or both appliances in a peer pair are passive this is known as split brain. Split brain can occur when the communication between the appliances is interrupted and the passive appliance becomes active. When an administrative user logs on, an administrative message appears to let you know that the split brain condition exists. See the Troubleshoot Cluster Management chapter for details.

Both appliances in the peer pair should have the same hardware/software configuration to prevent performance problems in the event of a failover (for example, a 60K HSA for a 60K PAS). Each Hot Standby Appliance peer pair can have only the two appliances in the peer pair. A Hot Standby Appliance cannot have a Hot Standby Appliance attached to it.

In a Hot Standby Appliance peer pair implementation, the role of the active appliance in the peer pair is to:

• Provide a full web interface to the entire SevOne NMS cluster.

• Update the cluster master peer with configuration changes.

• Pull config database changes from the cluster master peer.

• Communicate with other peers to get any non-local data.

• Discover/delete the devices you assign to the peer.

• Poll data from the assigned devices.

The role of the passive appliance in the peer pair is to:

• Replicate the config database information and the data database information from the active appliance.

• Switch to the active role when the current active appliance is not reachable.

An HSA implementation creates an appliance relationship that is designed to have the passive appliance receive a continuous stream of database replication (commonly known as binary logs of database actions) from the active appliance neighbor. The passive appliance replays this stream and makes precisely the same changes that the active appliance made (i.e., record collected data, add a new device configuration, delete a monitored device and all historic data, etc.). The passive appliance performs a continuous heartbeat check to the active appliance. In the event that the passive appliance determines that the active appliance's heartbeat has gone away for any reason for a specified time period (referred to as the minimum dead time), the passive appliance assumes that the active appliance has in deed suffered a catastrophic failure and the passive appliance takes over polling at that point for its respective devices and notifies the cluster master that it has taken over for what was the active appliance.

The passive appliance in the Hot Standby Appliance peer pair can take over for the active appliance at any time. Any changes you make to the active appliance (including settings updates, new polls, or device deletions) are replicated to the passive appliance as frequently as possible.

Hot Standby Appliance Peer Pair Ports

The primary appliance and the secondary appliance in a Hot Standby Appliance peer pair need to communicate with each other to maintain a consistent environment. The appliances need to have the following ports open between each other:

• TCP 3306 - MySQL replication

• TCP 3307 - MySQL replication

• TCP 5050 - Active/passive communication

Add an Appliance to Create a Hot Standby Appliance Peer Pair

There are two ways to implement a Hot Standby Appliance peer pair: VIP and non-VIP. Each has its advantages and disadvantages, and each works best in certain implementations.

Click Through from SevOne NMS to SevOne PLA Data

In SevOne NMS you can access log data from the Alerts page.

Alerts click through to log data

In SevOne NMS you can access log data from report attachments.

Report attachment click through to log data

The Performance Log Appliance SevOne PLA software enables you to monitor log data independently from the SevOne NMS software. See the SevOne Performance Log Appliance Administration and User Guide for details.

Peer Level Cluster Management

Peer Level – At the Peer level, on the Peer Settings tab, the Primary/Secondary subtab enables you to view the IP addresses for the two appliances that act as one SevOne NMS peer in a Hot Standby Appliance (HSA) peer pair implementation. The Primary appliance is initially set up to be the active appliance. If the Primary appliance fails, it is still the Primary appliance but its role changes to the passive appliance. The Secondary appliance is initially set up to be the passive appliance. If the Primary appliance fails, the Secondary appliance is still the Secondary appliance but it becomes the active appliance. You define the appliance IP address upon initial installation and implementation. See the SevOne NMS Installation and Implementation Guide for details.

• The Primary Appliance IP Address field displays the IP address of the primary appliance.

• The Secondary Appliance IP Address field displays the IP address of the secondary appliance.

• The Virtual IP Address field appears empty unless you implement the primary appliance and the secondary appliance to share a virtual IP address (VIP HAS implementation).

• The Failover Time field enables you to enter the number of seconds for the passive appliance to wait for the active appliance to respond before the passive appliance takes over. SevOne NMS pings every 2 seconds and the timeout for a ping is 5 seconds.

Appliance Level Cluster Management

Appliance Level – At the Appliance level the appliance IP address displays. For a Hot Standby Appliance peer pair implementation two appliances appear.

• The Primary appliance appears first in the peer pair.

• The Secondary appliance appears second in the peer pair.

• The passive appliance in the peer pair displays (passive).

• The active appliance that is actively polling does not display any additional indicators.

Click on an appliance and appears above the right side on the Cluster Manager. Click to display options that are dependent on the appliance you select in the hierarchy on the left side.

• Select Device Summary to access the Device Summary for the appliance. When there are report templates that are applicable for the device, a link appears to the Device Summary along with links to the report templates.

• Select Fail Over to have the active appliance in a Hot Standby Appliance peer pair become the passive appliance in the peer pair. This option appears when you select the active appliance in a Hot Standby Appliance peer pair in the hierarchy.

• Select Take Over to have the passive appliance in a Hot Standby Appliance peer pair become the active appliance in the peer pair. This option appears when you select the passive appliance in a Hot Standby Appliance peer pair in the hierarchy.

• Select Resynchronize Data Database to have an active appliance pull the data from its own config database to its data database or to have the passive appliance in a Hot Standby Appliance peer pair pull the data from the active appliance's data database. This is the only option that appears when you select the cluster master peer's appliance in the hierarchy.

• Select Resynchronize Config Database to have an active appliance pull the data from the cluster master peer's config database to the active peer's config database or to have the passive appliance in a Hot Standby Appliance peer pair pull the data from the active appliance's config database.

• Select Rectify Split Brain to rectify situations when both appliances in a Hot Standby Appliance peer pair think they are active or both appliances think they are passive. See the Split Brain Hot Standby Appliances chapter for details.

SevOne NMS User Interface

The Cluster Manager displays the cluster hierarchy on the left and enables you to rectify split brain occurrences. From the navigation bar click the Administration menu and select Cluster Manager to display the Cluster Manager.

In the cluster hierarchy, click on the name of the peer pair that is the Hot Standby Appliance peer pair to display the two IP addresses of the two appliances that make up the peer pair. The Primary appliance displays first.

• In a split brain situation where both appliances are active, neither appliance displays Passive.

• In a split brain situation where both appliances are passive, both appliances display Passive.

Cluster Manager – Appliance Level – Split Brain – Both Passive

Select one of the affected appliances in the hierarchy on the left side. Click the to display a Rectify Split Brain option.

• When both appliances think they are passive and you select this option, the appliance for which you select this option becomes the active appliance in the Hot Standby Appliance peer pair.

• When both appliances think they are active and you select this option, the appliance for which you select this option becomes the passive appliance in the Hot Standby Appliance peer pair.

Cluster Manager – Appliance Level – Split Brain - Both Active