SevOne NMS 5.4 Quick Start Guide - Self-monitoring

CPU

When it comes to the CPU, we want to focus on the total aggregate CPU usage. Looking at the sum of the CPU cores rather than at individual cores provides an overview of the CPU activity. If there's a problem with the total aggregate usage, it may be a good idea to start looking at individual cores to determine whether it's just specific cores or all cores that are acting up. This can help us pinpoint whether the problem is related to a particular process or to the entire system.

Ideally, CPU usage will be in the following ranges:

• Idle time >= 50% - most of the time, the CPU shouldn't be doing a whole lot.

• Waiting time <= 10% - if the waiting time is consistently 10% or higher, the system is doing a little too much waiting, and we need to find out why.

Disk

When looking at disks, our two main areas of focus are free disk space and input/output (I/O). The typical SevOne appliance has three main disk components:

• sda

  • / - contains the entire operating system, including libraries, executables, etc.

  • /index - is used by the database for indexing activities. This lets us distribute our read/write requests across another disk to improve performance.

• sdb

  • /data - contains large amounts of data (covering long time spans), flow data, etc.

• fioa

  • /ioDrive - optional Fusion-io SSD, included with higher-capacity appliances.
    
    

The following are the ideal amounts of free disk space:

• /, /ioDrive >= 20% free

• /data >= 10% free - if this is too full, the database can freeze up.

SNMP reports I/O statistics for entire disks as well as the individual partitions of each disk. For self-monitoring purposes, we're interested in the entire disk. The following are three I/O states and information about what they mean:

• I/O is low - this means there's not a whole lot going on at the moment. Generally speaking, there's nothing to worry about here.

• I/O is up and down - this means that there's some reading and writing going on. This isn't cause for concern–just work as usual.

• I/O is high - if it's consistently high, this means that the disk is constantly reading and writing. This is a red flag and requires a closer look.

  One possible cause of high I/O is hot standby synchronization, which involves copying lots of data from one appliance to another. However, it's always good to check into the exact cause of high I/O.

RAID

It's important to keep an eye on RAID behavior. By monitoring RAID, you can get information about potential disk failure before it happens. You can then take appropriate action to ensure that the performance is minimally affected or not affected at all.

You'll want to keep an eye on the following things:

• RAID array state. This applies to the array. The RAID array state should be 4. Anything other than 4 is cause for concern.

• RAID disk predictive failure count. This applies to the individual disks that make up the array. Predictive failures result from trends of certain errors and act as warning signs, indicating that you may need to replace the disk.

• RAID disk firmware state. This also applies to individual disks. The firmware state should be 7. If it's not 7 on any of the disks, then there may be a problem with that disk.

Memory

There are a couple of things to keep in mind about memory as it applies to SevOne appliances. First, you'll notice pretty high memory usage. Depending on how much RAM an appliance has, it might be close to 100%. Memory usage on appliances with a greater amount of RAM may not be quite so close to the 100% mark, but it'll still be high. This is because the Linux kernel caches all kinds of stuff in RAM. For example, if you write to a file, as long as there's free space in RAM, the kernel will take the whole file and dump it in RAM. So, when you read and write to the file, you're reading and writing to memory, which makes the process nice and fast! The kernel then caches the file to disk.

Next, there's the issue of swap. From some perspectives, swap is great. However, swap isn't such a good thing from our current perspective. SevOne appliances ship with a ton of memory–so much, in fact, that there shouldn't be any need for swap. Swapping should almost never occur. It should only occur when absolutely necessary, and that shouldn't be often. If there is anything in swap, it shouldn't be over 1 GB. Long story short, if we're using swap, it means that we've used up a whole lot of memory. Now we need to find out why.

Possible causes of swap include:

• Memory leak

• A poorly designed script

Used Memory

You can use SNMP stats to look at the amount of used memory, among other things. One of the indicators you'll use is Used Memory. But you'll also want to include the Cached indicator and the Buffers indicator. This will give you a more accurate picture of the memory being used, because some of that memory may be cached. Compare the two screenshots below. The first one shows only the results for used memory. In the second one, you can see that same amount of used memory, but you'll also see how much memory is cached.

Processes

Make sure to monitor all SevOne processes–everything with SevOne as part of its name. Below is information on process monitoring using the Process Poller and SNMP.

Process Poller

The Process Poller provides stats for processes related to the following:

• Apache

• Bash

• CRON scheduler

• MySQL

• PHP

• etc.

Refer to the Process Poller monitoring information in SevOne NMS for a list of processes that are identified and monitored for the following information:

• Availability

• CPU Time

• Instances of the process

• System memory used - this refers to actual memory and doesn't include virtual.

SNMP

All major SevOne NMS processes connect to the database and issue queries. Every SevOne daemon exports SNMP information about its database use. You can look at the following daemons with SNMP:

• SevOne-clusterd

• SevOne–datad

• SevOne-dispatchd

• SevOne-polld

• SevOne–requestd

• SevOne-trapd

• etc.

SNMP indicators provide several database statistics for processes. The following are a few that you'll want to keep an eye on:

• Query counts - any sudden changes in query counts may indicate a problem.

• Query errors - query errors may be the result of a schema or code issue.

• Number of connections

• Number of reconnections

• Number of traps received, processed, etc. (applies to SevOne-trapd) - you'll want to make sure that the number of traps received isn't much different from the number processed. The numbers should be close to each other (within 100, for example), if not the same. If there's a big difference in the number of traps received and processed, then you may be receiving more traps than you can handle.

Hot Standby Appliance

A hot standby appliance (HSA) consists of an active peer and a passive peer. The active peer pulls configuration data from the cluster master (in the case of a cluster) and polls data from objects and indicators on the devices that are assigned to it. The passive peer maintains a redundant copy of all configuration data and poll data. The following are important things to keep an eye on if you have an HSA:

• Make sure MySQL replication is working. If Availability drops to 0%, then there may be an issue with the MySQL daemon (mysqld).

• Make sure that the poller is running on the active appliance but not on the passive one.

• Make sure that there's a lot of database activity on the active appliance but not on the passive one.

For HSA monitoring, we recommend using the Deferred Data plugin. You'll want to select the SevOne Statistics object ( for example, when using Instant Graphs) and the following indicators:

• Is master

• Is primary

If the passive peer of an HSA is reporting an Is primary value of 1, this indicates that a switchover has occurred. If both the active peer and the passive peer are reporting an Is primary value of 1, then a split-brain condition exists.

Interface

Because SevOne NMS monitors network devices, your SevOne appliance requires some bandwidth but not much. The normal range for a single appliance would fall somewhere between 1 and 3%. For an HSA, you'll see anywhere from two to four times as much bandwidth used. If the amount of bandwidth used is consistently high, this could indicate a problem, and you'll need to look into what's causing the high bandwidth.

With an HSA, you'll see peaks in traffic every two hours. This happens because data is being transferred from short-term storage to long-term storage, which results in additional MySQL replication.

The graph below displays bandwidth information using the indicators HC In Octets and HC Out Octets.

The SevOne Device Group

In the Best Practices section, we recommended putting all of your appliances in their own device group. There's an existing device group called SevOne, and we strongly suggest adding any SevOne appliances that you want to monitor to this group.

To look at the SevOne device group, perform the following steps:

1. From the navigation bar, click Devices and select Grouping. Then select Device Groups.

   

2. On the left, under Device Groups, click next to All Device Groups to expand the section.

3. Click next to Manufacturer.

4. Select the SevOne device group.

On the upper right pane, under Membership Rules for SevOne, you'll notice that sysDescr is set to SEVONE. For more information on device groups, refer to our documentation on the topic.

There's no need to do anything at this point. We'll be talking about the SevOne device group again in the next section.

Add SevOne Appliance to Device Manager

In order to monitor your SevOne appliance, you'll need to add it to the Device Manager just as you would any other device that you want to monitor. The following steps will walk you through the process.

1. First, select the appliance that you would like to monitor your appliance from. You'll be working from that appliance. For example, let's say you want to set up Appliance A to be monitored, and you've decided that Appliance B will do the monitoring. That means you'll need to work from Appliance B.
   

2. Now that you've selected the appliance that will do the monitoring, log into SevOne NMS on that appliance.
   

3. From the navigation bar, click Devices and select Device Manager.
   

4. Click and select Add New Device.
   
   

5. At the top of the New Device page, in the Name field, enter the name of the appliance that you want to monitor.
   

6. In the Alternate Name field, enter an alternate name for the appliance. Users can search for the appliance by this name.
   

7. In the Description field, enter a description of the appliance. You can use this to provide additional information about the appliance, such as location, etc.
   

8. In the IP Address field, enter the IP address of the appliance.
   

9. The Allow Deletion check box appears to the admin user only and is selected by default. When selected, it enables users to delete the device. If you would like to prevent users from deleting the appliance as a device, clear the check box.
   

10. Below that, click the Device Groups drop-down. Click + next to Manufacturer and select the SevOne check box.
    

11. Configure other settings as needed. For more information on the settings, please see our documentation on adding/editing devices.
    

12. Click Save As New.
    

13. You'll see a message at the top of the page letting you know that the device is being queued for discovery.

Enable Self-monitoring on Appliance

In this section, you'll be working with the appliance that you want to monitor.

1. Open your SSH client and log into the appliance that you want to enable self-monitoring on.
   

2. Enter the following command to run the install.sh script:
   

   /usr/local/scripts/utilities/plugins/selfmon/install.sh

3. You'll see a couple of warnings followed by the text Press Ctrl-C to abort or any key to continue... Hit Enter to proceed.
   
   

4. You'll see one more message, Press Ctrl-C to abort or any key if you are sure... Hit Enter again.
   

5. Next, you'll be prompted to ente r the password for the API user SevOneStats. Enter the following:

   n3v3rd13

   

Now you should see some information about your appliance–such as number of peers, whether the appliance is part of an HSA pair, and the IP address used for it. Soon after that, the installation process will start. You'll see a long list of information, especially about objects and indicator types. Once that's done, you should receive confirmation that the installation is complete.

Create Object Rules

By creating object rules, you can make sure you're monitoring the objects you care about on your SevOne appliance. You can also use object rules to exclude the objects that you don't want to monitor. Perform the following steps to create object rules.

For this part, go to the appliance that will be doing the monitoring and log into SevOne NMS.

1. To access the Object Rules page from the navigation bar, click Administration. Under Administration, select Monitoring Configuration, then Object Rules.

   

2. Click Add Rule to bring up the Add Rule pop-up.

   

3. Click the Device Group drop-down and select a device group. Select the device group that the appliance you wish to monitor belongs to (for example, the SevOne device group).

4. Click the Plugin drop-down and select a plugin.

5. Click the Object Type drop-down and select an object type.

6. Click the Subtype drop-down and select an object subtype.

7. The next drop-down is set to Match. This means that the object rule will be applied if the object name matches the expression that you specify (you'll specify this in the next step). If you want the object rule to apply when the object name doesn't match the expression, then click the drop-down and select Do not match.

   You'll need to set at least one match expression. You can set an expression here, for the object name, or you can set one below, for the object description. You can also set expressions for both the name and the description if you like.

8. In the text field directly below, enter a Perl regular expression that you want the object name to match or not match. If you don't want to check the object name against an expression, just leave this blank.

9. The next Match/Do not Match drop-down applies to the object description, rather than the object name. Click the drop-down and select Match or Do not match if you plan on specifying an expression in the next step.

10. In the text field directly below, enter a Perl regular expression that you want the object description to match or not match. If you don't want to check the object description against an expression, just leave this blank.

11. Select the Case Sensitive check box to require expressions to match the case used in the object name/description. If you don't require the case to match, leave the check box clear.

12. In the Notes field, enter any comments that you would like to include about the object rule. This is optional.

13. Click Save.

Policies and Thresholds

Alerts

The Alerts page lets you look at the current, active alerts in the system. These include the threshold violations, trap notifications, and web site errors that you define on the Policy Browser or the Threshold Browser.

To access the Alerts page from the navigation bar, click Events and select Alerts.

Instant Graphs

On the Instant Graphs page, you can create statistical graphs for the objects and indicators on devices. This is a great place to start because instant graphs are very easy and fast to set up, allowing you to get an immediate look at potential problem areas.

In this section we're going to look at CPU activity for the past 24 hours.

To access the Instant Graphs page from the navigation bar, click Reports and select Instant Graphs.

1. Click the Device Group drop-down to select a device group. We're going to select the SevOne device group. Under All Device Groups, click + next to Manufacturer and select SevOne.

2. Click the Device drop-down and select a device to base the instant graph on.

3. Click the Object drop-down to select an object. Let's go with CPU Total0 - CPU Total under SNMP Poller.

4. Click the Indicator drop-down to select an indicator. Select Idle CPU Time.

5. Click the Select button. This will move the indicator you selected to the Indicators to Graph field.

6. Let's throw in one more indicator. Click the Indicator drop-down and select Waiting CPU Time. Then click the Select button to add it.

7. Now you should see both indicators in the Indicators to Graph field. You'll notice that each is preceded by the device name and the object name.
   

   We just added two indicators from the same device and object. You can also add indicators from other devices and objects to display everything in a single graph.

8. The Graph Type is set to Line Graph by default. Let's go with that.

9. Click the Time Span drop-down and select Past 24 hours.

10. In the Units section, the Scaled to Max check box is selected by default. Just leave it that way. This means that the graph will be scaled to the largest actual value for the specified time span. Clearing the check box scales the graph to the maximum potential value. The Scaled to Max setting is irrelevant for indicators that don't have a defined maximum value.

11. Leave the Percentage check box clear for our example.

12. You'll notice two radio buttons, Bits and Bytes. Select Bytes. Bits is used for network-oriented data, while Bytes applies to server-oriented data.

13. Click the Draw Graph button to display a graph of your selected indicators. The graph will appear below the button .

    

14. Your new graph comes with the following controls, which you can use to manipulate the graph presentation:

    • Auto-refresh every __ seconds check box - refreshes the graph every <n> seconds. Enter the number of seconds in the text field.

    • - re-centers the graph. Click this icon and then click the point of the graph that you would like to be in the center. For example, if you were to click 8:00 on the graph in the screenshot above, the 8:00 point would move to the center.

    • - zooms in. Click this icon and then click the point on the graph where you would like to zoom in for a closer look.

    • - zooms out. Click this icon and then click the point on the graph from where you would like to zoom out.

    • - focuses in on a selected range. First, click this icon. After that, click once on the starting point of your desired range, and then click on the end point of the range. Your selected range will appear highlighted. Click anywhere within the highlighted range to view a graph of it.

    • Detach - adds the instant graph as an attachment in a report on a new browser tab.

      You can modify reports to add other attachments and you can save reports to the Report Manager. Report workflows enable you to designate reports to be your favorite reports and to define one report to appear as your custom dashboard.

    • PDF - exports the report to a .pdf format.

    • Calendar - provides a view of the indicator performance over time. You can do some neat stuff with the calendar. Try the following, for example:

      1. Click the Calendar button. The calendar will appear in a new browser tab. When we set up our graph, we specified the past 24 hours as our time span. You'll notice that our new calendar covers an entire month.
         
         

      2. Each day on the calendar has a graph, which summarizes the indicator's activity for that day. Select a day and click the date in the upper right corner of that day.
         
         

      3. Now you should see a breakdown of the day's activity in the form of four six-hour graphs (from 0:00 to 24:00).
         
         

    • NetFlow - displays associated flow data if an indicator in the graph has mapped flow dat a. This button only appears if there is mapped flow data. When you enable the SNMP plugin for a device, many SNMP objects are automatically mapped to their corresponding flow interface, and the Object Mapping page enables you to map additional objects for flow monitoring.

    • NBAR - displays associated NBAR data if an indicator in the graph has mapped NBAR data . This button only appears if there is mapped NBAR data. You can map objects for NBAR monitoring on the Object Mapping page.

Using the Table

The Plugin and Alert Condition columns provide the specific information that you'll need for creating each policy (see Policies and Thresholds for more information on creating policies). The following screenshot shows an example of a policy.

General Settings Tab

The Plugin column in the example above indicates that you'll need to specify Process Poller as the plugin. At the top of the Alert Conditions column, you'll find the object type that you'll need, along with the object subtype if there is one. In this example, the object type is Process, and we have an object subtype of SevOne-polld. Not all policies will require an object subtype. Immediately below the object type information is the recommended severity level, which is Alert in our example.

Trigger Conditions Tab

The Alert Condition column also contains the information you'll need for setting up your policy's trigger condition(s). This information appears below the object type and severity level information. For example, in the screenshot above, the following condition is recommended:

Average Availability < 50% over 30 minutes

When setting up the trigger condition, this means the following:

• Average - specifies Average as the recommended Aggregation setting for this condition.

• Availability - specifies Availability as the indicator to use for this condition. The indicator will appear in bold font.

• < - specifies that the less than operator should be used for this condition.

• 50% - specifies a threshold of 50% for this condition.

• over 30 minutes - indicates a duration of 30 minutes for the condition.

Below is a screenshot of the the pop-up where you specify the settings for your condition.

Additional Information

Type

When you create a condition for a policy, the Type drop-down is set to Static by default. For most of the conditions in the table, this is what you'll need. A few conditions will require a different Type (for example, Baseline Delta, Baseline Percentage, etc.). These exceptions are noted in the Alert Condition column.

Multiple Conditions

Some policies include more than one condition. In cases where both (or all) conditions must be met, this will be indicated by the AND Boolean operator. In the following screenshot, both of the conditions listed under Alert Condition must be met.

An OR Boolean operator indicates that only one condition (or set of conditions) must be met in order to trigger a threshold. In the screenshot below, only one of the conditions (Rule 1 or Rule 2) listed under Alert Condition needs to be met. In this case, you would need a separate Rule for each condition when creating conditions on the Trigger Conditions tab. (See information on using Rules.)

Recommended Policies

The following table contains recommended policies for SevOne self-monitoring.

MySQL Server: mysqlconfig and mysqldata

Indicators

Aborted_clients
Aborted_connects
Binlog_cache_disk_use
Binlog_cache_use
Bytes_received
Bytes_sent
Compression
Connections
Created_tmp_disk_tables
Created_tmp_files
Created_tmp_tables
Delayed_errors
Delayed_insert_threads
Delayed_writes
Flush_commands
Handler_commit
Handler_delete
Handler_prepare
Handler_read_first
Handler_read_key
Handler_read_next
Handler_read_prev
Handler_read_rnd
Handler_read_rnd_next
Handler_rollback
Handler_savepoint
Handler_savepoint_rollback
Handler_update
Handler_write
Key_blocks_not_flushed
Key_blocks_unused
Key_blocks_used
Key_read_requests
Key_reads
Key_write_requests
Key_writes
Max_used_connections
Not_flushed_delayed_rows
Open_files
Open_streams
Opened_tables
Prepared_stmt_count
Qcache_free_blocks
Qcache_free_memory
Qcache_hits
Qcache_inserts
Qcache_lowmem_prunes
Qcache_not_cached
Qcache_queries_in_cache
Qcache_total_blocks
Questions
Select_full_join
Select_full_range_join
Select_range
Select_range_check
Select_scan
Slave_open_temp_tables
Slave_retried_transactions
Slave_running
Slow_launch_threads
Slow_queries
Sort_merge_passes
Sort_range
Sort_rows
Table_locks_immediate
Table_locks_waited
Threads_cached
Threads_connected
Threads_created
Threads_running
Uptime
Percentage_prepared_stmt_count
Thread_cache_miss_rate

Redis Server

Indicators

Connections Received
Commands Processed
Used Memory
Keyspace Hits
Keyspace Misses
Expired Keys
Evicted Keys
PubSub Channels
PubSub Patterns

Raid Array

Indicators

Media Error Count
Other Error Count
Predictive Failure Count
Firmware State

SevOne-polld and SevOne-highpolld

Instead of using the Kron daemon, we now use the services SevOne-polld (regular SNMP polling) and SevOne-highpolld (high frequency SNMP polling) with modified metrics. The first section below lists these modified metrics. The second section lists the indicators that have been added.

Modified metrics:

Free Threads Minimum (Threads Idle Minimum)
Free Threads Maximum (Threads Idle Maximum)
Free Threads Average (Replaced with Threads Idle)
Used Threads Minimum (Threads Active Minimum)
Used Threads Maximum (Threads Active Maximum)
Used Threads Average (Threads Active)
Age of Threads Minimum (Gone)
Age of Threads Maximum (Gone)
Age of Threads Average (Gone)
Device Delay Minimum (Polling Delay Minimum)
Device Delay Maximum (Polling Delay Maximum)
Device Delay Average (Gone)
Polled Devices Minimum (Gone)
Polled Devices Maximum (Gone)
Polled Devices Average (Gone)
Suicidal Threads Minimum (Gone)
Suicidal Threads Maximum (Gone)
Suicidal Threads Average (Gone)

New indicators for SevOne-polld/SevOne-highpolld:

Database connections active
Database connections closed
Database connections opened
Database queries active
Database queries finished
Database queries started
Database queries errors
Database reconnections failed
Database reconnections succeeded
Database reconnections tried
Datad entries sent
Poller requests
Poller tasks finished
Poller tasks started
Start Time
Thread count