Technology
Stefan Damian, , Leave a Comment on Using SNCDiag Tool v2
, ,

Using SNCDiag Tool v2

Self-Hosted Instance Diagnostics Tool

SNCDiag.sh is a bash script designed to assist in diagnosing self-hosted instances. The script runs several diagnostics commands and produces a report to assist in diagnosing issues affecting your instance or database systems. It can also collect logs and configuration from instance nodes installed on the system it runs on.

Please always use the latest version available on this page, as the script is continuously being improved.

Download and Setup

The latest version of the script can be downloaded from the following link:
SNCDiagV2.5.1.tar.gz
MD5: f204e0136ef1b452b6bcf19c6c37d2ed

Or, if the system you wish to diagnose has internet access:
wget https://shotrepo.service-now.com/binaries/SNCDiagV2.5.1.tar.gz

Once the downloaded archive is in your system, use the following command to extract the script:
tar -zxvf SNCDiagV2.5.1.tar.gz

You should now have a file named SNCDiag.sh in your local directory. To use it, you may need to make it executable with the following command:
chmod 755 SNCDiag.sh

Alternatively, you may also run it as follows:
/bin/bash SNCDiag.sh

The script is now ready to be used.

Note:

To work properly, the script must be run as a user with permissions to read the node’s files. Ideally, this should be the user owning the node processes.
The user must also be able to run commands such as vmstat, dmesg or ps. Usually, running as “root” is not necessary, but systems may vary.

The script may take several minutes to run. The more nodes it scans, the longer it will take. Please allow it sufficient time to complete. The message “Diagnostics finished.” will appear by the end, along with the archive filename if --archive is used.

Please check the Requirements section for a list of commands the script needs to be able to run.

Using SNCDiag

The SNCDiag script is able to automatically search for and collect the most usually needed diagnostics data from any server type with just the following parameters:
./SNCDiag.sh --report --archive
The above command works for both app and database servers and will create a full diagnostics report.
This will output a compressed tarball file with the .tgz extension with a name like “diag_20240101_142536.tgz”, which we refer to as “archive”.
The archive will contain all the data collected by SNCDiag, including configuration and logs from any nodes found in the system, so it can be uploaded to us for investigating issues with the server and configuration.

When using the --archive parameter on a DB server, the results of the DB diagnostic queries will be saved to a HTML file in the “spool” directory inside the archive. The file name will be mentioned in the main report file.

On Application Servers:

Selecting Nodes:

By default, SNCDiag will collect diagnostics and logs from all ServiceNow nodes in the server.

If a single server contains multiple nodes, the script may take a long time and generate an excessively large archive, which will be difficult to upload.
In order to collect data just from a single node in a server, the --nodes parameter can be used as follows:
./SNCDiag.sh --report --archive --nodes=/path/to/node

Multiple nodes can be specified by separating with a comma:
./SNCDiag.sh --report --archive --nodes=/path/to/node1,/path/to/node2

Selecting logs by date:

By default the script will only collect log files written within the last 24 hours.

If you wish to also include older logs, you may use the parameter --max_days, such as follows, which would collect the logs from the last 3 days:
./SNCDiag.sh --report --archive --max_days=3

If you wish to collect logs from a specific date, you can use the --from_date parameter:
./SNCDiag.sh --report --archive --from_date=01/01/2024

Alternatively, you can use the --days_ago parameter to get the logs from a number of days ago:
./SNCDiag.sh --report --archive --days_ago=2

You can also combine--days_ago and --max_days to collect a range of days. For example, to collect all logs from 2 days ago and the 3 days before then:
./SNCDiag.sh --report --archive --days_ago=2--max_days=3 

To collect all logs between specific dates, you can combine the parameters --from_date and --to_date. For example, to collect all logs between the 1st and 10th of January:
./SNCDiag.sh --report --archive --from_date=01/01/2024 --to_date=10/01/2024
Note that the allowed formats for these parameters are DD-MM-YYYY or YYYY-MM-DD. The valid separators are ‘-‘, ‘/’ or ‘.’.

On MySQL/MariaDB Servers:

In most situations the parameters --report and --archive are enough to collect all the necessary diagnostics from database servers. On some cases, particularly when there are performance issues, it may also be necessary to get InnoDB diagnostics. To do so, the --innodb parameter is used:
./SNCDiag.sh --report --innodb --archive

If there are multiple database processes running on your system, you can filter a process by the port it’s listening on by using the --db_port parameter, for example:
./SNCDiag.sh --report --archive --db_port=3306

By default, SNCDiag attempts to use the root user on the mysql client and will prompt you for credentials if this fails. To select a different database user to run the diagnostics queries, you can use the --db_user parameter:
./SNCDiag.sh --report --archive --db_user=username

On Oracle DB servers:

Since version 2.5.0, SNCDiag is also capable of diagnosing Oracle databases. To do so, it is enough to run it with the following parameters:
./SNCDiag.sh --report --archive

Note that on OracleDB servers, SNCDiag will have to be run as the same user that owns the OracleDB processes (e.g. oracle) or as root, in order to connect to the database using sqlplus.

Parameters

  • --archive[=filename] – This parameter tells the script to collect the logs and configuration files from a node and add them to a compressed archive. If --report is also used, the report will be included in the archive. Any passwords or comments in the configuration files will not be included in the archive. If no filename is specified, the archive file will have a default name like “diag_date_time.tgz”.
  • --days_ago=number_of_days – Minimum age of logs collected. Default is 0.
  • --db_port=port[, port2, ...] – When scanning for running database processes, this tells the script to only diagnose the process listening on the specified port. Multiple ports can be specified by separating each with a comma.
  • --db_size[=db_name]– Displays data and index sizes for the specified database. If no database is specified, sizes for all databases will be shown. (MySQL/MariaDB only)
  • --db_tcp – Connect to local DB server over TCP instead of Unix socket. (MySQL/MariaDB only)
  • --db_user – Database user to connect as. (MySQL/MariaDB only)
  • --from_date – Date to begin collecting logs from. Allowed formats are DD-MM-YYYY or YYYY-MM-DD. Valid separators are ‘-‘, ‘/’ or ‘.’.
  • --to_date – Date up to which logs are collected.
  • --innodb – When diagnosing a MySQL/MariaDB server, also include INNODB STATUS. (MySQL/MariaDB only)
  • --log[=filename] – Writes the output of the script to a file. If no filename is specified, the filename will default to “diag_date_time.log”.
  • --max_days=number_of_days – Defines the maximum age in days a log file can have to be copied into the archive. Default is 1 day (24 hours).
  • --migration – Gathers the system information needed for instance migrations to the cloud.
  • --nologs – Disable log collection when --archive is used.
  • --nodes=node[, node2, ...] – Limits the diagnostics to the specified node, ignoring any other nodes that may be on the system. Useful for reducing the size of the diagnostics report and archive. Multiple nodes can be specified by separating with a comma. Nodes can be specified using their full path (e.g. /glide/nodes/examplenode/) or the node’s directory name (e.g. examplenode).
  • --report[=option[, option2, ...]] – Tells the script to run diagnostics and output a report. This parameter accepts the following options:
    • cpu – Includes CPU information on the report. Enabled by default.
    • dmesg – Includes the last 50 lines of dmesg in the report. Enabled by default.
    • procs – Includes process information on the report. Enabled by default.
    • stats – Includes the output of the node’s stats.do in the report. Enabled by default.
    • logs – Includes the last 1000 lines of the localhost and wrapper logs in the report. Disabled by default.
      Multiple options can be specified by separating with a comma. Adding any option will disable all options not specified, even if enabled by default.

The --help parameter will print more usage details.

Requirements

SNCDiag uses commands and tools present on most out-of-the-box CentOS and RHEL7/8 installations. 

The following list contains the commands needed by SNCDiag:

  • curl
  • df
  • dmesg
  • find
  • free
  • grep
  • gzip
  • iostat
  • pidstat
  • nc
  • netstat
  • vmstat
  • sar
  • tar

On MySQL/MariaDB servers, the script will also use the mysql client command to perform the diagnostics queries (e.g. PROCESSLIST, VARIABLES, STATUS).
Likewise, on OracleDB servers, the script will need to use the sqlplus client command to perform the diagnostics queries.

Leave a Reply