diff --git a/README.md b/README.md index 8df5c90..ee99c37 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Server Bundle Release -This bundle provides packages to set up two optional Relativity Server features: +This release bundle contains packages that are required to set up or upgrade two optional Relativity Server features: - **Environment Watch**: [Click here to learn more about Environment Watch](docs/environment_watch_product_overview.md) @@ -20,5 +20,8 @@ This bundle provides packages to set up two optional Relativity Server features: ## Getting Started -- [Environment Watch and Data Grid Audit Installation](docs/environment_watch_installation.md) -- [Troubleshooting Guide](/docs/environment_watch_troubleshooting.md) +- **First-Time Installation**: [Click here for installation instructions](docs/environment_watch_installation.md). + +- **Upgrade**: [Click here for upgrade instructions](docs/environment_watch_upgrade.md). + +- **Troubleshooting**: [Click here for the troubleshooting guide](/docs/environment_watch_troubleshooting.md). diff --git a/docs/datagrid_audit_setup.md b/docs/datagrid_audit_setup.md index bc5ed76..d47b8cd 100644 --- a/docs/datagrid_audit_setup.md +++ b/docs/datagrid_audit_setup.md @@ -28,10 +28,10 @@ After installing the required Elastic components for Data Grid Audit, the integr Follow these steps to set up Data Grid Audit using the Relativity Server CLI. All setup will occur on the SQL Primary server. -**Step 1** - Open elevated command prompt/powershell. Navigate to the directory where the CLI is extracted, and run ".\relsvr.exe setup". Select Datagrid +**Step 1** - Open elevated command prompt/powershell. Run below command. Select Datagrid ``` -C:\Server.Bundle.x.y.z> .\relsvr.exe setup +C:\Server.Bundle.x.y.z\relsvr.exe setup Relativity Server CLI - 24.0.1196 Copyright (c) 2025, Relativity ODA LLC diff --git a/docs/elasticsearch_pre_installation_overview.md b/docs/elasticsearch_pre_installation_overview.md index 045f8ee..53cac32 100644 --- a/docs/elasticsearch_pre_installation_overview.md +++ b/docs/elasticsearch_pre_installation_overview.md @@ -16,9 +16,9 @@ The first step of the **Environment Watch** and/or **Data Grid Audit** setup inv > [!NOTE] > At this stage, there is no integration between any Elastic components and Relativity. Integration will be configured during **Step 2**. -In **Step 1**, you will: -- Set up and verify your **Elasticsearch** cluster -- Set up **Kibana** and/or **APM Server**, depending on your product configuration +In **Step 1**, the following actions are performed: +- Set up and verify the **Elasticsearch** cluster +- Set up **Kibana** and/or **APM Server**, depending on the product configuration This step covers configuring a **shared Elasticsearch cluster** to be used for both **Environment Watch** and **Data Grid Audit**. @@ -49,7 +49,7 @@ See below for more information on Elasticsearch cluster configuration and high a #### Elasticsearch Nodes -An Elasticsearch node is a single server that is a part of a cluster. If you are running a single node of Elasticsearch, then you have a cluster of one node. A node can have one or many roles that define the role(s) it plays within the cluster. You define a node’s roles in the elasticsearch.yml file. The node role(s) is defined within the elasticsearch.yml . If you don’t set roles, the node is assigned to most available roles. +An Elasticsearch node is a single server that is a part of a cluster. If a single node of Elasticsearch is running, then there is a cluster of one node. A node can have one or many roles that define the role(s) it plays within the cluster. The node role(s) is defined within the elasticsearch.yml . If roles are not set, the node is assigned to most available roles. See [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-name-settings) for more information on Elasticsearch node settings and roles. @@ -70,7 +70,7 @@ Every Elasticsearch cluster requires at least one node designated master and dat [See the port diagram for network requirements.](environment-watch/port-diagram.md) -Kibana is a user interface that lets you visualize your Elasticsearch data and navigate the Elastic stack. See below for more information on Kibana. +Kibana is a user interface that lets users visualize Elasticsearch data and navigate the Elastic stack. See below for more information on Kibana. - [What is Kibana?](https://www.elastic.co/guide/en/kibana/current/introduction.html) - [Kibana key concepts](https://www.elastic.co/guide/en/kibana/current/kibana-concepts-analysts.html) @@ -89,9 +89,9 @@ See [here](https://www.elastic.co/guide/en/observability/current/apm.html) for m ## System Requirements and Licensing > [!NOTE] -> The documentation below includes several links to instructions in Elastic’s official documentation. Whenever you are jumping into Elastic documentation, ensure the proper Elasticsearch, Kibana, or APM Server documentation version is selected. +> The documentation below includes several links to instructions in Elastic’s official documentation. Whenever jumping into Elastic documentation, ensure the proper Elasticsearch, Kibana, or APM Server documentation version is selected. -Setting up your Elastic stack components will require you to install Elastic software on one or more servers. This installation guide covers configuring a shared Elasticsearch cluster to use for both Environment Watch and Data Grid Audit. The **_Hardware Recommendations by Environment Size_** section below provides hardware guidance for different deployment scenarios and reference environments. +Setting up the Elastic stack components will require installing Elastic software on one or more servers. This installation guide covers configuring a shared Elasticsearch cluster to use for both Environment Watch and Data Grid Audit. The **_Hardware Recommendations by Environment Size_** section below provides hardware guidance for different deployment scenarios and reference environments. ![](../resources/elasticsearch_setup_002.png) @@ -105,16 +105,16 @@ Any server being used to host Elastic components requires: ### Hardware Recommendations by Environment Size -The number of servers and hardware specifications that you need to host the Elastic components will vary depending on the size of your Relativity instance and whether you intend to use the cluster for Environment Watch, Data Grid Audit, or both. Below you will find recommendations based on four Relativity Server environment sizes. These are only recommendations. You can adjust the node counts and role blends for your environment based on observed and desired performance and reliability needs. +The number of servers and hardware specifications needed to host the Elastic components will vary depending on the size of the Relativity instance and whether the cluster is intended for Environment Watch, Data Grid Audit, or both. Below are recommendations based on four Relativity Server environment sizes. These are only recommendations. The node counts and role blends can be adjusted for the environment based on observed and desired performance and reliability needs. **A few other key notes and reminders:** -- **Tuning for speed** – Review Elastic’s guidance on how to tune your environment for speed [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html). -- **Hosting Elastic** – While the guidance below recommends installing the Elastic components on many dedicated servers, there are no hard requirements to isolate Elasticsearch, Kibana, or APM Server on dedicated hosts. As evident with the Development environment specifications, you can deploy the full Elastic stack on a single host if that server can meet your storage needs. +- **Tuning for speed** – Review Elastic’s guidance on how to tune the environment for speed [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html). +- **Hosting Elastic** – While the guidance below recommends installing the Elastic components on many dedicated servers, there are no hard requirements to isolate Elasticsearch, Kibana, or APM Server on dedicated hosts. As evident with the Development environment specifications, the full Elastic stack can be deployed on a single host if that server can meet the storage needs. - **Kibana and APM Server hosting:** - - For Small environments, we recommend dedicated servers for Kibana and APM Server, but you can consider installing Kibana and/or APM Server on a single server or even on the same server being used as an Elasticsearch node for development and very small environments. + - For Small environments, we recommend dedicated servers for Kibana and APM Server, but can consider installing Kibana and/or APM Server on a single server or even on the same server being used as an Elasticsearch node for development and very small environments. - For Medium environments and above, we strongly recommend installing Kibana and APM Server each on dedicated servers. -- **Nodes in a shared Environment Watch/Data Grid cluster** – In a cluster being used for both Environment Watch and Data Grid Audit, you are not required to designate data nodes for one or the other. Any node in the cluster can support operations for either product, though dedicated node assignments may be needed for certain workloads. +- **Nodes in a shared Environment Watch/Data Grid cluster** – In a cluster being used for both Environment Watch and Data Grid Audit, data nodes are not required to be designated for one or the other. Any node in the cluster can support operations for either product, though dedicated node assignments may be needed for certain workloads. **Environment Size** diff --git a/docs/environment-watch/post-install-verification/monitoring-agents.md b/docs/environment-watch/post-install-verification/monitoring-agents.md index 5c3dc38..f4ed35b 100644 --- a/docs/environment-watch/post-install-verification/monitoring-agents.md +++ b/docs/environment-watch/post-install-verification/monitoring-agents.md @@ -3,13 +3,12 @@ ## Table of Contents -- [Post-Install Verification for Monitoring Agents](#post-install-verification-for-monitoring-agents) - - [Verify Monitoring Agent Dashboard Exists](#verify-monitoring-agent-dashboard-exists) - - [Verify Monitoring Agents Dashboard Tags](#verify-monitoring-agents-dashboard-tags) - - [Verify Dashboard Filters Populate with Data](#verify-dashboard-filters-populate-with-data) - - [Verify Data Population by Changing Filters and Time Ranges](#verify-data-population-by-changing-filters-and-time-ranges) - - [Verify Hosts and Agent Versions](#verify-hosts-and-agent-versions) - - [Verify Data in Discover](#verify-data-in-discover) +- [Verify Monitoring Agent Dashboard Exists](#verify-monitoring-agent-dashboard-exists) +- [Verify Monitoring Agents Dashboard Tags](#verify-monitoring-agents-dashboard-tags) +- [Verify Dashboard Filters Populate with Data](#verify-dashboard-filters-populate-with-data) +- [Verify Data Population by Changing Filters and Time Ranges](#verify-data-population-by-changing-filters-and-time-ranges) +- [Verify Hosts and Agent Versions](#verify-hosts-and-agent-versions) +- [Verify Data in Discover](#verify-data-in-discover) --- @@ -149,5 +148,3 @@ Ensure dashboard data is reflected in Discover. ![Screenshot: Discover view 02](../../../resources/post-install-verification-images/monitoring-agents/discover-view.png) --- -## Next Step - diff --git a/docs/environment_watch_troubleshooting.md b/docs/environment_watch_troubleshooting.md index f9baac5..22dc97b 100644 --- a/docs/environment_watch_troubleshooting.md +++ b/docs/environment_watch_troubleshooting.md @@ -1,19 +1,23 @@ -# EnvironmentWatch Troubleshooting Guide +# Environment Watch Troubleshooting Guide This document provides quick reference links to detailed troubleshooting guides for all components in the Relativity Server Bundle environment. ## Component Troubleshooting Guides -### [Elasticsearch Troubleshooting →](troubleshooting/elasticsearch.md) +### [Pre-requisite Troubleshooting](troubleshooting/pre-requisite-troubleshooting.md) -### [Kibana Troubleshooting →](troubleshooting/kibana.md) +### [Elasticsearch Troubleshooting](troubleshooting/elasticsearch.md) -### [APM Server Troubleshooting →](troubleshooting/apm-server.md) +### [Kibana Troubleshooting](troubleshooting/kibana.md) -### [Environment Watch Monitoring Agent and Open Telemetry Collector Troubleshooting →](troubleshooting/monitoring-agent-and-otel-collector.md) +### [APM Server Troubleshooting](troubleshooting/apm-server.md) + +### [Environment Watch Monitoring Agent and Open Telemetry Collector Troubleshooting](troubleshooting/monitoring-agent-and-otel-collector.md) + +### [Relativity Server CLI Troubleshooting](troubleshooting/relativity-server-cli.md) + +### [Relativity Alerts Troubleshooting](troubleshooting/relativity_alerts_troubleshooting.md) -### [Relativity Server CLI Troubleshooting →](troubleshooting/relativity-server-cli.md) -### [Relativity Alerts Troubleshooting →](troubleshooting/relativity_alerts_troubleshooting.md) diff --git a/docs/environment_watch_upgrade.md b/docs/environment_watch_upgrade.md new file mode 100644 index 0000000..24bad9b --- /dev/null +++ b/docs/environment_watch_upgrade.md @@ -0,0 +1,119 @@ +# Upgrade Environment Watch + +This document provides the necessary steps to upgrade the Environment Watch components in a Relativity environment. It includes upgrading the Environment Watch using the CLI and upgrading the Environment Watch Installer. + +## Upgrade CLI + +This section guides you through upgrading Environment Watch using the Relativity Server CLI. + +### Prerequisites + +1. The new Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z` + +### Upgrade Instructions + +**Step 1** - Open elevated command prompt/powershell. Run the below command. Select Environment Watch. + +``` +C:\Server.Bundle.x.y.z\relsvr.exe setup + +Relativity Server CLI - 24.0.1196 +Copyright (c) 2025, Relativity ODA LLC + +What would you like to setup? + DataGrid +> Environment watch + Exit +``` + +**Step 2** - Enter 'y' to proceed. +``` +Confirm you would like to perform the 'Environment Watch' setup [y/n] (y): y +``` + +**Step 3** - The CLI will detect that setup has already been performed. Select the **Upgrade** option to proceed. + +``` +Setup is already performed. Select an option to proceed: +> Upgrade + Rerun Setup + Exit +``` + +**Step 4** - The CLI will retrieve the existing settings and perform the upgrade. Please wait for the process to finish. + +``` +Retrieved existing settings +The Relativity Environment Watch Kibana Upgrade setup is executing in silent or unattended mode. +API Key creation and validation completed ------------------------- 100% +OAuth2 client exists ---------------------------------------------- 100% +Relativity secret store updated ----------------------------------- 100% +Relativity logging updated ---------------------------------------- 100% +Relativity toggles updated ---------------------------------------- 100% +Relativity AppDomain monitoring enabled --------------------------- 100% +APM settings updated ---------------------------------------------- 100% +Elasticsearch indexes updated ------------------------------------- 100% +Kibana Tag imported ----------------------------------------------- 100% +Kibana IndexPattern imported -------------------------------------- 100% +Kibana SavedSearch imported --------------------------------------- 100% +Kibana Dashboard imported ----------------------------------------- 100% +Kibana Alert imported --------------------------------------------- 100% +Kibana Custom Role created. --------------------------------------- 100% +Relativity installation SQL record updated ------------------------ 100% + +The Environment Watch setup has been completed. The Relativity Environment Watch installer package should now be installed within each server contained within this Relativity instance. As each server is setup for monitoring, restart all Relativity services within the machine including "kCura Edds Agent Manager," "kCura Edds Web Processing Manager," and "kCura Service Host Manager" to begin sending telemetry to Elasticsearch. +``` + +Successful completion indicates that Environment Watch is configured for the new version. + +Refer to the [Troubleshooting Guide](troubleshooting/relativity-server-cli.md) if you encounter any issues. + +--- + +## Upgrade Environment Watch Installer + +This document outlines the steps to upgrade the Environment Watch monitoring agent. It is recommended to first upgrade the agent on the SQL Primary server, verify the installation, and then repeat the steps for all other servers in the environment. + +### Prerequisites + +1. The Environment Watch CLI upgrade has been completed. +2. The new Server-bundle zip file has been downloaded and extracted. + +### Step 1: Install the Upgraded Monitoring Agent + +1. From the new extracted Server Bundle, run `Relativity.EnvironmentWatch.Installer.xx.x.xxxx.exe`. +2. Accept the license terms and click **Install**. +3. Once the installation is complete, click **Close**. + +### Step 2: Verify the Installation + +1. Open **Services** and confirm the **Relativity Environment Watch** service is running. +2. Open **Task Manager** and verify the following processes are running: + * `rel-envwatch-service.exe` + * `rel-infrawatch-agent.exe` + * `otelcol-relativity.exe` +3. Check for new log files in `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs`. +4. Verify that the `otelcol-config-auto-generated.yaml` configuration file is updated in: + `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\` + +### Step 3: Verify Metrics in Kibana + +1. In Kibana, navigate to **Dashboards** and open the **[Relativity] Host Infrastructure Overview** dashboard. +2. Confirm that CPU, RAM, and Disk metrics are visible for the upgraded host. + ![Host metrics visible in Kibana](../resources/Installer_hostmetric.png) +3. Navigate to the **[Relativity] Monitoring Agent** dashboard to verify: + - The monitoring agent version is updated to the new version. + - Host information and other important fields are properly reflected. + - Agent status and health metrics are showing current data. + +### Step 4: Repeat Installation for Other Servers + +After verifying the first upgrade, repeat these steps for the remaining servers in the following order: +1. SQL Distributed Servers +2. Web Servers +3. Agent Servers +4. Other Servers (e.g., File Share, Analytics, Worker) + +> [!NOTE] +> For troubleshooting, refer to the installer logs in the `%TEMP%` directory or see the [Monitoring Agent Troubleshooting Guide](troubleshooting/monitoring-agent-and-otel-collector.md). + diff --git a/docs/relativity_server_cli_setup.md b/docs/relativity_server_cli_setup.md index e8069e4..4b210a5 100644 --- a/docs/relativity_server_cli_setup.md +++ b/docs/relativity_server_cli_setup.md @@ -17,10 +17,10 @@ ### Set up instructions -**Step 1** - Open elevated command prompt/powershell. Navigate to the directory where the CLI is extracted, and run ".\relsvr.exe setup". Select Environment Watch +**Step 1** - Open elevated command prompt/powershell. Run the below command. Select Environment Watch ``` -C:\Server.Bundle.x.y.z> .\relsvr.exe setup +C:\Server.Bundle.x.y.z\relsvr.exe setup Relativity Server CLI - 24.0.1196 Copyright (c) 2025, Relativity ODA LLC diff --git a/docs/troubleshooting/apm-server.md b/docs/troubleshooting/apm-server.md index 01b86dd..ed97254 100644 --- a/docs/troubleshooting/apm-server.md +++ b/docs/troubleshooting/apm-server.md @@ -7,16 +7,13 @@ This document provides troubleshooting guidance for common APM Server issues enc ## Table of Contents -- [1. Windows Service Issues](#1-windows-service-issues) - - [1.1 APM Server Service Not Starting](#11-apm-server-service-not-starting) - - [1.2 Service Crashes or Stops Unexpectedly](#12-service-crashes-or-stops-unexpectedly) - - [1.3 Permission and Access Issues](#13-permission-and-access-issues) -- [2. Port Configuration Issues](#2-port-configuration-issues) - - [2.1 Port Conflicts](#21-port-conflicts) - - [2.2 Network Connectivity Problems](#22-network-connectivity-problems) -- [3. Service Verification](#3-service-verification) - - [3.1 Verifying APM Server Health and Status](#31-verifying-apm-server-health-and-status) -- [4. Self-Instrumentation](#4-self-instrumentation) + - [1. Windows Service Issues](#1-windows-service-issues) + - [1.1 APM Server Service Not Starting](#11-apm-server-service-not-starting) + - [1.2 Service Crashes or Stops Unexpectedly](#12-service-crashes-or-stops-unexpectedly) + - [1.3 Permission and Access Issues](#13-permission-and-access-issues) + - [2. Memory Issues](#2-memory-issues) + - [3. Self-Instrumentation](#3-self-instrumentation) + - [4. Service Verification](#4-service-verification) --- @@ -162,134 +159,73 @@ This document provides troubleshooting guidance for common APM Server issues enc --- -## 2. Port Configuration Issues +## 2. Memory Issues -### 2.1 Port Conflicts +### 2.1 High Memory Usage **Symptoms:** -- APM Server fails to bind to default port -- "bind: address already in use" errors in logs -- APM agents cannot connect to server +- APM Server consuming excessive memory +- System performance degradation +- Out of memory errors in logs **Troubleshooting Steps:** -* **Check Default Port:** - - Default APM Server port: 8200 - - Verify port availability: - ```powershell - netstat -an | findstr ":8200" +* **Check APM Server Logs:** + - Look for memory-related error messages or stack traces. + - Example error message: ``` -
- Expected response - + fatal error: out of memory ``` - (No output if port is available. If you see LISTENING, ESTABLISHED, port is in use.) - ``` -
- -* **Identify Port Conflicts:** - ```powershell - Get-NetTCPConnection -LocalPort 8200 -State Listen - ``` -
- Expected response - - ``` - # results if port is available. - Get-NetTCPConnection : No matching MSFT_NetTCPConnection objects found by CIM query for instances of the - ROOT/StandardCimv2/MSFT_NetTCPConnection class on the CIM server: SELECT * FROM MSFT_NetTCPConnection WHERE - ((LocalPort = 8200)) AND ((State = 2)). Verify query parameters and retry. - - # results if another process is using port 8200 - LocalAddress LocalPort RemoteAddress RemotePort State AppliedSetting - ------------ --------- ------------- ---------- ----- -------------- - 10.0.2.2 8200 0.0.0.0 0 Listen - ``` -
- -> [!IMPORTANT] -> Do not change the APM Server port. Instead, identify and stop the conflicting service using port 8200, as changing the APM Server port requires extensive configuration changes across Environment Watch, Relativity, and other components. - ---- - -### 2.2 Network Connectivity Problems - -**Symptoms:** -- Service Not Running: APM Server or Elasticsearch may not be running or listening on the expected endpoints. -- Incorrect Configuration: The APM Server or Elasticsearch endpoint URLs may be misconfigured (wrong host, port, or protocol). -- Firewall Rules: Firewalls on the VM host or network may be blocking required ports (e.g., 9200 for Elasticsearch, 8200 for APM Server). -**Troubleshooting Steps:** +* **Review APM Server Configuration:** + - Check `apm-server.yml` for memory-related settings. + - Common settings to review: + - `apm-server.memory.limit`: Maximum memory APM Server can use. + - `apm-server.memory.queue`: Size of the memory queue for incoming events. -* **Verify Network Binding:** - - Check `apm-server.yml` configuration: + - Example configuration: ```yaml apm-server: - host: "0.0.0.0:8200" # Listen on all interfaces - # or - host: ":8200" + memory: + limit: 512mb + queue: 1000 ``` -* **Test Remote Connectivity:** - ```powershell - Test-NetConnection -ComputerName -Port 8200 - ``` -
- Expected response - - ``` - TcpTestSucceeded : True - ``` -
- ---- - -## 3. Service Verification +* **Monitor System Resources:** + - Use Task Manager or Resource Monitor to check APM Server memory usage. + - Compare with configured limits in `apm-server.yml`. -### 3.1 Verifying APM Server Health and Status +* **Adjust Memory Settings:** + - If APM Server is using too much memory, consider adjusting the following settings in `apm-server.yml`: + - `apm-server.memory.limit`: Decrease the maximum memory limit. + - `apm-server.memory.queue`: Decrease the memory queue size. -**Symptoms:** -- Need to confirm APM Server is operating correctly -- Performance monitoring requirements -- Health check automation + - Example: + ```yaml + apm-server: + memory: + limit: 256mb + queue: 500 + ``` -**Troubleshooting Steps:** +* **Restart APM Server:** + - After making changes, restart the APM Server service: + ```powershell + Restart-Service apm-server + ``` -* **Verify Server Configuration:** - ```powershell - C:\elastic\apm-server\apm-server.exe test config -c "C:\elastic\apm-server\apm-server.yml" - ``` -
+
Expected response ``` - Config OK - ``` -
- -* **Check Elasticsearch Connection:** - ```powershell - C:\elastic\apm-server\apm-server.exe test output -c "C:\elastic\apm-server\apm-server.yml" - ``` -
- Expected output for successful connection - - ``` - elasticsearch: https://:9200... - parse url... OK - connection... - parse host... OK - dns lookup... OK - addresses: 192.168.1.100 - dial up... OK - TLS... WARN secure connection disabled - talk to server... OK + WARNING: Waiting for service 'apm-server + (apm-server)' to stop... ```
--- -## 4. Self-Instrumentation +## 3. Self-Instrumentation **Symptoms:** - Need to monitor APM Server itself for performance and health metrics @@ -297,9 +233,6 @@ This document provides troubleshooting guidance for common APM Server issues enc **Troubleshooting Steps:** -> [!NOTE] -> For detailed self-instrumentation configuration steps, see the [Install and Configure APM Server](../elasticsearch_setup_development.md#step-3-install-and-configure-apm-server). - * **Enable Self-Instrumentation:** - Self-instrumentation allows APM Server to monitor its own performance. - This feature is configured in the `apm-server.yml` file. @@ -322,3 +255,32 @@ This document provides troubleshooting guidance for common APM Server issues enc - Check Kibana to verify that APM Server metrics are being collected +--- + +## 4. Service Verification + +### 4.1 Verifying APM Server Health + +**Symptoms:** +- Need to confirm APM Server is operating correctly +- Health check automation + +**Troubleshooting Steps:** + +* **Check APM Server Status:** + ```powershell + curl.exe -k "http://:8200/" + ``` +
+ Expected output + + ```json + { + "build_date": "...", + "build_sha": "...", + "publish_ready": true, + "version": "8.17.3" + } + ``` +
+ diff --git a/docs/troubleshooting/elasticsearch.md b/docs/troubleshooting/elasticsearch.md index 7813637..b3e766a 100644 --- a/docs/troubleshooting/elasticsearch.md +++ b/docs/troubleshooting/elasticsearch.md @@ -11,16 +11,13 @@ This document provides troubleshooting guidance for common Elasticsearch issues - [1. Windows Service Issues](#1-windows-service-issues) - [1.1 Elasticsearch Service Not Starting](#11-elasticsearch-service-not-starting) - [1.2 Service Crashes or Stops Unexpectedly](#12-service-crashes-or-stops-unexpectedly) - - [2. Port Configuration Issues](#2-port-configuration-issues) - - [2.1 Port Conflicts](#21-port-conflicts) - - [2.2 Network Connectivity Problems](#22-network-connectivity-problems) - - [3. Memory Issues](#3-memory-issues) - - [3.1 Insufficient Memory Allocation](#31-insufficient-memory-allocation) - - [4. Authentication Issues](#4-authentication-issues) - - [4.1 Username/Password Authentication Problems](#41-usernamepassword-authentication-problems) - - [4.2 SSL/TLS Certificate Issues](#42-ssltls-certificate-issues) - - [5. Service Verification](#5-service-verification) - - [5.1 Verifying Elasticsearch Health](#51-verifying-elasticsearch-health) + - [2. Memory Issues](#2-memory-issues) + - [2.1 Insufficient Memory Allocation](#21-insufficient-memory-allocation) + - [3. Authentication Issues](#3-authentication-issues) + - [3.1 Username/Password Authentication Problems](#31-usernamepassword-authentication-problems) + + - [4. Service Verification](#4-service-verification) + - [4.1 Verifying Elasticsearch Health](#41-verifying-elasticsearch-health) --- @@ -160,136 +157,14 @@ This document provides troubleshooting guidance for common Elasticsearch issues ```
---- - -## 2. Port Configuration Issues - -### 2.1 Port Conflicts - -**Symptoms:** -- Elasticsearch fails to bind to default ports -- "Address already in use" errors in logs -- Cannot access Elasticsearch via HTTP/HTTPS - > [!NOTE] -> Click here to view the latest port diagram that includes the Elastic Stack [Port Diagram](../environment-watch/port-diagram.md) - -**Troubleshooting Steps:** - -* **Check Default Ports:** - - HTTP: 9200 - - Transport: 9300 - - Verify these ports are available: - ```powershell - netstat -an | findstr ":9200" - netstat -an | findstr ":9300" - ``` -
- Expected output - - ``` - TCP 0.0.0.0:9200 0.0.0.0:0 LISTENING - TCP 0.0.0.0:9300 0.0.0.0:0 LISTENING - ``` -
- -* **Test Elasticsearch Connectivity:** - ```powershell - curl.exe -k -u : -X GET "https://:9200/" - ``` -
- Expected output - - ```json - { - "name" : "EMTTEST", - "cluster_name" : "elasticsearch", - "cluster_uuid" : "", - "version" : { - "number" : "8.17.3", - "build_flavor" : "default", - "build_type" : "zip", - "build_hash" : "", - "build_date" : "2025-02-28T10:07:26.089129809Z", - "build_snapshot" : false, - "lucene_version" : "9.12.0", - "minimum_wire_compatibility_version" : "7.17.0", - "minimum_index_compatibility_version" : "7.0.0" - }, - "tagline" : "You Know, for Search" - } - ``` -
- -* **Identify Port Conflicts:** - ```powershell - Get-NetTCPConnection -LocalPort 9200 -State Listen - Get-NetTCPConnection -LocalPort 9300 -State Listen - ``` -
- Expected output - - ``` - LocalAddress LocalPort State - ----------- --------- ----- - 0.0.0.0 9200 Listen - 0.0.0.0 9300 Listen - ``` -
- -### 2.2 Network Connectivity Problems - -**Symptoms:** -- Cannot connect to Elasticsearch from remote hosts -- Connection timeouts -- "No route to host" errors - -**Troubleshooting Steps:** - -* **Verify Network Binding:** - - Check `C:\elastic\elasticsearch\config\elasticsearch.yml` configuration: - ```yaml - network.host: 0.0.0.0 # For all interfaces - # or - network.host: ["_local_", "_site_"] - ``` - -* **Test Connectivity (from any server):** - 1. First, check if the port is accessible: - ```powershell - Test-NetConnection -ComputerName -Port 9200 - ``` -
- Expected output - - ``` - ComputerName : - RemoteAddress : - RemotePort : 9200 - TcpTestSucceeded : True - ``` -
- 2. If the port is accessible, test the Elasticsearch endpoint: - ```powershell - curl.exe -k -u : -X GET "https://:9200/" - ``` -
- Expected output - - ```json - { - "name" : "EMTTEST", - "cluster_name" : "elasticsearch", - ... - } - ``` -
+> For port-related issues, see the [Port Configuration Troubleshooting](port-configuration-troubleshooting.md) guide. --- -## 3. Memory Issues +## 2. Memory Issues -### 3.1 Insufficient Memory Allocation +### 2.1 Insufficient Memory Allocation **Symptoms:** - OutOfMemoryError in Elasticsearch logs @@ -334,9 +209,9 @@ This document provides troubleshooting guidance for common Elasticsearch issues --- -## 4. Authentication Issues +## 3. Authentication Issues -### 4.1 Username/Password Authentication Problems +### 3.1 Username/Password Authentication Problems **Symptoms:** - Login failures @@ -385,72 +260,12 @@ This document provides troubleshooting guidance for common Elasticsearch issues > [!NOTE] > Also verify that the URL you are using is `https://:9200/` -### 4.2 SSL/TLS Certificate Issues - -**Symptoms:** -- SSL handshake failures -- "certificate verify failed" errors -- Unable to establish secure connections -- Browser shows "not secure" warning for Elasticsearch URL - -**Troubleshooting Steps:** - -* **Install SSL Certificate in Trusted Store** - - If your browser shows a "not secure" warning when accessing the Elasticsearch URL, you may need to install the certificate into your trusted store. - - a. In your browser, view the certificate details and export the root certificate authority (CA) certificate. Save it to a local directory. - - b. Double-click the downloaded certificate file and click **Install Certificate**. - - ![Install Certificate](../../resources/troubleshooting-images/installcertificate.png) - - c. Select **Local Machine** and click **Next**. - - ![Select Local Machine](../../resources/troubleshooting-images/localmachine.png) - - d. Select **Place all certificates in the following store**, click **Browse**, and select **Trusted Root Certification Authorities**. Click **OK**, then **Next**, and **Finish**. - - e. To confirm, open the Microsoft Management Console (MMC): - i. Run `mmc.exe`. - ii. Go to **File > Add/Remove Snap-in...**. - iii. Select **Certificates** and click **Add**. - - ![Add/Remove Snap-in](../../resources/troubleshooting-images/Add-removesnipin.png) - - ![Add Certificates Snap-in](../../resources/troubleshooting-images/addcerts.png) - - f. Choose **Computer account** and click **Next**, then **Finish**, and **OK**. - - ![Select Computer Account](../../resources/troubleshooting-images/clickcomputeraccount.png) - - g. Expand **Certificates (Local Computer) > Trusted Root Certification Authorities > Certificates** and verify your certificate is listed. - - h. Close your browser and reopen the Elasticsearch URL. It should now show as secure. - - ![Secure Connection](../../resources/troubleshooting-images/sslenabled.png) - -* **Verify Certificate Path in `elasticsearch.yml`** - - Ensure the `elasticsearch.yml` file points to the correct certificate files. - - - Check `C:\elastic\elasticsearch\config\elasticsearch.yml`: - ```yaml - xpack.security.transport.ssl: - keystore.path: certs/transport.p12 - truststore.path: certs/transport.p12 - ``` - -* **Check Elasticsearch Logs for SSL Errors** - - Navigate to `C:\elastic\elasticsearch\logs\`. - - Review the `elasticsearch.log` file for any SSL-related errors. - - For every error in the Elasticsearch log, provide troubleshooting for that specific error. --- -## 5. Service Verification +## 4. Service Verification -### 5.1 Verifying Elasticsearch Health +### 4.1 Verifying Elasticsearch Health **Symptoms:** - Uncertainty about cluster status diff --git a/docs/troubleshooting/kibana.md b/docs/troubleshooting/kibana.md index d30c584..26d5657 100644 --- a/docs/troubleshooting/kibana.md +++ b/docs/troubleshooting/kibana.md @@ -12,18 +12,16 @@ This document provides troubleshooting guidance for common Kibana issues encount - [1.2 Service Crashes or Stops Unexpectedly](#12-service-crashes-or-stops-unexpectedly) - [2. Authentication Issues](#2-authentication-issues) - [2.1 Username/Password Authentication Issues](#21-usernamepassword-authentication-issues) -- [3. Port Configuration Issues](#3-port-configuration-issues) - - [3.1 Port Conflicts](#31-port-conflicts) - - [3.2 Network Binding Problems](#32-network-binding-problems) -- [4. Memory Issues](#4-memory-issues) - - [4.1 Insufficient Memory Allocation](#41-insufficient-memory-allocation) -- [5. Kibana Encryption Keys Configuration](#5-kibana-encryption-keys-configuration) - - [5.1 Missing or Invalid Encryption Keys](#51-missing-or-invalid-encryption-keys) -- [6. Service Verification](#6-service-verification) - - [6.1 Verifying Kibana Health and Status](#61-verifying-kibana-health-and-status) -- [7. Additional Diagnostic Commands](#7-additional-diagnostic-commands) -- [8. Installation Issues](#8-installation-issues) - - [8.1 Long Path Issues During Extraction](#81-long-path-issues-during-extraction) +- [3. Memory Issues](#3-memory-issues) + - [3.1 Insufficient Memory Allocation](#31-insufficient-memory-allocation) +- [4. Kibana Encryption Keys Configuration](#4-kibana-encryption-keys-configuration) + - [4.1 Missing or Invalid Encryption Keys](#41-missing-or-invalid-encryption-keys) +- [5. Service Verification](#5-service-verification) + - [5.1 Verifying Kibana Health and Status](#51-verifying-kibana-health-and-status) +- [6. Additional Diagnostic Commands](#6-additional-diagnostic-commands) + - [6.1 Configuration Validation](#61-configuration-validation) +- [7. Installation Issues](#7-installation-issues) + - [7.1 Long Path Issues During Extraction](#71-long-path-issues-during-extraction) --- @@ -172,86 +170,14 @@ This document provides troubleshooting guidance for common Kibana issues encount ---- - -## 3. Port Configuration Issues - -### 3.1 Port Conflicts - -**Symptoms:** -- Kibana fails to bind to default port -- "EADDRINUSE" errors in logs -- Cannot access Kibana web interface - -**Troubleshooting Steps:** - -* **Check Default Port:** - - Default Kibana port: 5601 - - Verify port availability: - ```powershell - netstat -an | findstr ":5601" - ``` -
- Expected output - - ``` - TCP 0.0.0.0:5601 0.0.0.0:0 LISTENING - ``` -
- -* **Test Kibana Connectivity:** - ```powershell - (curl.exe -s -k -u : -X GET "http://:5601/api/status" | ConvertFrom-Json).status.overall | ConvertTo-Json -Depth 10 - ``` -
- Expected output - - ```json - { - "level": "available", - "summary": "All services and plugins are available" - } - ``` -
- - - -* **Update Firewall Rules:** - ```powershell - New-NetFirewallRule -DisplayName "Kibana Web Interface" -Direction Inbound -Protocol TCP -LocalPort 5601 -Action Allow - ``` -
- Expected output - - ``` - (No output if successful. Rule will appear in Windows Firewall.) - ``` -
- -### 3.2 Network Binding Problems - -**Symptoms:** -- Cannot access Kibana from remote hosts -- Connection refused errors -- Kibana only accessible from localhost - -**Troubleshooting Steps:** - -* **Verify Network Configuration:** - - Check `C:\elastic\kibana\config\kibana.yml` configuration: - ```yaml - server.host: "0.0.0.0" # For all interfaces - # or - server.host: "" - ``` - - +> [!NOTE] +> For port-related issues, see the [Port Configuration Troubleshooting](port-configuration-troubleshooting.md) guide. --- -## 4. Memory Issues +## 3. Memory Issues -### 4.1 Insufficient Memory Allocation +### 3.1 Insufficient Memory Allocation **Symptoms:** - Kibana becomes unresponsive @@ -285,9 +211,9 @@ This document provides troubleshooting guidance for common Kibana issues encount --- -## 5. Kibana Encryption Keys Configuration +## 4. Kibana Encryption Keys Configuration -### 5.1 Missing or Invalid Encryption Keys +### 4.1 Missing or Invalid Encryption Keys **Symptoms:** - Kibana fails to start with encryption-related errors @@ -316,9 +242,9 @@ This document provides troubleshooting guidance for common Kibana issues encount --- -## 6. Service Verification +## 5. Service Verification -### 6.1 Verifying Kibana Health and Status +### 5.1 Verifying Kibana Health and Status **Symptoms:** - Need to confirm Kibana is operating correctly @@ -351,9 +277,9 @@ This document provides troubleshooting guidance for common Kibana issues encount --- -## 7. Additional Diagnostic Commands +## 6. Additional Diagnostic Commands -### 7.1 Configuration Validation +### 6.1 Configuration Validation **Validate YAML syntax** @@ -372,9 +298,9 @@ This document provides troubleshooting guidance for common Kibana issues encount --- -## 8. Installation Issues +## 7. Installation Issues -### 8.1 Long Path Issues During Extraction +### 7.1 Long Path Issues During Extraction **Symptoms:** - Errors when extracting the `kibana-8.xx.x-windows-x86_64.zip` file. diff --git a/docs/troubleshooting/monitoring-agent-and-otel-collector.md b/docs/troubleshooting/monitoring-agent-and-otel-collector.md index bc05835..fe63c60 100644 --- a/docs/troubleshooting/monitoring-agent-and-otel-collector.md +++ b/docs/troubleshooting/monitoring-agent-and-otel-collector.md @@ -8,35 +8,67 @@ This document provides a stepwise troubleshooting guide for the Relativity Envir ## Table of Contents -- [Environment Watch Monitoring Agent and Open Telemetry Collector Troubleshooting](#environment-watch-monitoring-agent-and-open-telemetry-collector-troubleshooting) - - [Verify the Elastic Stack Servers are Running](#verify-the-elastic-stack-servers-are-running) - - [Verify the Monitoring Agent Hosts are Present and Sending Metrics](#verify-the-monitoring-agent-hosts-are-present-and-sending-metrics) - - [Verify the Environment Watch Service and Open Telemetry Collector](#verify-the-environment-watch-service-and-open-telemetry-collector) - - [Verify the Open Telemetry Collector Logs](#verify-the-open-telemetry-collector-logs) - - [Additional Pre-requisite Access Checks](#additional-pre-requisite-access-checks) - - [BCP Share Access Verification](#bcp-share-access-verification) - - [Secret Store Access Verification](#secret-store-access-verification) - - [Kepler (SSL Certificate) Verification](#kepler-ssl-certificate-verification) - - [Relativity Service Account Verification](#relativity-service-account-verification) - - [Installer and Service Errors](#installer-and-service-errors) - - [User Not in Local Security Policy](#user-not-in-local-security-policy) - - [Invalid Secrets](#invalid-secrets) +1. [Verify the Elastic Stack Servers are Running](#1-verify-the-elastic-stack-servers-are-running) +2. [Verify the Monitoring Agent Hosts are Present and Sending Metrics](#2-verify-the-monitoring-agent-hosts-are-present-and-sending-metrics) +3. [Verify the Environment Watch Service and Open Telemetry Collector](#3-verify-the-environment-watch-service-and-open-telemetry-collector) +4. [Verify the Open Telemetry Collector Logs](#4-verify-the-open-telemetry-collector-logs) +5. [Additional Pre-requisite Access Checks](#5-additional-pre-requisite-access-checks) + - 5.1. [BCP Share Access Verification](#51-bcp-share-access-verification) + - 5.2. [Kepler and Web (SSL Certificate) Verification](#52-kepler-and-web-ssl-certificate-verification) + - 5.3. [Relativity Service Account Verification](#53-relativity-service-account-verification) +6. [Installer and Service Errors](#6-installer-and-service-errors) + - 6.1. [User Not in Local Security Policy](#61-user-not-in-local-security-policy) + - 6.2. [Invalid Secrets](#62-invalid-secrets) --- -## Verify the Elastic Stack Servers are Running +## 1. Verify the Elastic Stack Servers are Running First, ensure that the core Elastic Stack components (Elasticsearch, Kibana, and APM Server) are running and accessible. If you are not seeing any data in dashboards, this strongly suggests a problem with the Elastic Stack itself. -- For connectivity and troubleshooting, see: - - [ElasticSearch Troubleshooting](elasticsearch.md) - - [Kibana Troubleshooting](kibana.md) - - [APM-Server Troubleshooting](apm-server.md) ---- +**Check Service Accessibility:** -## Verify the Monitoring Agent Hosts are Present and Sending Metrics +- **Elasticsearch:** + ```powershell + curl.exe -k -u : -X GET "https://:9200/" + ``` +
+ Expected output + + A JSON response with cluster details indicates success. +
+ +- **Kibana:** + ```powershell + curl.exe -k -u : -X GET "http://:5601/api/status" + ``` +
+ Expected output + + A JSON response with `"level": "available"` indicates success. +
+ +- **APM Server:** + ```powershell + curl.exe -k "http://:8200/" + ``` +
+ Expected output + + A JSON response with version details indicates success. +
+ +**If a service is not running, refer to its specific troubleshooting guide:** +- [Elasticsearch Troubleshooting](elasticsearch.md) +- [Kibana Troubleshooting](kibana.md) +- [APM Server Troubleshooting](apm-server.md) + + +- For port-related issues, see the [Pre-requisite Troubleshooting](pre-requisite-troubleshooting.md) guide. + +## 2. Verify the Monitoring Agent Hosts are Present and Sending Metrics If the Elastic Stack is running, check that your monitoring agent hosts are present in Kibana and are sending metrics. If hosts are missing or not updating, the issue may be with the agent or host configuration. @@ -69,85 +101,59 @@ If the Elastic Stack is running, check that your monitoring agent hosts are pres --- -## Verify the Environment Watch Service and Open Telemetry Collector +## 3. Verify the Environment Watch Service and Open Telemetry Collector If a specific host is not reporting, check that the Environment Watch Windows service is running on that host. This service is responsible for managing the Open Telemetry Collector. Also verify that the Open Telemetry Collector process is running, its port is listening, logs are being generated, and the auto-generated YAML file exists. **How to check:** -1. Open PowerShell and run: - - ```powershell - Get-Service 'Relativity Environment Watch' - ``` +1. Open PowerShell and run the following to check the service status: + ```powershell + Get-Service 'Relativity Environment Watch' + ``` +
+ Expected output + + ``` + Status Name Display Name + ------ ---- --------- + Running Relativity Envi... Relativity Environment Watch + ``` +
+ If the service is not running, restart it: + ```powershell + Restart-Service -Name "Relativity Environment Watch" + ``` +
+ Expected output + + No output if successful. Service status will be "Running" after execution. +
+2. Verify logs are being generated: + - Check the directory: + `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs` + - Ensure files like `otelcol-relativity-stderr.log` and `otelcol-relativity-stdout.log` are present and updating.
Expected output - ``` - Status Name Display Name - ------ ---- --------- - Running Relativity Envi... Relativity Environment Watch - ``` -
-2. If status is not running, restart the service: - - ```powershell - Restart-Service -Name "Relativity Environment Watch" - ``` -
- Expected output - - No output if successful. Service status will be "Running" after execution. + Log files are present and their timestamps are updating as new data is written.
-3. Verify logs are being generated: - - Check the directory: - `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs` - - Ensure files like `otelcol-relativity-stderr.log` and `otelcol-relativity-stdout.log` are present and updating. -
- Expected output - - Log files are present and their timestamps are updating as new data is written. -
-4. Open Task Manager and look for `otelcol-relativity.exe` under the Processes tab. - - Alternatively, use PowerShell: - ```powershell - Get-Process -Name otelcol-relativity - ``` -
- Expected output - - ``` - Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName - ------- ------ ----- ----- ------ -- -- ----------- - ... ... ... ... ... ... ... otelcol-relativity - ``` - *(If not running, no output.)* -
-5. Check port status: - - ```powershell - netstat -an | findstr ":4318" - ``` -
- Expected output - - ``` - TCP 0.0.0.0:4318 0.0.0.0:0 LISTENING - ``` - (Only present when service is running; no output if stopped.) -
- +3. Open Task Manager and look for `otelcol-relativity.exe` under the Processes tab. + - Alternatively, use PowerShell: ```powershell - Get-NetTCPConnection -LocalPort 4318 -State Listen + Get-Process -Name otelcol-relativity ```
Expected output ``` - LocalAddress LocalPort RemoteAddress RemotePort State - ------------ --------- ------------- ---------- ----- - :: 4318 :: 0 Listen - 0.0.0.0 4318 0.0.0.0 0 Listen + Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName + ------- ------ ----- ----- ------ -- -- ----------- + ... ... ... ... ... ... ... otelcol-relativity ``` - (Only present when service is running; no output if stopped.) + *(If not running, no output.)*
+4. Check port status in the [Port Configuration Troubleshooting](port-configuration-troubleshooting.md) guide. > [!NOTE] > When running, both `rel-envwatch-service` and `otelcol-relativity` processes are present. When stopped, neither process is present. Port 4318 is listening only when service is running. @@ -156,7 +162,7 @@ If a specific host is not reporting, check that the Environment Watch Windows se -## Verify the Open Telemetry Collector Logs +## 4. Verify the Open Telemetry Collector Logs If the service and collector are running but data is still missing, check the logs for errors or misconfiguration. @@ -230,11 +236,11 @@ If the service and collector are running but data is still missing, check the lo --- -## Additional Pre-requisite Access Checks +## 5. Additional Pre-requisite Access Checks If the above steps do not resolve the issue, verify the following access and configuration requirements: -### BCP Share Access Verification +### 5.1. BCP Share Access Verification > [!NOTE] > If you are not logged in as the Relativity Service Account, use the commands below. @@ -253,76 +259,11 @@ True *(If the path is accessible; otherwise, False.)* -### Secret Store Access Verification - -```powershell -Test-NetConnection -ComputerName -Port 443 -``` - -**Expected output:** - -``` -ComputerName : -RemoteAddress : -RemotePort : 443 -TcpTestSucceeded : True -``` - -**Test API access:** - -- Open in an elevated PowerShell and run the following command: - - ```powershell - C:\Program Files\Relativity Secret Store\Client\secretstore.exe secret list / - ``` - - The output will look similar to: - ``` - Secret Store URL: https://:9090/ - Client Certificate Thumbprint: 20F8F2516EC86EBF993075F64B0C6EA6777A4F83 - ``` - -- Copy the Client Certificate Thumbprint and Secret Store URL. - -- To check the seal status, open in an elevated PowerShell ISE and run the following script: - - ```powershell - $thumbprint = "" - $store = New-Object System.Security.Cryptography.X509Certificates.X509Store("My", "LocalMachine") - $store.Open("ReadOnly") - $cert = $store.Certificates | Where-Object { $_.Thumbprint -eq $thumbprint } - - if (-not $cert) { - Write-Error "Certificate with thumbprint $thumbprint not found." - $store.Close() - return - } - - if (-not $cert.HasPrivateKey) { - Write-Error "Certificate does not have a private key." - $store.Close() - return - } - - $uri = "https://:9090/v1/system/seal-status" - try { - $response = Invoke-RestMethod -Uri $uri -Method Get -Certificate $cert - Write-Output "Seal status: $response" - } catch { - Write-Error "API call failed: $($_.Exception.Message)" - } - finally { - $store.Close() - } - ``` - - The output will look similar to: - ``` - Seal status: False - ``` +> [!NOTE] +> For troubleshooting steps related to the Relativity Secret Store, please refer to the [Pre-requisite Troubleshooting](pre-requisite-troubleshooting.md) guide. -### Kepler & Web (SSL Certificate) Verification -- The required web certificate must be installed on the server (check with your certificate management process or MMC snap-in for Certificates). +### 5.2. Kepler and Web (SSL Certificate) Verification +- The required web certificate must be installed on the server. For certificate troubleshooting, see the [Pre-requisite Troubleshooting](pre-requisite-troubleshooting.md) guide. - Verify Kepler API status: ```powershell curl.exe -k -u : -X GET "https:///relativity.rest/api/relativity-infrawatch-services/v1/infrawatch-manager/getkeplerstatusasync" @@ -333,17 +274,17 @@ TcpTestSucceeded : True JSON response with `"status": "OK"`. -### Relativity Service Account Verification +### 5.3. Relativity Service Account Verification > [!NOTE] > For service account requirements and troubleshooting, see [Environment_Watch_Installer](../environment_watch_installation.md) --- -## Installer and Service Errors +## 6. Installer and Service Errors This section covers issues related to the Environment Watch installer and the underlying Windows services it manages. -### User Not in Local Security Policy +### 6.1. User Not in Local Security Policy **Symptoms:** - The product installation fails with an error indicating the user is not added to the Local Security Policy. @@ -358,7 +299,7 @@ This section covers issues related to the Environment Watch installer and the un ![User added to Local Security Policy](../../resources/troubleshooting-images/useraddedtolocalsecurity.png) -### Invalid Secrets +### 6.2. Invalid Secrets **Symptoms:** - The installation fails with an error message "one or more secrets are invalid". @@ -371,5 +312,5 @@ This section covers issues related to the Environment Watch installer and the un --- For additional troubleshooting, refer to the main documentation: -[Environment_Watch_Installer](../environment_watch_installation.md) +[Environment Watch Installer](../environment_watch_installation.md) diff --git a/docs/troubleshooting/pre-requisite-troubleshooting.md b/docs/troubleshooting/pre-requisite-troubleshooting.md new file mode 100644 index 0000000..9715c7e --- /dev/null +++ b/docs/troubleshooting/pre-requisite-troubleshooting.md @@ -0,0 +1,409 @@ +# Pre-requisite Troubleshooting + +This document provides troubleshooting steps for common pre-requisites like port configuration and Secret Store access. + +## Table of Contents + +- [Port Configuration](#port-configuration-troubleshooting) +- [Secret Store](#secret-store-troubleshooting) +- [Certificate](#certificate-troubleshooting) + +--- + +## Port Configuration Troubleshooting + +### Default Port Reference +The following table summarizes the default ports used by the Elastic Stack and Environment Watch components. + +| Component | Port | Protocol | Inbound | Outbound | Purpose | +| :--- | :--- | :--- | :---: | :---: | :--- | +| Elasticsearch | 9200 | HTTP/HTTPS | ✅ | ✅ | Client communication and REST API | +| | 9300 | TCP | ✅ | ✅ | Inter-node communication | +| Kibana | 5601 | HTTP/HTTPS | ✅ | | Kibana web interface | +| APM Server | 8200 | HTTP/HTTPS | ✅ | | APM agent data ingestion | +| OTEL Collector | 4318 | HTTP | | | OTLP data reception (HTTP) for local traffic (localhost). This deployment uses the agent model, with a collector on each server. See the [OpenTelemetry agent documentation](https://opentelemetry.io/docs/collector/deployment/agent/) for more details. | + +--- + +### Elasticsearch Port Issues + +#### Symptoms: +- Elasticsearch fails to bind to default ports. +- "Address already in use" errors in logs. +- Cannot access Elasticsearch via HTTP/HTTPS. + +#### Troubleshooting Steps: + +* **Check if Ports are in Use:** + Verify that ports 9200 and 9300 are listening. + ```powershell + netstat -an | findstr ":9200" + netstat -an | findstr ":9300" + ``` +
+ Expected output + + ``` + TCP 0.0.0.0:9200 0.0.0.0:0 LISTENING + TCP 0.0.0.0:9300 0.0.0.0:0 LISTENING + ``` +
+ +* **Identify Conflicting Processes:** + If a port is in use by another application, identify the process. + ```powershell + Get-NetTCPConnection -LocalPort 9200 -State Listen + Get-NetTCPConnection -LocalPort 9300 -State Listen + ``` + +* **Test Elasticsearch Connectivity:** + ```powershell + curl.exe -k -u : -X GET "https://:9200/" + ``` + +* **Verify Network Binding:** + Check `C:\elastic\elasticsearch\config\elasticsearch.yml` configuration: + ```yaml + network.host: 0.0.0.0 # For all interfaces + ``` + +--- + +### Kibana Port Issues + +#### Symptoms: +- Kibana fails to bind to the default port. +- "EADDRINUSE" errors in logs. +- Cannot access Kibana web interface. + +#### Troubleshooting Steps: + +* **Check if Port is in Use:** + ```powershell + netstat -an | findstr ":5601" + ``` +
+ Expected output + + ``` + TCP 0.0.0.0:5601 0.0.0.0:0 LISTENING + ``` +
+ +* **Test Kibana Connectivity:** + ```powershell + (curl.exe -s -k -u : -X GET "http://:5601/api/status" | ConvertFrom-Json).status.overall | ConvertTo-Json -Depth 10 + ``` + +* **Verify Network Binding:** + Check `C:\elastic\kibana\config\kibana.yml` configuration: + ```yaml + server.host: "0.0.0.0" # For all interfaces + ``` + +--- + +### APM Server Port Issues + +#### Symptoms: +- APM Server fails to bind to the default port. +- "Address already in use" errors in logs. +- APM agents cannot connect to the server. + +#### Troubleshooting Steps: + +* **Check if Port is in Use:** + ```powershell + netstat -an | findstr ":8200" + ``` +
+ Expected output + + ``` + TCP 0.0.0.0:8200 0.0.0.0:0 LISTENING + ``` +
+ +* **Test APM Server Connectivity:** + ```powershell + curl.exe -k "http://:8200/" + ``` +
+ Expected output + + ```json + { + "build_date": "...", + "build_sha": "...", + "publish_ready": true, + "version": "8.17.3" + } + ``` +
+ +* **Verify Network Binding:** + Check `C:\elastic\apm-server\apm-server.yml` configuration: + ```yaml + host: "0.0.0.0:8200" + ``` + +--- + +### OpenTelemetry Collector Port Issues + +#### Symptoms: +- The `otelcol-relativity.exe` process is running, but no data is being sent. +- Port 4318 is not in a listening state. + +#### Troubleshooting Steps: + +* **Check if Port is in Use:** + This port is used by the OpenTelemetry Collector to receive data. The `Relativity Environment Watch` service must be running. + ```powershell + netstat -an | findstr ":4318" + ``` +
+ Expected output + + ``` + TCP 0.0.0.0:4318 0.0.0.0:0 LISTENING + ``` +
+ + You can also use `Get-NetTCPConnection`: + ```powershell + Get-NetTCPConnection -LocalPort 4318 -State Listen + ``` + +--- + +### General Port Troubleshooting + +#### Firewall Rules: + Ensure that Windows Firewall or any other network security software is not blocking the required ports. You may need to create inbound rules to allow traffic on these ports. + + **Example for Kibana (port 5601):** + ```powershell + New-NetFirewallRule -DisplayName "Kibana Web Interface" -Direction Inbound -Protocol TCP -LocalPort 5601 -Action Allow + ``` + +#### Network Connectivity: + Use `Test-NetConnection` to verify that a remote server can reach the port. + ```powershell + Test-NetConnection -ComputerName -Port + ``` +
+ Expected output + + ``` + ComputerName : + RemoteAddress : + RemotePort : + TcpTestSucceeded : True + ``` +
+ +--- + +## Secret Store Troubleshooting + +### Secret Store Access Verification + +#### Network Connectivity Test + +Verify that the Secret Store host is reachable on port 443. + +```powershell +Test-NetConnection -ComputerName -Port 443 +``` + +
+Expected output + +``` +ComputerName : +RemoteAddress : +RemotePort : 443 +TcpTestSucceeded : True +``` +
+ +#### API Access Test + +1. Open an elevated PowerShell and run the following command to list secrets and retrieve connection details: + + ```powershell + C:\Program Files\Relativity Secret Store\Client\secretstore.exe secret list / + ``` + + The output will look similar to: + ``` + Secret Store URL: https://:9090/ + Client Certificate Thumbprint: 20F8F2516EC86EBF993075F64B0C6EA6777A4F83 + ``` + +2. Copy the **Client Certificate Thumbprint** and **Secret Store URL** from the output. + +#### Seal Status Check + +To check the seal status of the Secret Store, run the following script in an elevated PowerShell ISE. + +1. Replace `` with the thumbprint you copied. +2. Replace `` with the URL you copied. + +```powershell +$thumbprint = "" +$url = "" + +# Find the client certificate +$store = New-Object System.Security.Cryptography.X509Certificates.X509Store("My", "LocalMachine") +$store.Open("ReadOnly") +$cert = $store.Certificates | Where-Object { $_.Thumbprint -eq $thumbprint } + +if (-not $cert) { + Write-Error "Certificate with thumbprint $thumbprint not found." + return +} + +# Check the seal status +$response = Invoke-RestMethod -Uri "$url/v1/sys/seal-status" -Certificate $cert +$response | ConvertTo-Json +``` + +
+Expected output (for a healthy, unsealed store) + +```json +{ + "type": "shamir", + "initialized": true, + "sealed": false, + "t": 3, + "n": 5, + "progress": 0, + "nonce": "", + "version": "1.6.2", + "migration": false, + "cluster_name": "secret-store", + "cluster_id": "...", + "recovery_seal": false, + "storage_type": "raft" +} +``` +
+ + +## Certificate Troubleshooting + +### SSL/TLS Certificate Issues + +#### Symptoms: +- SSL handshake failures +- "certificate verify failed" errors +- Unable to establish secure connections +- Browser shows "not secure" warning for Elasticsearch URL + +#### Troubleshooting Steps: + +* **Verify Secure URL** + - The master node domain name URL should be secure for Elasticsearch node servers, agent servers, and web servers. + - The data node domain name URL should be secured for Elasticsearch node servers. + +* **Install SSL Certificate in Trusted Store** + + If your browser shows a "not secure" warning when accessing the Elasticsearch URL, you may need to install the certificate into your trusted store. + + a. In your browser, view the certificate details and export the root certificate authority (CA) certificate. Save it to a local directory. + + b. Double-click the downloaded certificate file and click **Install Certificate**. + + ![Install Certificate](../../resources/troubleshooting-images/installcertificate.png) + + c. Select **Local Machine** and click **Next**. + + ![Select Local Machine](../../resources/troubleshooting-images/localmachine.png) + + d. Select **Place all certificates in the following store**, click **Browse**, and select **Trusted Root Certification Authorities**. Click **OK**, then **Next**, and **Finish**. + + e. To confirm, open the Microsoft Management Console (MMC): + i. Run `mmc.exe`. + ii. Go to **File > Add/Remove Snap-in...**. + iii. Select **Certificates** and click **Add**. + + ![Add/Remove Snap-in](../../resources/troubleshooting-images/Add-removesnipin.png) + + ![Add Certificates Snap-in](../../resources/troubleshooting-images/addcerts.png) + + f. Choose **Computer account** and click **Next**, then **Finish**, and **OK**. + + ![Select Computer Account](../../resources/troubleshooting-images/clickcomputeraccount.png) + + g. Expand **Certificates (Local Computer) > Trusted Root Certification Authorities > Certificates** and verify your certificate is listed. + + h. Close your browser and reopen the Elasticsearch URL. It should now show as secure. + + ![Secure Connection](../../resources/troubleshooting-images/sslenabled.png) + +* **Verify Certificate Path in `elasticsearch.yml`** + + Ensure the `elasticsearch.yml` file points to the correct certificate files. + + - Check `C:\elastic\elasticsearch\config\elasticsearch.yml`: + ```yaml + xpack.security.transport.ssl: + keystore.path: certs/transport.p12 + truststore.path: certs/transport.p12 + ``` + +* **Check Elasticsearch Logs for SSL Errors** + - Navigate to `C:\elastic\elasticsearch\logs\`. + - Review the `elasticsearch.log` file for any SSL-related errors. + - For every error in the Elasticsearch log, provide troubleshooting for that specific error. + +--- + +### TLS Version Mismatch + +#### Symptoms: +- **Connection Failure**: During installation of Environment Watch Installer, an error pop-up may appear with a message like: + > The HTTP request submitted to the server `https://:9200/` failed because of an unexpected error. Verify the server is accessible and URL is correct. Check the logs for more details or refer to the following troubleshooting guide. +- **Log Errors**: Logs on the application's server indicate a failure to establish a secure connection or mention outdated TLS versions (like TLSv1.0 or TLSv1.1). +- "SSLHandshakeException: client requested protocol TLSv1 is not enabled or supported in server context" + +#### Cause: +The machine's .NET Framework is not configured to use strong cryptography, preventing it from negotiating a secure connection with modern servers that require TLS 1.2 or higher. By default, some .NET applications may attempt to use older, insecure TLS versions. Default TLS version supported by Elasticsearch is TLSv1.2 and TLSv1.3, hence causing issue + +#### Troubleshooting Steps: + +To resolve this, the .NET Framework on the machine must be configured to use the system's default security protocols, which allows it to use newer versions like TLS 1.2/1.3. + +1. **Verify TLS Settings**: + - Navigate to **Control Panel > Network and Internet > Internet Options > Advanced**, ensure that **Use TLS 1.2** and **Use TLS 1.3** are checked. + +2. **Open Registry Editor**: + - Press `Win + R`, type `regedit`, and press Enter. + +3. **Navigate to .NET Framework Registry Keys**: + A new value will need to be added in two locations. + + - **For 64-bit applications:** + ``` + HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319 + ``` + - **For 32-bit applications running on a 64-bit machine:** + ``` + HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319 + ``` + +4. **Create the `SchUseStrongCrypto` Value**: + - In each of the keys mentioned above, right-click in the right-hand pane and select **New > DWORD (32-bit) Value**. + - Name the new value `SchUseStrongCrypto`. + - Double-click the new value and set its **Value data** to `1`. Click **OK**. + + > **Note:** This registry key forces .NET 4.x applications to use strong cryptography, enabling support for newer TLS versions. + +5. **Reboot the System**: + - A system reboot is required for the changes to take effect. + +6. **Verify the Fix**: + - After rebooting, re-run the Environment Watch installer or restart the application. The connection to Elasticsearch should now succeed. \ No newline at end of file diff --git a/resources/relativity_alerts_verification_004_005.png b/resources/relativity_alerts_verification_004_005.png index f42618e..e6712f6 100644 Binary files a/resources/relativity_alerts_verification_004_005.png and b/resources/relativity_alerts_verification_004_005.png differ