> ## Documentation Index
> Fetch the complete documentation index at: https://docs.resolve.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Splunk Enterprise

Connect ResolveAI to Splunk to query logs and saved searches during investigations.

<Info>
  **Prerequisite**: First install the [Resolve Satellite](/resolve-satellite) to create the Splunk connection.
</Info>

### Create Splunk Service Account

First [Set up a dedicated service account in Splunk](https://docs.splunk.com/Documentation/SIM/current/User/Setupserviceaccount). Create a Service Account User role with "**Power**" Role to provide access to the Splunk REST API endpoint.

Additionally, ensure the role has the **`rest_access_server_endpoints`** capability, which is not included in the Power role by default.

<img src="https://mintcdn.com/resolveai-0e94a547/GuuijYwp3RfE7Xrj/images/external/710294d76d29.png?fit=max&auto=format&n=GuuijYwp3RfE7Xrj&q=85&s=22dd94e9db80dbc65bfba89759b039aa" alt="" width="1600" height="1376" data-path="images/external/710294d76d29.png" />

### Create Splunk authentication token for REST API access

Create REST API endpoint with the Service Account information in the prerequisites.  This user will have "**Power**" Role permissions applied to it.

<Accordion title="Option 1: Create Splunk authentication token via GUI [Recommended]">
  Official Splunk documentation - [Create authentication tokens](https://docs.splunk.com/Documentation/Splunk/9.4.0/Security/CreateAuthTokens)

  Input the following information in the required fields.

  * In the "**User**" field set the service account user name that you created above.
  * In the "**Audience**" field add a specific identifier for the target application (eg resolve-ai-access)

  <img src="https://mintcdn.com/resolveai-0e94a547/GuuijYwp3RfE7Xrj/images/external/af22dbc1e6dc.png?fit=max&auto=format&n=GuuijYwp3RfE7Xrj&q=85&s=e46852af770df36e597b7c467447e47a" width="48%" alt="" data-path="images/external/af22dbc1e6dc.png" />
</Accordion>

<Accordion title="Option 2: Create Splunk authentication token via REST API">
  Official Splunk documentation - [Use REST to create authentication tokens](https://docs.splunk.com/Documentation/Splunk/9.4.0/Security/CreateAuthTokens#Example_API_calls_for_creating_tokens)

  Run the following command. Please ensure that you replace the following properties:

  * **poweruser-name:** This is the username of the service account that was configured.
  * **poweruser-password:** This is the password of the service account that was configured.
  * **splunk-server**: This is your Splunk endpoint.

  ```shell create token via API theme={null}
  curl -k -u <poweruser-name>:<poweruser-password> -X POST https://<splunk-server>:8089/services/authorization/tokens?output_mode=json \
  --data name=resolve-access-key  --data audience=Power
  ```
</Accordion>

***

<Info>
  **Note**: When creating Splunk Authentication Token you must copy the Bearer Token.  You will not be able to retrieve it from Splunk after creation.  Store this token in a secure manner, and do not check into source code. This token will be used in your ResolveAI configuration.
</Info>

## Configure the Splunk Integration in the Satellite

Below is an example of how to setup the Splunk integration in the satellite with the url property as well as using the Kubernetes secret (potentially backed by an AWS secret manager or another mechanism) for authentication. The secret format in this example is for a Splunk REST authentication token.

<Steps>
  <Step title="Create a Kubernetes secret">
    Create a Kubernetes secret of the following form. Note that the structure of the secret is important and for a Splunk authentication token, it must have the top-level key token: ‘token-value’.

    ```yaml secret creation theme={null}
    apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name:  splunk-resolve-access-token
    stringData:
      token: "<your splunk token>"
    ```

    <Warning>
      **Security Best Practice:** Never store credentials in plaintext in configuration files or source control.
      Always use Kubernetes secrets and encrypt etcd or use enterprise secret management systems.
      See [Secret Management](/secret-management) for detailed guidance.
    </Warning>

    To apply the secret run

    ```shell apply secret theme={null}
    kubectl apply -f splunk-resolve-access-token.yml
    ```
  </Step>

  <Step title="Configure your Splunk authentication token in the Resolve Satellite">
    Update your helm values override file with the following information (e.g.: *resolve-values.yaml*)

    ```yaml resolve-values.yaml theme={null}
    integrations:
      splunkIntegration:
        type: splunk
        create: true
        secretName:  "splunk-resolve-access-token"
        connection:
          url: "<your splunk server>"
          mandatoryQueryFields: ["index", "splunk_server_group","sourcetype"]
    ```

    <Info>
      **mandatoryQueryFields**: Ensure every query generated by Resolve includes explicit scoping labels, preventing overly broad wildcard searches and ensuring efficient, targeted query execution. The example above includes index, splunk\_server\_group, and sourcetype.
    </Info>

    Install the satellite and apply the values from the yaml file that you have just updated. e.g.: *resolve-values.yaml.* To find the latest version, visit ResolveAI's docker hub repository for the [helm chart](https://hub.docker.com/r/resolveaihq/satellite-chart/tags) and [satellite image](https://hub.docker.com/r/resolveaihq/satellite/tags).

    ```shell apply config to satellite and redeploy theme={null}
    helm upgrade --install oci://registry-1.docker.io/resolveaihq/satellite-chart --version <LatestChart> --values resolve-values.yaml --set image.tag=<LatestImage>
    ```

    Once your satellite is deployed, we will automatically create an integration instance for you.
  </Step>

  <Step title="Verify your integration status in ResolveAI">
    In ResolveAI, go to the [Splunk Integration Page](https://app0.resolve.ai/integrations/splunk/connect) to validate that the health checks were successful.
  </Step>
</Steps>

## Send alerts to ResolveAI via webhook

If you use create and manage alerts in Splunk, follow the steps to complete the alert integration for your environment.

<Steps>
  <Step title="Go to the Splunk integration">
    Go to the Splunk integration you have set up and click into it.

    <img src="https://mintcdn.com/resolveai-0e94a547/GuuijYwp3RfE7Xrj/images/external/1e62f93ead79.png?fit=max&auto=format&n=GuuijYwp3RfE7Xrj&q=85&s=51789fa150cb501b43bf7adf5fceebaa" alt="" width="1786" height="426" data-path="images/external/1e62f93ead79.png" />
  </Step>

  <Step title="Scroll down to the Webhooks section">
    Scroll down to the **Webhooks** section. Click "Edit".
  </Step>

  <Step title="Add a new Webhook token">
    Click on the "+" icon to add a new Webhook token. Click "Save".

    <img src="https://mintcdn.com/resolveai-0e94a547/GuuijYwp3RfE7Xrj/images/external/6f82ad6ba31e.png?fit=max&auto=format&n=GuuijYwp3RfE7Xrj&q=85&s=283506d085533ee07edce54c5ef4461b" alt="" width="1786" height="426" data-path="images/external/6f82ad6ba31e.png" />
  </Step>

  <Step title="Add a new webhook integration in Splunk">
    Follow the instructions to add a new webhook integration in Splunk.

    * Log into your Splunk instance.
    * Add the ResolveAI webhook URL to your alert actions.
      * You’ll find this webhook URL in your ResolveAI setup instructions.
      * In Splunk, navigate to the alert(s) you want to forward and add the URL as a webhook alert action.
      * Ensure that the alert(s) you want to forward to ResolveAI have "Sharing" set to "**App**" or "**Global**".
    * Allowlist the ResolveAI webhook endpoint.
      * Ensure the provided URL is included in Splunk’s webhook allowlist so alerts can be sent successfully.
  </Step>
</Steps>

## (Optional) Obfuscate ResolveAI webhook API and token in Splunk internal logs

As an additional layer of security, mask the ResolveAI URI in the Splunk `_internal` index. The official Splunk documentation can be found here: [Anonymize data in Splunk](https://docs.splunk.com/Documentation/Splunk/9.4.0/Data/Anonymizedata)

<Steps>
  <Step title="Create a props.conf file">
    Create a `props.conf` file on the Splunk Search heads that will be sending the Splunk Alert to ResolveAI in the /opt/splunk/etc/system/local directory.
  </Step>

  <Step title="Place the following code in that props.conf file">
    Place the following code in that `props.conf` file

    ```yaml props.conf theme={null}
    [source::.../var/log/splunk/splunkd*]
    SEDCMD-url = s/https?:\/\/api.app0.resolve.ai\/(.*)/https:\/\/api.app0.resolve.ai\/XXXX-XXXX-XXXX/g
    ```
  </Step>

  <Step title="Restart splunkd process to apply changes in props.conf">
    Restart **splunkd** process to apply changes in `props.conf`

    Option 1: CLI restart

    ```shell CLI restart theme={null}
    /opt/splunk/bin/splunk restart
    ```

    Option 2: GUI restart

    Navigate to Settings > Server Controls > Restart Splunk

    <img src="https://mintcdn.com/resolveai-0e94a547/GuuijYwp3RfE7Xrj/images/external/3f0cab3b10de.png?fit=max&auto=format&n=GuuijYwp3RfE7Xrj&q=85&s=be82bbfdf08411828eabf3b74a4ef3d1" width="58%" alt="" data-path="images/external/3f0cab3b10de.png" />
  </Step>
</Steps>
