Splunk

connect resolve ai to splunk to query observability data for investigations prerequisite first install the resolve satellite /resolve satellite md to create the splunk connection create splunk service account first set up a dedicated service account in splunk https //docs splunk com/documentation/sim/current/user/setupserviceaccount create a service account user role with " power " role to provide access to the splunk rest api endpoint create splunk authentication token for rest api access create rest api endpoint with the service account information in the prerequisites this user will have " power " role permissions applied to it option 1 create splunk authentication token via gui \[recommended] official splunk documentation create authentication tokens https //docs splunk com/documentation/splunk/9 4 0/security/createauthtokens input the following information in the required fields in the " user " field set the service account user name that you created above in the " audience " field add a specific identifier for the target application (eg resolve ai access) option 2 create splunk authentication token via rest api official splunk documentation use rest to create authentication tokens https //docs splunk com/documentation/splunk/9 4 0/security/createauthtokens#example api calls for creating tokens run the following command please ensure that you replace the following properties poweruser name this is the username of the service account that was configured poweruser password this is the password of the service account that was configured splunk server this is your splunk endpoint create token via api curl k u \<poweruser name> \<poweruser password> x post https //\<splunk server> 8089/services/authorization/tokens?output mode=json \\ \ data name=resolve access key data audience=power note when creating splunk authentication token you must copy the bearer token you will not be able to retrieve it from splunk after creation store this token in a secure manner, and do not check into source code this token will be used in your resolve ai configuration configure the splunk integration in the satellite below is an example of how to setup the splunk integration in the satellite with the url property as well as using the kubernetes secret (potentially backed by an aws secret manager or another mechanism) for authentication the secret format in this example is for a splunk rest authentication token create a kubernetes secret create a kubernetes secret of the following form note that the structure of the secret is important and for a splunk authentication token, it must have the top level key token ‘token value’ secret creation apiversion v1 kind secret type opaque metadata name splunk resolve access token stringdata token "\<your splunk token>" to apply the secret run apply secret kubectl apply f splunk resolve access token yml configure your splunk authentication token in the resolve satellite update your helm values override file with the following information (e g resolve values yaml ) resolve values yaml integrations splunkintegration type splunk create true secretname "splunk resolve access token" connection url "\<your splunk server>" install the satellite and apply the values from the yaml file that you have just updated e g resolve values yaml to find the latest version, visit resolve ai's docker hub repository for the helm chart and satellite image apply config to satellite and redeploy helm upgrade install oci //registry 1 docker io/resolveaihq/satellite chart version \<latestchart> values resolve values yaml set image tag=\<latestimage> once your satellite is deployed, we will automatically create an integration instance for you verify your integration status in resolve ai in resolve ai, go to the splunk integration page to validate that the health checks were successful send alerts to resolve ai via webhook if you use create and manage alerts in splunk, follow the steps to complete the alert integration for your environment go to the splunk integration you have set up and click into it scroll down to the webhooks section click "edit" click on the "+" icon to add a new webhook token click "save" follow the instructions to add a new webhook integration in splunk log into your splunk instance add the resolve ai webhook url to your alert actions you’ll find this webhook url in your resolve ai setup instructions in splunk, navigate to the alert(s) you want to forward and add the url as a webhook alert action ensure that the alert(s) you want to forward to resolve ai have "sharing" set to " app " or " global " allowlist the resolve ai webhook endpoint ensure the provided url is included in splunk’s webhook allowlist so alerts can be sent successfully (optional) obfuscate resolve ai webhook api and token in splunk internal logs as an additional layer of security, mask the resolve ai uri in the splunk internal index the official splunk documentation can be found here anonymize data in splunk https //docs splunk com/documentation/splunk/9 4 0/data/anonymizedata create a props conf file on the splunk search heads that will be sending the splunk alert to resolve ai in the /opt/splunk/etc/system/local directory place the following code in that props conf file props conf \[source /var/log/splunk/splunkd ] sedcmd url = s/https? \\/\\/api app0 resolve ai\\/( )/https \\/\\/api app0 resolve ai\\/xxxx xxxx xxxx/g restart splunkd process to apply changes in props conf option 1 cli restart cli restart /opt/splunk/bin/splunk restart option 2 gui restart navigate to settings > server controls > restart splunk