Resolve Satellite

the resolve satellite is a lightweight kubernetes native agent that runs within your infrastructure, securely proxying queries to your observability systems and streaming results back to resolve ai through encrypted channels while keeping credentials behind your firewall 1\ core capabilities understanding your infrastructure the satellite continuously monitors your kubernetes cluster through the api server, building a comprehensive model of your infrastructure it tracks pods, services, deployments, jobs, and more across your configured namespaces the satellite operates in a truly read only mode with zero impact on your api servers discovering service dependencies in real time traditional service dependency mapping requires instrumentation, apm agents, or manual configuration the resolve satellite takes a different approach with visibility into how your services interact, enabling resolve ai to understand dependencies without requiring changes to your application code connecting on premises observability tools the resolve satellite acts as a secure local proxy that enables resolve ai to query your local observability systems and stream results back through encrypted channels your credentials remain entirely within your infrastructure, and sensitive data can be redacted before transmission the satellite supports a comprehensive range of integration types scaling to meet your needs the satellite's architecture supports both vertical and horizontal scaling to handle environments of any size for most deployments, a single satellite is sufficient the satellite operates statelessly, and when you need additional capacity for very large clusters or high integration query volumes, it scales horizontally through multiple replicas load balancing across replicas is managed automatically by resolve's backend multiple replicas also provide high availability, allowing you to perform rolling upgrades without disrupting resolve's ability to investigate incidents 2\ deployment & upgrades getting the resolve satellite running in your environment takes just a few steps you'll generate an ingest token, configure your values file, install the helm chart, and verify connectivity prerequisites before installing the satellite, ensure you have kubernetes cluster access with kubectl , helm 3 x installed, and admin access to resolve ai to generate ingest tokens install the resolve satellite the installation process configures the satellite to connect securely to resolve's backend and begin a connection from your kubernetes cluster 1\ create ingest token the satellite communicates with the resolve backend over an authenticated, encrypted channel authentication is handled via an ingest token you generate in the resolve ui as an admin in resolve ai, open the https //app0 resolve ai/admin/tokens page click create ingest token , name it descriptively (e g , "production main cluster"), click create ingest token , and copy the token immediately you will need it in the next step 2\ configure your ingest token you'll need to create a resolve values yaml file with your configuration before proceeding, choose how you want to handle the ingest token based on your environment production/secure setup (recommended) for production environments, configure secure token storage using kubernetes secrets or external secrets managers continue to docid\ gtjlczhaf1mgx v 76iw3 to set up secure token storage, then return here for step 3 the secret management guide will walk you through storing tokens in kubernetes secrets with an encrypted etcd and integrating with external secrets managers like aws secrets manager, azure key vault, and hashicorp vault quick test setup (development only) for quick local testing, you can store the token directly in your values file this method provides no additional security and should only be used temporarily required values ingest token the token created in step 1 clustername global identifier for the satellite (e g , prod main ) environment e g , production , staging , qa match this environment name exactly with each integration you set up (e g , datadog) resolve values yaml ingest token \<your resolve satellite token> clustername \<your cluster name> environment \<your environment> 3\ install helm chart use this command to install the https //hub docker com/r/resolveaihq/satellite chart/tags helm install resolve satellite \\ oci //registry 1 docker io/resolveaihq/satellite chart \\ \ values resolve values yaml if installation is rejected due to missing labels some kubernetes clusters use admission controllers or policies that require specific labels on pods before they can be deployed if your installation fails with a message about missing labels, add the labels required by your cluster policies to your resolve values yaml podlabels team your team name \# add any other labels required by your cluster policies alternative install via cdk8s if you're using https //cdk8s io/ to manage your kubernetes deployments, you can deploy the satellite using a cdk8s helm chart construct instead of running helm commands directly the example below uses typescript and is for the latest version of cdk8s syntax may vary depending on your programming language (python, java, go, etc ) and cdk8s version refer to the https //cdk8s io/docs/latest/ for language specific examples example cdk8s typescript code import { app, chart } from 'cdk8s'; import { helm } from 'cdk8s'; const app = new app(); const chart = new chart(app, 'satellite'); new helm(chart, 'resolve satellite', { chart 'oci //registry 1 docker io/resolveaihq/satellite chart', namespace 'resolve', // specify namespace here createnamespace true, // create if doesn't exist releasename 'resolve satellite', values { ingest { tokensecretname 'satellite token', }, clustername 'prod cluster', environment 'production', }, }); app synth(); this approach generates the kubernetes yaml manifests from your cdk8s code you cannot directly edit yaml files or run helm commands when using cdk8s all configuration must be specified in the cdk8s chart definition 4\ check permissions the resolve satellite reads from the kubernetes api with a clusterrole verify the permissions granted describe permissions kubectl describe clusterrole resolve satellite core resources configmaps, events, namespaces, nodes, pods, services workload controllers daemonsets, deployments, replicasets, statefulsets crds argorollout, istiovirtualservice, istiogateway 5\ verify connection in the resolve ui, check https //app0 resolve ai/integrations/kubernetes/connect to verify that the satellite is connected next enable dns tap now that the resolve satellite is installed, enable cross application investigations with resolve using docid 3qezzvvnrr90beexvhe8 upgrading the satellite the satellite supports rolling upgrades with zero downtime when new versions are released, you can upgrade your deployment using helm while maintaining connectivity to resolve's backend upgrade to latest version to upgrade your satellite to the latest version, use the helm upgrade command this pulls the latest satellite image and applies any configuration changes from your values file helm upgrade resolve satellite \\ oci //registry 1 docker io/resolveaihq/satellite chart \\ \ values resolve values yaml upgrade to specific version to upgrade to a specific version, specify the chart version in your command view available versions at https //hub docker com/r/resolveaihq/satellite chart/tags helm upgrade resolve satellite \\ oci //registry 1 docker io/resolveaihq/satellite chart \\ \ version \<chart version> \\ \ values resolve values yaml verify upgrade after upgrading, verify the satellite is running the new version check the pod status and image version kubectl get pods l app=resolve satellite kubectl describe pod \<satellite pod name> | grep image check the resolve ui https //app0 resolve ai/integrations/kubernete/connect page to confirm the satellite is connected and healthy rollback if needed if you encounter issues after upgrading, rollback to the previous version helm rollback resolve satellite 3\ integrations the resolve satellite connects resolve ai to your on premises observability stack each integration is configured in your resolve values yaml file with credentials stored in kubernetes secrets the satellite automatically detects configuration changes through a built in file watcher—no manual restarts required configuration approach all integrations follow a consistent pattern create a kubernetes secret containing credentials (api tokens, passwords) reference the secret in your resolve values yaml file specify connection parameters (urls, filters, options) deploy or upgrade your satellite with the new configuration secret management for production environments, we strongly recommend using secure secret storage through kubernetes secrets with etcd encryption at rest or external secret managers like aws secrets manager, azure key vault, hashicorp vault, and google secret manager for a complete guide on configuring secrets, see docid\ gtjlczhaf1mgx v 76iw3 integration examples the kubernetes integration is included by default to customize namespace filtering, specify which namespaces to include in your connection configuration for example, to limit scraping to production, staging, and monitoring namespaces, set namespaceincludelist to a comma separated string of namespace names integrations kubernetesonprem type kubernetes create true secretname \<your kubernetes secret> connection namespaceincludelist "production,staging,monitoring" on premises observability tools example configurations for common tools demonstrate the standard integration pattern for prometheus, enable service map construction from traces if desired integrations prometheusonprem type prometheus create true secretname prometheus token connection url https //prometheus internal company com servicemapenabled true for grafana, enable alert scraping to ingest alerts from grafana's alertmanager integrations grafanaonprem type grafana create true secretname grafana token connection url https //grafana internal company com alertscrapingenabled true for elasticsearch, specify which indices to allow for log queries using an index pattern integrations elasticsearchonprem type elasticsearch create true secretname elasticsearch token connection url https //elasticsearch internal company com indexallowlist \ application logs \ system logs for detailed configuration parameters for each integration type, including filters, multi tenancy options, and advanced settings, see the docid\ kzviowhvnakd0nd5lb4zf git integration the git integration enables resolve ai to access repositories for investigations for a complete setup guide, see docid\ vxopp eddopxndviyq1wv 4\ security controls the resolve satellite is designed with security as a foundational principle this section covers the security features and controls available for hardening your deployment encrypted communication all communication between the satellite and resolve's backend is over http/2 0 with encryption these long lived connections span 30+ minutes and use token authentication to verify every request secret storage & management all integration credentials (api tokens, passwords) are stored in kubernetes secrets or within your secrets manager within your cluster the satellite mounts these secrets as files at /etc/secrets/\<secret name>/ raw secret values never leave your infrastructure or transit to resolve's cloud for production deployments, we recommend using kubernetes secrets with etcd encryption at rest or integrating external secret managers like aws secrets manager, hashicorp vault, and azure key vault rotate credentials regularly using secret updates without satellite downtime for a complete guide, see docid\ gtjlczhaf1mgx v 76iw3 configuration immutability (config override) the configoverride setting (default false ) controls whether resolve's saas platform can remotely modify integration configurations through the ui when disabled (default), all configuration changes require helm chart updates this enforces gitops workflows and change approval processes, ensuring configuration remains immutable and auditable no remote configuration tampering is possible when enabled, administrators can edit integration settings from the resolve web interface this is useful for rapid testing and development environments but not recommended for production by defaulting to false , the satellite enforces the principle of least privilege and configuration as code practices satellite config override domain allowlist the domainallowlist setting restricts outbound api requests from the satellite to specific integration domains this prevents data exfiltration and unauthorized external api calls the allowlist accepts an array of domain strings and supports wildcard patterns (e g , example com for subdomains) the satellite validates all http requests before execution and throws an error if the target domain is not allowed an empty list permits all domains (default) example for grafana integrations grafanaonprem type grafana create true secretname grafana resolve secret connection url https //yourteam grafana com domainallowlist \ yourteam grafana com \ " yourcompany com" audit logging audit logging captures detailed information about every http request made to external integrations for security, compliance, and debugging purposes when enabled, audit logging records the request method, url, request body, response status code, integration key, and success/failure status all audit logs are tagged with "audit" true for filtering enable audit logging in your integration configuration integrations kubernetesonprem type kubernetes create true secretname \<your kubernetes secret> connection auditlogging true view audit logs in your satellite pod kubectl logs deployment/resolve satellite tail=100 | grep '"audit"\ true' audit logging provides a comprehensive audit trail for security investigations and compliance audits sensitive data redaction the satellite supports configurable pii redaction for integrations that collect or process sensitive data specify json paths to redact in the integration configuration the satellite will mask values at those paths before transmitting data example for datadog integrations datadogonprem type datadog create true secretname datadog credentials connection site datadoghq com redactionconfig enabled true targetjsonpaths \ "$ tags user email" \ "$ attributes customer name" for a complete guide, see docid\ gyvtsegffutdvtrassock https certificate configuration for integrations behind custom certificate authorities or requiring mutual tls, configure certificates in your integration settings certificate options include httpagentcertificate (ca certificate for verifying the server), httpagentclientcert (client certificate for mutual tls), and httpagentclientkey (client private key for mutual tls) certificates can be specified as raw content or file paths example for splunk with mutual tls splunkintegration secretdir /etc/secrets/splunkintegration create true type splunk connection url https //splunk internal company com httpagentcertificatepath /certs/ca cert pem httpagentclientcertpath /certs/client cert pem httpagentclientkeypath /certs/client key pem resolve satellite security features config override the configoverride setting in the satellite resolve values yaml file (default false) controls whether the resolve saas platform can remotely modify the satellite's integration configurations through the ui when enabled (set to true), administrators can edit integration settings from the resolve web interface; when disabled, the edit config is not shown in the ui, ensuring that only the local helm values file controls the satellite's configuration this is beneficial for security because it enforces the principle of least privilege and configuration as code practices organizations can prevent unauthorized or accidental changes to critical infrastructure integrations by requiring all configuration changes to go through their standard gitops workflows and approval processes rather than allowing real time ui modifications by defaulting to false in production deployments, the satellite ensures that configuration remains immutable and auditable, with changes only possible through version controlled helm chart updates that can be reviewed, tested, and rolled back if needed satellite config override domain allowlist domainallowlist is a security configuration for http based integrations running on the satellite that restricts outbound api requests to specific domains it accepts an array of domain strings with support for wildcard patterns (e g , example com for subdomains) when configured, the satellite validates all http request urls against the list before execution, throwing an error if the target domain is not allowed if the list is empty or undefined, all domains are permitted by default here is an example for grafana integrations grafanaonprem type grafana create true secretname "grafana resolve secret name" connection url https //yourteam grafana com domainallowlist \ yourteam grafana com \ " yourcompany com" frequently asked questions what are the resource requirements? cpu 1 core memory 8 gb storage 1 gb the satellite is stateless and keeps all working data in memory storage is used primarily for container images and temporary file sources where can i find the satellite helm chart and image? https //hub docker com/r/resolveaihq/satellite chart/tags https //hub docker com/r/resolveaihq/satellite/tags what data sources does the resolve satellite use? the satellite relies on two primary sources, the kubernetes api server and dns tap kubernetes api server collects from pods, services, deployments, jobs, configmaps, events, nodes, pvcs, crds uses watch api with caching and concurrency default interval every 10 minutes (configurable) namespace concurrency 10 threads (contact us to configure) scrapes customresourcedefinitions from kubernetes ingest popular crds like argorollout, istiovirtualservice, istiogateway ingest custom crds (contact us to configure) dns tap open source dns log monitoring captures service to service dns traffic (e g , from coredns) enables automatic runtime dependency discovery non intrusive push model (no polling) does the satellite write to the kubernetes api server? no it is read only and leverages the kubernetes watch cache , minimizing impact on cluster performance how often does the satellite collect data? kubernetes api reads occur every 5 minutes by default this interval is configurable (e g , every 10 or 15 minutes depending on your needs) note you will need to get in touch with your resolve ai contact to make this configuration change namespace level concurrency is also tunable (default 10 concurrent read threads) does it work with our internal proxy? if your proxy supports long lived http2 0 connections the satellite communicates with the resolve server using a bidirectional grpc/http2 0 streaming connection these are long lived connections spanning over 30 minutes if the proxy supports long lived http2 0 streaming connections, then yes, satellites should be able to support it otherwise allowlist the static ip for the resolve server so the satellite can communicate with it what ip addresses should i allowlist? we have a cidr block of ip addresses for the satellite to securely connect to the resolve ai backend add this set of addresses to your allowlist, a /29 block of 8 ips 18 97 138 16/29 kubernetes api reads occur every 5 minutes by default this interval is configurable (e g , every 10 or 15 minutes depending on your needs) note you will need to get in touch with your resolve ai contact to make this configuration change namespace level concurrency is also tunable (default 10 concurrent read threads) how does the satellite communicate with the resolve server? the resolve satellite is a lightweight proxy that establishes a bidirectional grpc/http2 0 streaming connection with the resolve server and executes http requests to local on prem integrations — both observability data sources and kubernetes what is the performance impact? the satellite periodically fetches data from kubernetes and other on prem integrations to update its knowledge graph, but this frequency is configurable and we ensure minimal load on your servers assuming that your typical cluster sizes are namespaces 1000 nodes 400 pods 10,000 here are some numbers that illustrate the load placed on your kubernetes api server polling interval is 10 minutes (configurable) each poll is sharded and fetches about 1 2mib of data we further limit the concurrency of these shards to 10 (configurable) and it roughly approximates to 25 requests per second there should not be any observable load or latency on your system how do i handle pii data? resolve supports docid\ gyvtsegffutdvtrassock to handle pii what are the scaling and performance limits for an individual satellite, and when should we scale it up or out? the satellite architecture supports horizontal scaling through multiple replicas, allowing it to handle very large loads by adjusting the replica count as needed load balancing for the satellites is managed on our backend, ensuring efficient distribution of data and operational control can i configure the satellite to support integrating to multiple kubernetes clusters? yes one satellite instance can connect to multiple kubernetes clusters as long as the kubeconfig is supplied within the secret value first, create a kubernetes secret containing your kubeconfig yaml file the stringdata field in a kubernetes secret enables you to provide secret values as plain text strings, which are then automatically base64 encoded and stored in the data field secret definition example apiversion v1 kind secret type opaque metadata name k8s resolve stringdata kubeconfig yaml | apiversion v1 kind config clusters \ name your cluster cluster server https //your cluster api server 6443 certificate authority data \<base64 encoded ca cert> users \ name your user user token \<your service account token> contexts \ name your context context cluster your cluster user your user current context your context then, apply your secret kubectl apply f k8s resolve yaml lastly, add this into your resolve values yaml file integrations kubernetesintegration type kubernetes create true secretname "k8s resolve" connection contextname "your context" # optional environment "production" how do i configure https certificates for an integration behind a satellite? you can provide the httpagentcertificate for authentication, along with the httpagentclientcert and httpagentclientkey if the endpoint requires mutual tls each can be specified either as a raw certificate or as a file path example using file paths (splunk) splunkintegration secretdir /etc/secrets/splunkintegration create true type splunk connection url \<your splunk url> httpagentcertificatepath \<path to ca cert> httpagentclientcertpath \<path to client cert> httpagentclientkeypath \<path to client key> example using raw certificates (splunk) splunkintegration secretdir /etc/secrets/splunkintegration create true type splunk connection url \<your splunk url> httpagentcertificate \<ca certificate content> httpagentclientcert \<client certificate content> httpagentclientkey \<client key content>