Satellite on ECS

deploy the resolve satellite on aws ecs fargate using cloudformation the satellite runs in your aws environment and securely connects to resolve saas to provide alert investigation, monitoring, and integration capabilities for kubernetes based deployments, see docid\ zx9llo6ncp3yfe1h9htcp architecture compute ecs fargate (serverless containers) networking runs in your vpc with configurable subnets and security groups storage optional efs volume for git integration persistence secrets aws secrets manager for credentials logging cloudwatch logs installation step 1 prepare resources gather the following information before deployment see docid\ ogskcpcoy1aealjcbkidz below for how to create each resource parameter description example ingesttokensecretarn arn of ingest token secret arn\ aws\ secretsmanager\ us east 1 123456789\ secret\ resolve/satellite/ingest token abcdef executionrolearn arn of ecs task execution role arn\ aws\ iam 123456789\ role/resolvesatelliterole taskrolearn arn of ecs task role (can be the same as execution role) arn\ aws\ iam 123456789\ role/resolvesatelliterole ecsclusterarn arn of ecs cluster arn\ aws\ ecs\ us east 1 123456789\ cluster/my cluster securitygroupid security group id sg 0a1b2c3d subnetids comma separated subnet ids subnet abc123,subnet def456 vpcid vpc id (only if enablegitvolume=true ) vpc 12345678 satellitename name for this satellite my satellite at least one integration must be configured for the satellite to appear in the resolve ui step 2 deploy with cloudformation deploy directly from the aws cloudformation console by clicking https //console aws amazon com/cloudformation/home#/stacks/create?stackname=resolve satellite\&templateurl=https //resolve satellite cloudformation s3 us east 2 amazonaws com/releases/latest/satellite yaml fill in the required parameters in the form click launch stack above (opens aws cloudformation console) choose your preferred region in the top right corner of the console fill in the required parameters from step 1 configure the integrationsconfig parameter with your integration yaml (see docid\ ogskcpcoy1aealjcbkidz for examples) click next through the remaining pages, then submit the launch stack button uses the latest template version to deploy a specific version, replace latest in the template url with a version number (e g , v1 1 22 ) for automated or ci/cd deployments, use the aws cli download the cloudformation template curl o satellite yaml https //resolve satellite cloudformation s3 us east 2 amazonaws com/releases/latest/satellite yaml to pin a specific version, replace latest with a version number https //resolve satellite cloudformation s3 us east 2 amazonaws com/releases/v1 1 22/satellite yaml deploy the stack git integration (github app) aws cloudformation deploy \\ \ template file satellite yaml \\ \ stack name resolve satellite \\ \ region us east 1 \\ \ parameter overrides \\ ingesttokensecretarn=arn\ aws\ secretsmanager\ us east 1 123456789\ secret\ resolve/satellite/ingest token abcdef \\ executionrolearn=arn\ aws\ iam 123456789\ role/resolvesatelliterole \\ taskrolearn=arn\ aws\ iam 123456789\ role/resolvesatelliterole \\ ecsclusterarn=arn\ aws\ ecs\ us east 1 123456789\ cluster/resolve satellite cluster \\ securitygroupid=sg 0a1b2c3d \\ subnetids=subnet abc123,subnet def456 \\ satellitename=my satellite \\ enablegitvolume=true \\ vpcid=vpc 12345678 \\ integrationsconfig="git type git create true connection authconfigs myrepos type github" git integration (token auth) aws cloudformation deploy \\ \ template file satellite yaml \\ \ stack name resolve satellite \\ \ region us east 1 \\ \ parameter overrides \\ ingesttokensecretarn=arn\ aws\ secretsmanager\ us east 1 123456789\ secret\ resolve/satellite/ingest token abcdef \\ executionrolearn=arn\ aws\ iam 123456789\ role/resolvesatelliterole \\ taskrolearn=arn\ aws\ iam 123456789\ role/resolvesatelliterole \\ ecsclusterarn=arn\ aws\ ecs\ us east 1 123456789\ cluster/resolve satellite cluster \\ securitygroupid=sg 0a1b2c3d \\ subnetids=subnet abc123,subnet def456 \\ satellitename=my satellite \\ enablegitvolume=true \\ vpcid=vpc 12345678 \\ integrationsconfig="git type git create true secretname arn\ aws\ secretsmanager\ us east 1 123456789\ secret\ resolve/satellite/git credentials xyz123 connection authconfigs myrepos type token tokenauthremoteurls \ https //github com/your org/repo 1 git \ https //github com/your org/repo 2 git" multiple integrations aws cloudformation deploy \\ \ template file satellite yaml \\ \ stack name resolve satellite \\ \ region us east 1 \\ \ parameter overrides \\ ingesttokensecretarn=arn\ aws\ secretsmanager\ us east 1 123456789\ secret\ resolve/satellite/ingest token abcdef \\ executionrolearn=arn\ aws\ iam 123456789\ role/resolvesatelliterole \\ taskrolearn=arn\ aws\ iam 123456789\ role/resolvesatelliterole \\ ecsclusterarn=arn\ aws\ ecs\ us east 1 123456789\ cluster/resolve satellite cluster \\ securitygroupid=sg 0a1b2c3d \\ subnetids=subnet abc123,subnet def456 \\ satellitename=my satellite \\ enablegitvolume=true \\ vpcid=vpc 12345678 \\ integrationsconfig="datadog type datadog create true secretname arn\ aws\ secretsmanager\ us east 1 123456789\ secret\ resolve/satellite/datadog credentials abc123 connection site datadoghq com git type git create true connection authconfigs myrepos type github" github app auth does not require a git credentials secret after deployment, open your git integration in the resolve ui and click install github app to complete authorization resolve satellite supports additional integrations including grafana, prometheus, loki, elasticsearch, splunk, kubernetes, aws cloudwatch, and more for the full list of integration types and their configuration schemas, see docid\ fc7ntg ks5fehrlid7t46 step 3 monitor deployment the deployment typically takes 3 5 minutes \# check stack status aws cloudformation describe stacks \\ \ stack name resolve satellite \\ \ query 'stacks\[0] stackstatus' \# get service name from stack outputs service name=$(aws cloudformation describe stacks \\ \ stack name resolve satellite \\ \ query 'stacks\[0] outputs\[?outputkey==`servicename`] outputvalue' \\ \ output text) \# check service status aws ecs describe services \\ \ cluster resolve satellite cluster \\ \ services $service name resource setup reference the cloudformation template does not create the following resources they must exist before deployment 1\ iam role the satellite needs an iam role with the ecs tasks amazonaws com trust policy you can use a single role for both executionrolearn and taskrolearn by combining all required permissions into one role trust policy { "version" "2012 10 17", "statement" \[ { "effect" "allow", "principal" { "service" "ecs tasks amazonaws com" }, "action" "sts\ assumerole" } ] } required managed policies arn\ aws\ iam aws\ policy/service role/amazonecstaskexecutionrolepolicy required inline policies secrets manager access { "version" "2012 10 17", "statement" \[ { "effect" "allow", "action" \["secretsmanager\ getsecretvalue"], "resource" \["arn\ aws\ secretsmanager\ region\ account id\ secret\ resolve/satellite/ "] } ] } efs access (only if using git integration) { "version" "2012 10 17", "statement" \[ { "effect" "allow", "action" \["elasticfilesystem\ clientmount", "elasticfilesystem\ clientwrite"], "resource" "arn\ aws\ elasticfilesystem\ region\ account id\ file system/ " } ] } if your organization requires separate permission boundaries, you can create two roles — one for task execution (image pull, secrets, logs) and one for task runtime (efs access) — and pass them as different executionrolearn and taskrolearn values aws cli commands to create the role # create the role aws iam create role \\ \ role name resolvesatelliterole \\ \ assume role policy document '{ "version" "2012 10 17", "statement" \[{ "effect" "allow", "principal" {"service" "ecs tasks amazonaws com"}, "action" "sts\ assumerole" }] }' \# attach managed policy aws iam attach role policy \\ \ role name resolvesatelliterole \\ \ policy arn arn\ aws\ iam aws\ policy/service role/amazonecstaskexecutionrolepolicy \# add secrets manager permissions aws iam put role policy \\ \ role name resolvesatelliterole \\ \ policy name secretsmanageraccess \\ \ policy document '{ "version" "2012 10 17", "statement" \[{ "effect" "allow", "action" \["secretsmanager\ getsecretvalue"], "resource" \["arn\ aws\ secretsmanager secret\ resolve/satellite/ "] }] }' \# add efs permissions (only if using git integration) aws iam put role policy \\ \ role name resolvesatelliterole \\ \ policy name efsaccess \\ \ policy document '{ "version" "2012 10 17", "statement" \[{ "effect" "allow", "action" \[ "elasticfilesystem\ clientmount", "elasticfilesystem\ clientwrite" ], "resource" "arn\ aws\ elasticfilesystem file system/ " }] }' 2\ aws secrets manager secrets a ingest token (required) the ingest token authenticates your satellite with resolve saas generate it from resolve ui > admin > tokens secret format { "token" "your ingest token here" } aws secretsmanager create secret \\ \ name resolve/satellite/ingest token \\ \ description "resolve satellite ingest token" \\ \ secret string '{"token" "your token from resolve ui"}' note the arn after creation (format arn\ aws\ secretsmanager\ region\ account id\ secret\ resolve/satellite/ingest token xxxxx ) you will need it for deployment b git integration credentials (token auth only) required only if using git integration with token based auth github app auth does not require a secret — it is configured through the resolve ui after deployment creating a personal access token github (fine grained token, recommended) go to settings > developer settings > personal access tokens > fine grained tokens choose the resource owner and select the repositories you want to grant access to under repository permissions, set contents to read only (or read and write if you need write/remediation) if your org uses sso, authorize the token for sso after creation github (classic token) go to settings > developer settings > personal access tokens > tokens (classic) select scope repo for private repositories or public repo for public only if your org uses sso, authorize the token for sso after creation classic pats with repo scope are broad — prefer fine grained pats when possible gitlab go to preferences > access tokens add scope read repository (or additional write scopes if required) { "tokenauthcredentials" { "myrepos" { "username" "your username", "token" "your personal access token" } } } aws secretsmanager create secret \\ \ name resolve/satellite/git credentials \\ \ description "git integration credentials" \\ \ secret string file //git credentials json c datadog integration credentials (optional) required only if using datadog integration { "apikey" "your datadog api key", "appkey" "your datadog app key" } aws secretsmanager create secret \\ \ name resolve/satellite/datadog credentials \\ \ description "datadog integration credentials" \\ \ secret string '{"apikey" "your api key","appkey" "your app key"}' 3\ network infrastructure vpc requirements dns resolution must be enabled ( enablednssupport true ) dns hostnames must be enabled ( enablednshostnames true ) internet access required for communication with resolve saas \# verify vpc dns settings aws ec2 describe vpc attribute vpc id vpc xxxxx attribute enablednssupport aws ec2 describe vpc attribute vpc id vpc xxxxx attribute enablednshostnames subnets minimum 2 subnets (recommended for high availability) type private subnets with nat gateway, or public subnets with auto assign public ip internet access must have route to internet (via nat gateway or internet gateway) security group create a security group for the satellite tasks required egress rules https (port 443) to resolve saas https (port 443) to integration endpoints (datadog, grafana, etc ) if using git with efs nfs (port 2049) to efs security group aws ec2 create security group \\ \ group name resolve satellite sg \\ \ description "security group for resolve satellite" \\ \ vpc id vpc xxxxx 4\ ecs cluster you must have an existing ecs cluster the satellite will be deployed as a fargate service aws ecs describe clusters clusters your cluster name configuration reference required parameters parameter description ingesttokensecretarn arn of secrets manager secret containing ingest token executionrolearn arn of ecs task execution role taskrolearn arn of ecs task role (can be the same as execution role) ecsclusterarn arn of ecs cluster securitygroupid security group id for satellite tasks subnetids comma separated list of subnet ids satellitename name for this satellite instance optional parameters parameter default description image resolveaihq/satellite container image repository imagetag latest container image tag loglevel info log level ( debug , info , warn , error ) cpu 1024 cpu units (1024 = 1 vcpu) memory 8192 memory in mb desiredcount 1 number of tasks to run assignpublicip disabled assign public ip ( enabled for public subnets without nat) enablegitvolume false enable efs volume for git integration integrationsconfig "" yaml configuration for integrations git volume (efs) when enablegitvolume=true , the cloudformation template automatically creates efs filesystem encrypted, generalpurpose performance mode efs security group allows nfs (port 2049) traffic from satellite tasks efs access point isolated path /git with posix uid/gid 1000 efs mount targets one per subnet (takes 2 3 minutes to become available) additional parameters parameter required default description vpcid yes vpc id for efs security group efsperformancemode no generalpurpose efs performance mode efsthroughputmode no bursting efs throughput mode cost considerations efs standard storage $0 30/gb/month (pay only for actual usage) no provisioned capacity needed bursting throughput included at no extra cost integrations configuration the integrationsconfig parameter accepts yaml configuration for integrations all credentials must be stored in aws secrets manager git integration git integration requires efs volume for repository storage ( enablegitvolume=true ) no credentials secret needed auth is configured through the resolve ui after deployment git type git create true connection authconfigs myrepos type github requires a credentials secret in aws secrets manager (see docid\ ogskcpcoy1aealjcbkidz ) git type git create true secretname arn\ aws\ secretsmanager\ us east 1 123456789\ secret\ resolve/satellite/git credentials connection authconfigs myrepos type token tokenauthremoteurls \ "https //github com/your org/repo 1 git" \ "https //github com/your org/repo 2 git" for detailed git integration configuration including auth options, write/remediation behavior, and token setup, see docid\ wzkfdunpbtwyyj4fbzl4p datadog integration datadog type datadog create true secretname arn\ aws\ secretsmanager\ us east 1 123456789\ secret\ resolve/satellite/datadog credentials connection site datadoghq com supported sites datadoghq com , datadoghq eu , us3 datadoghq com , us5 datadoghq com , ddog gov com additional integrations resolve satellite supports many additional integrations including grafana, prometheus, loki, elasticsearch, splunk, kubernetes, aws cloudwatch, and more for complete integration schemas and configuration examples, see docid\ fc7ntg ks5fehrlid7t46 verification 1\ check task is running aws ecs list tasks \\ \ cluster resolve satellite cluster \\ \ service name resolve satellite my satellite \\ \ desired status running \# get task details aws ecs describe tasks \\ \ cluster resolve satellite cluster \\ \ tasks task arn task should be in running state with laststatus running 2\ check cloudwatch logs log group=$(aws cloudformation describe stacks \\ \ stack name resolve satellite \\ \ query 'stacks\[0] outputs\[?outputkey==`loggroupname`] outputvalue' \\ \ output text) aws logs tail $log group follow expected log messages satellite starting connected to resolve saas integration \[name] initialized health check passed 3\ check satellite in resolve ui log in to resolve ui navigate to admin > satellites verify your satellite appears with status "connected" 4\ test health check task arn=$(aws ecs list tasks \\ \ cluster resolve satellite cluster \\ \ service name resolve satellite my satellite \\ \ query 'taskarns\[0]' \\ \ output text) task ip=$(aws ecs describe tasks \\ \ cluster resolve satellite cluster \\ \ tasks $task arn \\ \ query 'tasks\[0] attachments\[0] details\[?name==`privateipv4address`] value' \\ \ output text) \# from a host in the same vpc curl http //$task ip 13131/live expected http 200 ok 5\ verify efs mount (if using git integration) \# enable ecs exec (one time setup) aws ecs update service \\ \ cluster resolve satellite cluster \\ \ service resolve satellite my satellite \\ \ enable execute command \# exec into container aws ecs execute command \\ \ cluster resolve satellite cluster \\ \ task task arn \\ \ container satellite \\ \ interactive \\ \ command "/bin/sh" \# inside container, check mount df h | grep /integrationdata/git ls la /integrationdata/git updating the satellite update to new version aws cloudformation deploy \\ \ template file satellite yaml \\ \ stack name resolve satellite \\ \ parameter overrides \\ imagetag=v1 2 3 \\ \# other parameters remain the same cloudformation performs a rolling update with zero downtime (if desiredcount > 1) add git integration to existing deployment aws cloudformation deploy \\ \ template file satellite yaml \\ \ stack name resolve satellite \\ \ parameter overrides \\ enablegitvolume=true \\ vpcid=vpc xxxxx \\ integrationsconfig=" " \\ \# other parameters remain the same this creates new efs resources and restarts the task deleting the satellite delete cloudformation stack aws cloudformation delete stack stack name resolve satellite \# wait for deletion to complete aws cloudformation wait stack delete complete \\ \ stack name resolve satellite the efs filesystem has deletionpolicy retain to prevent accidental data loss after deleting the stack, delete the efs filesystem manually if no longer needed aws efs delete file system file system id fs xxxxx cleanup iam roles and secrets iam roles and secrets manager secrets are not managed by the cloudformation stack delete them manually if no longer needed \# delete secrets aws secretsmanager delete secret \\ \ secret id resolve/satellite/ingest token \\ \ force delete without recovery \# delete role (detach policies first) aws iam delete role role name resolvesatelliterole troubleshooting task fails to start task transitions from pending to stopped immediately cause resolution invalid secrets arn verify secret exists aws secretsmanager describe secret secret id secret arn insufficient iam permissions verify role has secrets manager access image pull failure verify image is accessible docker pull resolveaihq/satellite\ latest no internet access verify subnet route table has nat gateway or internet gateway efs mount fails task logs show failed to mount efs or health check never passes cause resolution mount targets not ready wait 2 3 minutes after stack creation check status aws efs describe mount targets file system id fs xxxxx security group blocking nfs check efs security group allows port 2049 from satellite security group role missing efs permissions verify role has elasticfilesystem\ clientmount and clientwrite vpc dns not enabled verify vpc has dns resolution and hostnames enabled satellite not appearing in resolve ui task is running but satellite doesn't show as connected cause resolution invalid ingest token verify token value in secrets manager matches resolve ui network blocked verify security group allows outbound https to resolve saas domain high memory usage task getting killed due to oom (out of memory) increase memory parameter (default is 8192 mb) review cloudwatch container insights metrics integration not working integration appears in ui but queries fail verify secret format matches the expected structure for each integration type verify security group allows outbound https to integration endpoints for token auth, verify the secret key names match authconfigs key names check cloudwatch logs linked from the ecs task for detailed error messages