Knowledge Setup Guide

resolve uses a two tier knowledge model — org level knowledge applies company wide, while team level knowledge is team specific when both exist, team knowledge takes priority see teams & knowledge docid\ m4f0bsnc1so f7zedqxrw for more detail resolve pulls knowledge differently depending on context alert investigations resolve examines all knowledge associated with the alert and its team(s) one runbook per alert rule or monitor makes it easier for resolve to surface the most relevant guidance chats resolve pulls in skills and relevant knowledge sections concise documents with clearly marked headings help resolve match the right knowledge to your question faster in both cases clearly stating when a piece of knowledge applies (e g , "use this when investigating latency spikes in the checkout service") helps resolve pull in the right guidance at the right time getting started before you start make sure your observability integrations are connected resolve queries these tools during investigations and chats, and the knowledge you author is what tells resolve how to use them see connect an integration docid\ ig yfnpcmge2yt3oq5i4b start with the highest impact content and refine over time here's the recommended order create your team and add a brief description so resolve understands what this team owns teams work best when they mirror your on call rotations, so the right knowledge is accessible to the right responders each team member has one of two roles team admin full access to edit team content, configuration, and manage members team member can view team content, configuration, and members, but cannot make changes write your team's resolve md with a system overview, glossary, and key query guidance for your observability tools add dashboard guidance for your most used dashboards resolve auto generates a starting point; you can review and refine it attach alert runbooks for your highest volume or most critical alerts these can imported from github or confluence or written directly in resolve include general documents for specialized knowledge that applies to specific scenarios like runbooks, these can be imported from external sources rather than written from scratch add skills for the procedural parts of your operations; such as postmortem templates, oncall handoff reports start from a resolve provided template or import existing skills from your github integration in resolve iterate as you use resolve fill in context gaps as they come up knowledge types resolve runs a team of specialized agents during an investigation, and each one reads the knowledge most relevant to its role dashboard guidance lands with the agent reading dashboards alert runbooks anchor the investigation lead's first move resolve md and team docs inform the whole team's reasoning skills are invoked dynamically when the task calls for them authoring the right type of knowledge is what makes it land in front of the right agent at the right moment that's where the biggest investigation quality gains come from there are 5 types of team knowledge, which are all markdown https //commonmark org/help/ files 1\ resolve md team specific agent instructions used in all chats and alert investigations tips for writing resolve md use markdown formatting with clear section headers include a brief system overview and a glossary of your team's terminology create separate sections for different observability types (e g , "metrics guidance" vs "logs guidance") and custom tools, if applicable include context about when the guidance applies (e g , "when troubleshooting production issues, use 'app3 prod cluster' as the environment") keep it concise, about 10,000 characters this content is referenced frequently, so overly long files may reduce effectiveness resolve may have drafted some content for you when you connect a new integration, resolve can draft proposed updates to your resolve md based on what it discovered, such as label conventions, common query patterns, and other context it can infer from the source these land as pending suggestions in your resolve md editor; review them and accept or reject most useful today for logs integrations example resolve md clusters we have the following clusters app3 cluster, dev2 cluster, stgl cluster app3 cluster is the production cluster if the user mentions a question around particular cluster(s) then pass that information to all tools and agents for eg if a user is asking to look in logs for app3, then you should always mention the cluster information during explorations logs guidance always apply cluster filter the logs are fetched from a grafana cloud instance that has logs for many other organizations for this organization, you must always apply one of the following filters cluster="app3 cluster" or cluster="dev2 cluster" if investigating a kubernetes pod, you should also use the kubernetes log integration for most services, you can get the logs for the right level using a query like {cluster="app3 cluster", namespace="checkout assistant", service name=" \<service name> "} | detected level=" \<level> " levels can be info, warn, debug, error the labels like cluster, namespace, service name are indexed and make the query efficient as much as possible, avoid just a keyword based search without label filters as that scans a lot of logs 2\ alert runbooks alert specific guidance always examined during mapped alert investigations when a runbook is linked to the specific alert that fired, resolve uses its full content rather than pulling excerpts by relevance this makes alert linked runbooks the highest leverage place to tune behavior for a known alert pattern tips for writing alert runbooks write sequential, actionable steps resolve can follow in order add examples of exact or templated queries with clear placeholders (e g , "run service name=\<service name> environment\ prod error — replace \<service name> with the actual service") when referencing dashboards, add corresponding dashboard guidance for those dashboards include guidance on examples for queries or custom tool use with unique syntax (e g , "if the country is germany, the log index should be index=index ge ") either use titles that match the alert name or link the runbook to the alert rule on the alerts page example alert runbooks use traces to find the appropriate rds instances and related services use 'rds overview (us east 2)' dashboard to get the health of the overall rds instances use 'rds performance insights (us east 2)' dashboard to triage further use the 'pgstats' command via the awscli tool to determine if there are specific queries resulting in high cpu from step 1, use the related services to determine the radius of impact 3\ dashboard guidance instructions for interpreting each dashboard, including variable selection, filters, and chart sections resolve will auto generate guidance for the dashboards you add review and edit the markdown file to improve how resolve uses your dashboard tips for writing dashboard guidance be specific about how to set filters and variables (e g , " service name should be set to \<servicename> prod ") call out specific charts or sections that are most useful for particular issue types example dashboard guidance when to use this dashboard troubleshoot any service issues including latency, performance etc dashboard variables replace \<servicename> and \<country code> according to the service and country for all variables set to \<servicename> \<country code> except aws ecs service append " backend" service should be set to \<servicename> \<country code> host service should be set to \<servicename> \<country code> aws ecs service should be set to \<servicename> \<country code> backend if the service name and country code are both provided, do not append an additional country code example if the service is "checkout" and country is germany, use the following values service should be set to checkout ge host service should be set to checkout ge aws ecs service should be set to checkout ge backend 4\ docs guidance for specific issue scenarios, which is pulled in based on relevance tips for writing effective docs clearly state when this guidance applies so resolve can retrieve it at the right moment provide rich context for services and systems (e g , "the recommendation service is rarely mission critical; failures here are almost always rooted in other services") include guidance on which attributes or fields to use for filtering and grouping in your specific observability tools describe how and when to use custom tools (e g , "when investigating payment processing issues, use the paymentdb tool to check transaction status ") example docs general guidance for the frontend service you can often use the @http url @http path group and @http target attributes to group by or filter for certain spans/traces investigation guidance the frontend is the entrypoint of our app (sitting behind frontend proxy which receives requests from users) therefore lots of errors might bubble up from our other microservices and trigger frontend alerts just because there is an error or alert triggered on the frontend service does not mean it is the root cause of the problem in fact its likely that it is not the root cause error logs in the frontend might reference other services and we can use traces to determine if any dependencies are the source of our errors evidence queries guidance you can often use the @http url @http path group and @http target attributes to group by or filter for certain spans/traces the service makes http and grpc requests to downstream services so you might see common error codes from those protocols in any error logs 5\ skills procedural workflows that resolve invokes dynamically based on the task description, or that you can invoke directly via slash commands tips for writing skills the description is the only thing resolve sees during discovery make it specific about when to use the skill (alert names, services, keywords), not just what it does keep each skill focused on one coherent task skills compose well; over stuffed skills don't be prescriptive for fragile operations (exact flags, exact sequence); describe goals for flexible ones skills can be invoked directly by typing / in resolve the skill's name is the slash command name example skills name service owner description look up which team owns a service and how to reach them use when the user types /service owner or asks who owns or who to page for a service \ for the named service 1\ find the owning team in the service catalog 2\ return the team name, primary on call channel, and pagerduty escalation policy if the service isn't in the catalog, say so — don't guess checkout skills docid\ nlilpthw611fjeawk8o0k for the full format reference and more examples importing from external sources import alert runbooks, docs, and skills from github or confluence instead of duplicating them in resolve resolve fetches the latest version at investigation time, so your guidance always stays in sync with the source of truth to set this up, go to external docs on your team knowledge page external sources are configured per team , so each team can wire up its own set of repositories or spaces independently how the external sources sync works an initial sync runs automatically as soon as you add a source, so your existing knowledge appears in resolve right away after that, resolve refreshes on a recurring schedule you can also trigger a manual sync from the external docs page at any time sync status and the time of the last successful refresh are visible in the ui for each source failed syncs surface there too if "last synced" stops advancing, that's the first place to look syncs are reconciling , not append only if a file is moved out of the watched directory or deleted from the source, its artifact is removed from resolve as well, so stale knowledge doesn't linger edits flow one way external sourced documents are owned by the source system any edits you make in resolve to an imported document will be overwritten on the next sync to change an imported document, edit it in github or confluence and let the sync pick it up tips for setting up external sources scope filters to a specific directory or page tree , not a whole repo or space a broad filter creates lots of low relevance documents and dilutes retrieval for github, files named skill md are automatically imported as skills (following the agent skills https //agentskills io/home format); other markdown files become docs or runbooks this lets a single repo serve both kinds of content prerequisite the source system needs to be connected as an integration first if you don't see your github or confluence instance in the picker, connect it via connect an integration docid\ ig yfnpcmge2yt3oq5i4b before adding a source