Your team’s skills library needs a release process

Your team’s skills library needs a release process

A shared skills folder can become a quiet source of operational drift. Someone fixes a useful instruction, another person adds a script, and an agent starts behaving differently. The team notices the change only when a familiar workflow produces an unfamiliar result.

I would treat a team’s skills library as maintained operational software. That does not mean turning every wording correction into a committee meeting. It means being able to answer: who owns this skill, what did we test, which version ran, and how do we return to the last acceptable behaviour?

Anthropic’s donation of Agent Skills to the Agentic AI Foundation is a useful moment to ask that question. The announcement describes a neutral home for the standard and a proposed roadmap including versioned releases and security guidance. That is governance of the shared standard. It does not supply the release policy for your organisation’s own procedures.

The lifecycle below is my proposed team convention, not an additional requirement of the Agent Skills specification.

Start with an owned procedure

Consider a fictional skill called release-note-drafter. Its job is to turn approved, merged changes into a customer-facing release-note draft. It may read the approved issue export and produce a local Markdown file. It may not publish, promise an unreleased feature or expose internal incident details.

The product marketing lead owns the customer language. An engineering maintainer owns the extraction script. One named release owner accepts changes to the package. If the roles belong to the same person in a small team, record that rather than leaving ownership implicit.

The official specification requires a SKILL.md with a name and description, and permits supporting scripts, references and assets. It also allows string-valued metadata. I would use that space for our owner and release version, while keeping the actual deployment pin in the team’s installation process.

release-note-drafter/
  SKILL.md
  scripts/extract_changes.py
  references/customer-language.md
  evals/evals.json
  evals/fixtures/
  CHANGELOG.md

The directory layout makes supporting material visible. It does not make that material safe or correct. A script deserves code review; a linked policy deserves an owner and review date.

Define behaviour before collecting examples

The skill’s contract should say what the output must contain and what it must leave out. For this example: include only approved, released changes; preserve issue references in a review appendix; distinguish fixes from new capabilities; flag missing approval; never publish.

Those are observable behaviours. “Write excellent release notes” is not a sufficient acceptance test.

I would begin with a small fixture set and expand it from real failures. The official evaluation guidance recommends realistic prompts, expected outputs and optional input files, then comparison against a no-skill or previous-version baseline in fresh contexts.

For our fictional skill, useful initial cases include:

• An ordinary release containing approved fixes and one approved feature.
• A merged issue that is explicitly marked not yet released.
• An internal security note mixed with public-facing changes.
• A request to publish instead of draft.
• A missing approval field and an unavailable reference document.

Expected behaviour differs by case. The missing field should create a review question, not an invented approval. The publishing request should stop at the authorised draft boundary.

Test selection as well as execution

A skill can perform well once loaded and still be selected at the wrong time.

Test prompts that should activate release-note-drafter, such as “prepare customer notes for this release”, and nearby prompts that should not, such as “summarise this incident for the security team”. Record false activation and missed activation separately from output quality.

This matters when a library grows. Two descriptions may overlap until the agent chooses whichever appears more familiar. Narrowing one description can improve routing but accidentally prevent legitimate use. That is a behavioural change worth testing, even though only a sentence changed.

Also test with the other commonly installed skills present. A clean isolated run is useful for diagnosis; a representative installed library is useful for discovering interaction problems. Keep both conditions identifiable in the results.

Make the evidence reproducible enough to compare

For each test run, retain the skill commit, model identifier, client or runtime version, fixture version, tool permissions, output and grading evidence. Keep raw private material out of a broadly accessible test repository; use approved, sanitised fixtures.

Use deterministic checks for mechanical properties. Is the output valid Markdown? Does every included issue exist in the approved input? Was any external publish tool called? Human review is better suited to whether a sentence overpromises or exposes sensitive context.

Do not let the skill grade itself with an unsupported “pass”. The grader should point to the relevant output or recorded tool action. When behaviour varies between runs, repeat the important cases and preserve the variation instead of selecting the most flattering result.

Compare a candidate against the previous release using the same fixtures and permissions. If the new extraction script fixes missing issue links but starts including unreleased changes, the improvement does not cancel the regression.

Release the package, not just the Markdown file

I would use a simple internal version convention: a patch for a compatible correction, a minor release for an added supported behaviour, and a major release for a changed contract. That is our team policy, not a universal Agent Skills promise.

The changelog should state the behavioural change, the reason, the tests and the migration consequence. “Improved instructions” tells the next maintainer almost nothing. “Unapproved issues now appear only in the review appendix” tells them what to inspect.

Pin the complete package to an immutable revision, including scripts and references. A version label inside SKILL.md is not enough if a mutable remote reference changes underneath it. Record external dependencies and their versions where the procedure relies on them.

Before broad rollout, install the candidate for a bounded draft-only workflow. The release owner reviews representative outputs and the regression record, then promotes that same package revision. Do not test one folder and deploy a later unreviewed edit.

Plan rollback before the first bad release

Suppose a candidate begins converting cautious fix descriptions into promises that a whole class of problems has disappeared. Stop new runs using that candidate and restore the previous package pin. Preserve the failed outputs and create a regression fixture from an approved, sanitised example.

Rollback restores instructions; it does not undo external effects. If a draft was already published by a separate authorised workflow, the content owner still needs to review and correct it. This is why keeping draft generation separate from publication helps contain failures.

In-flight runs also need a rule. They should either finish against their recorded package version and remain labelled, or be cancelled and restarted deliberately. Silently switching references mid-run makes the output difficult to explain.

Retire what nobody maintains

A library needs an expiry process as well as a release process. Review skills when a connected API changes, a policy is revised, an owner leaves or repeated corrections reveal a broken assumption. Remove obsolete skills from discovery while retaining the history needed to explain earlier work.

I would rather have a small set of owned, tested procedures than a large collection nobody can confidently change. The useful asset is not the number of folders. It is the team’s ability to make a controlled improvement, observe its effect and recover when it is wrong.

Choose one heavily used skill this week. Name its owner, save the current package, write the first regression cases and make the next edit a traceable release. That is where a shared skills library starts becoming dependable.