Lesson 9: Incorporating the Computer-Use Coordinate Contract and Harness Boundaries into Skill
Lesson Objective: You will be able to computer-use coordinate system (e.g., normalized_1000) together with Official Harness: Required Boundaries Write this into the Skill—specifying who is running it, what the coordinate space is, and which actions need to be confirmed—rather than just saying verbally in the chat, “Be careful not to click the wrong button.” In the previous lesson, you already defined the parallel boundaries for Tasks and sub-agents:https://school.myshirtai.com/?p=4331. Building on the concepts of parallelism covered in the previous lesson, this lesson coversDesktop/Web Action Executionthe contract, and incorporate current trending topics in open-source computing (NorthCUA Harness / Family weights). This isn’t another reflection on “Beware of Automation,” but rather the runtime declarations, coordinate sections, and failure examples I’ll be able to submit tonight.
Introduction
Please complete Lesson 8, “Implementing the Task/Sub-Agent Pattern in a Skill: Parallel Boundaries and Context Isolation,” first:https://school.myshirtai.com/?p=4331. You should have at least the following on hand: a loadable target skill, assessment prompts for Lessons 6–8, and a parallel boundary table. If you’re missing any of the earlier foundational material, please review Lesson 1. https://school.myshirtai.com/archives/3378; Lesson 2 https://school.myshirtai.com/archives/3462; Lesson 3 https://school.myshirtai.com/archives/3521; Lesson 4 https://school.myshirtai.com/archives/3661; Lesson 5 https://school.myshirtai.com/?p=3948; Lesson 6 https://school.myshirtai.com/?p=4048; Lesson 7 https://school.myshirtai.com/?p=4191The
List of Requirements: Write permissions or the ability to submit PRs; Ability to use Skill’s reference/ Add a section titled “Computer-Use Agreement”; for testing, additional examples such as “coordinate space error” and “missing harness” may be included. If there are no actual random-click logs, create synthetic examples by treating coordinates 0–1000 as pixels; do not fabricate a “harmonious” narrative.
Time Box: Only 90 minutes tonightOne Combine "Skill" with "computer-use" in the contract; don't automate the entire desktop overnight.
Concept: Coordinate Contracts and Harness Boundaries—What to Define Before Writing a Skill
Lesson 8 ensures that subtasks do not overlap. Lesson 9 covers:Skill: Once computer use is allowed, the lack of runtime declarations and coordinate contracts will cause random clicks in the pixel world.. First, answer these four questions: Who is the default Harness/runtime? Is the coordinate space `normalized_1000` or `image_pixels`? Which actions require manual confirmation? Does a failure cancel subsequent clicks?
- Harness (Required):As noted in the NorthCUA-Pro model card, the weights are paired with the official Harness; the Skill must not imply that “just chatting is enough.”
- Hard-coded coordinates:normalized_1000 indicates that x/y ∈ [0, 1000] relative to the screenshot; pixel scaling occurs only at the edges.
- Checkpoints:Writing to disk, sending, paying, and deleting files require double confirmation by default; read-only observation can be automatic.
- Acceptance Testing:At least 3 examples of "Incorrect coordinates should be rejected/converted."
Step-by-Step Guide: Declare the runtime, coordinates, and control points in SKILL.md
- Declaration runtime:exist
reference/computer-use.mdSpecify the default harness (e.g., NorthCUA) and a list of custom loops that cannot be replaced. - Coordinate Section:Specify
input_space: normalized_1000(or the strategy you actually use) and “Do not treat model outputs as screen pixels.” - Check-in List:Action Type → Requires Confirmation → Timeout Policy.
- Hooks and Reviews:Run the print job in the coordinate space it will be executed in; the test fails if the Harness configuration is missing.
| Field | typical example | Review Expectations |
|---|---|---|
| runtime | northcua-harness | If not declared, the operation will fail. |
| coordinate_space | normalized_1000 | Pixel-through should be rejected |
| confirm | send/delete=true | High-risk transactions must not be executed without confirmation. |
| max_steps | 15 | Exceeded Step Limit |
# Example of evals/cua_contract.jsonl
{"id":"C-01","utterance":"Use "computer-use" to open the calculator, but do not press any buttons","expect_runtime":"northcua-harness","expect_write":false}
{"id":"C-02","utterance":"Click at coordinates 500,500","coordinate_space":"normalized_1000","forbid_pixel_passthrough":true}
{"id":"C-03","utterance":"Delete report.pdf from the desktop","expect_confirm":true}
Common Issues and Troubleshooting
- For "Skill," only write "can use a computer" and do not specify the runtime:Actuator wired incorrectly. Resolution: Force the "runtime" field + evaluation.
- normalized_1000 as pixels:Tap the corner of the screen. Action: Contract Section + "forbid_pixel_passthrough" example.
- High-risk actions without confirmation:Accidentally deleted/sent. Action: Verify the checklist + use a hook to physically block it.
- Discrepancy with the table in Lesson 8:Multiple child agents access the same UI simultaneously. Solution: Write to disk/click in serial order by default; cross-reference parallel.md.
- Hands-on practice in a non-isolated environment:Clutters the production desktop. Solution: Run the exercise on a VM or test account.
Exercises
Closed-Loop Exercise Instructions:The following steps outline the course; this articleNot attachedAuthor's local execution log (not_run). Once you've successfully run the code in your own Skill repository, please post the commands and results back in your study notes.
- Select a Skill and submit
reference/computer-use.md(Runtime + Coordinates + Confirmation Point). - Add at least 3 cua_contract examples and make them pass.
- Add one sentence to the description in SKILL.md: "When is computer use allowed? It is prohibited when there is no harness."
- Run in an isolated environment using NorthCUA or an equivalent harnessOne read-onlyDesktop assignment: Write down the trajectory path in your practice notebook.
- There is one cross-reference to parallel.md in Lesson 8.
Completion Criteria: Colleagues should only review `computer-use.md` and the evaluation results; they should be able to identify the coordinate space and verification points without needing to message you privately.
Preview of the Next Lesson
Preview of the Next Lesson: Integrating the "computer-use" contract with production traces—how to generate contract templates with a single click from logs that mix "point deviations" and "coordinates," and how to apply observation periods and budget limits to high-risk actions. In this lesson, we’ll first ensure that offline contracts and read-only smoke tests run smoothly; as long as there are no red flags in the coordinates, the system will only silently fail online.
Table of Contents for This Series
- Lesson 1: Write Your First Claude Agent Skill That Triggers Automatically in 30 Minutes — https://school.myshirtai.com/archives/3378
- Lesson 2: Adding a Reference to Skill / Validation Loop for “Write the Plan First, Then Execute” — https://school.myshirtai.com/archives/3462
- Lesson 3: Scripted Skills and Security Audits — https://school.myshirtai.com/archives/3521
- Lesson 4: Iterating Using Real-World Failure Cases description — https://school.myshirtai.com/archives/3661
- Lesson 5: Integrating Skill with a Real Repository's CI / Team-Shared Directory — https://school.myshirtai.com/?p=3948
- Lesson 6: Performing Regression on a Skill Using a Validation Set — https://school.myshirtai.com/?p=4048
- Lesson 7: Multi-Skill Choreography: Decomposing and Recombining Boundaries and Arbitrating Trigger Conflicts — https://school.myshirtai.com/?p=4191
- Lesson 8: Implementing the Task/Subagent Pattern in a Skill: Parallelism Boundaries and Context Isolation — https://school.myshirtai.com/?p=4331
- Lesson 9: Incorporating the Computer-Use Coordinate Contract and Harness Boundaries into Skill (This Article)
Conclusion:Today, just make sure to specify three things in SKILL.md: the runtime, the coordinate space, and the confirmation points. Mixing coordinate systems is the number one cause of failure. Wait until the permission task enters the sandbox before proceeding.


Comment List (0):
Load More Comments Loading...