NEWS & INSIGHTS

AI Times column: how AI agents read computer screens

Yongdeok Kim’s AI Times column examines Anthropic’s screen-image processing patent and the distinction between a patent disclosure and Claude’s implementation.

Announcement of Yongdeok Kim’s AI Times column on Anthropic’s screen-understanding patent

A new column by Yongdeok Kim

IPLEX managing patent attorney Yongdeok Kim has contributed an AI patent column to AI Times, a Korean publication specialising in artificial intelligence. The column considers US Patent No. 12,387,036, held by Anthropic, and the visual processing needed to connect a user’s instructions with actions on a computer.

Understanding an instruction is only part of that task. An agent also needs to distinguish text, buttons, menus and input fields, and relate their positions to the action requested. Claude’s computer-use capability illustrates this wider problem of interpreting a software interface.

Turning a screen into a sequence of tokens

The described approach divides images of different resolutions into small patches and arranges them in lines. Newline characters between successive lines preserve information about the image’s two-dimensional organisation when the patches are represented as a token sequence.

Image information and the user’s natural-language instructions are then combined in a single token stream for a transformer to process. Preserving spatial relationships may help the model locate small buttons and fields as well as understand the screen as a whole. The column explains how a visual display can become structured input for a language model.

Distinguishing disclosure from implementation

The patent presents a concrete approach to visual input for agents operating across different software interfaces. It does not establish that every described feature is implemented in Claude’s current computer-use capability. The patent disclosure and the internal design of a commercial model must be considered separately.

The article places this technical question in the wider shift from generating answers to performing tasks in digital environments. The full column is available from AI Times through the link below.

The original figure labels and forms are preserved; the text explains the source material.

Read the full AI Times column

Read the Korean source

This article reflects the information available when it was published. Contact us to discuss your circumstances.
Discuss this topic ↗All articles

Put your IP strategy into practice.