# Google’s document-understanding patent: reading the layout of headings, text and images together

US12393768B2 uses block-level and document-level encoding to represent content and layout. A hypothetical inspection manual illustrates the distinction from OCR and the structure retained by Document AI.

Source: https://www.iplexlaw.co.kr/en/blog/google-layout-aware-document-understanding

HOME / NEWS & INSIGHTS NEWS & INSIGHTS Google’s document-understanding patent: reading the layout of headings, text and images together US12393768B2 uses block-level and document-level encoding to represent content and layout. A hypothetical inspection manual illustrates the distinction from OCR and the structure retained by Document AI. Artificial Intelligence 2026.11.10 published IPLEX 7 min read Conceptual illustration of reading visual information, text and formatting in relation to document blocks. The original Korean labels are preserved. Flattening an equipment manual into a single text stream can obscure which heading governs a procedure and which photograph a caption describes. Google’s US patent US12393768B2 describes dividing a document into blocks, representing each block’s content and layout, and then processing relationships between neighboring blocks to represent the document as a whole. A hypothetical inspection manual helps explain these two stages and why document retrieval benefits from retaining links between headings, illustrations and body text. Why correctly recognized words can still lose their context Suppose a technician finds the instruction “Check after closing the cover.” To use it, the technician also needs to know which equipment it concerns and whether the illustration on the right belongs to that procedure. On a page containing two sections, the positions of headings and explanations provide valuable clues. Optical character recognition, or OCR, detects text in an image of a document and converts it into machine-readable characters. If only a flat list of recognized words is retained, relationships between elements that were close together on the original page may disappear. Recognizing the words and organizing them into the right document context are therefore complementary tasks. Google’s patent considers layout alongside text and images: where an element sits and how it is presented on the page. Bold type may indicate a section heading; a paragraph below it may continue that section; a caption may relate to a nearby figure. The approach supplies the model with information that can help it account for these relationships. Separate headings, paragraphs and figures into blocks The document is first divided into blocks. A block is a region with a role, such as a heading, paragraph, image or table. The disclosure describes using information about its position, font, color and formatting. For web documents, HTML tags identifying headings or paragraphs can also help determine the boundaries. Consider two lines at the same height in a two-column document. Reading the left line and automatically continuing with the right line can mix two separate sections. Identifying paragraph regions and retaining their positions gives the next model information for deciding which elements should be considered together. At this stage, the input connects content to layout. A block might combine the words “Routine inspection,” their large bold type and their position near the top of the page. An image block carries visual features, while the explanation beneath it can be represented as a separate block or associated content. Encode individual blocks, then their document-level relationships The first model converts each block into a numerical representation that can be processed computationally. This is encoding. A text block can contribute its wording and formatting, while an image block can contribute visual features. The patent also provides the layout data associated with each block as input to this model. The second model processes those block representations together with local connections between neighboring blocks. The first stage captures what a block contains; the next incorporates how it relates to other parts of the document. This produces a representation of the whole document, including its headings, explanations and images. That document representation can become the input to downstream tasks. The disclosure discusses finding related documents and classifying document types. It also describes suggesting a suitable image from a set of candidates for inclusion in the document. Across these uses, the central distinction is between representing a block individually and representing its relationships within the document. Pretraining for reading order and image–text relationships Training tasks can encourage the model to learn how headings and explanations fit together. For example, two blocks can be swapped and the model asked whether their order is appropriate. The patent also describes masking some text or an entire block and predicting it from the surrounding information. Other tasks concern images and text. An image can be replaced with an image from another training document, and the model trained to detect that replacement. Another task hides an image and selects a suitable replacement from candidates. These tasks make the relationship between a picture and its surrounding explanation relevant to learning, alongside the picture’s own features. This is pretraining that exposes the model to clues about document structure. When a new manual is processed in use, the trained model represents its blocks and their relationships. Subsequent processing depends on whether the representation is needed for retrieval, classification or document organization. Connect a heading and caption in a two-column inspection manual Imagine a manual with a large “Routine inspection” heading at the top, a procedure on the left and an equipment illustration with a short caption on the right. Separating the regions makes it possible to examine the heading, procedure paragraph, illustration and caption as distinct elements. Hypothetical two-column inspection manual. Dotted arrows show the heading–procedure and illustration–caption relationships. At block level, the heading contributes a short phrase in large type, the procedure contributes several lines of instructions, and the illustration contributes the equipment’s appearance. At document level, the heading–procedure relationship and the illustration–caption relationship both matter. Text at the same height in the two columns can still be understood as belonging to different regions. For a retrieval application, a result containing “Check the cover” could be accompanied by its section heading and the relevant illustration caption. This is a possible use of the structural information: it helps a reader understand which procedure the sentence belongs to, rather than encountering the sentence in isolation. What structure does Document AI return? Google Cloud documents several distinct processing capabilities. Enterprise Document OCR extracts text and layout information, including characters, lines, words and paragraphs. Such output can be used, for example, to make scanned manuals searchable. Document AI’s Layout Parser documentation describes identifying elements such as headings, tables, lists and figures in PDFs and returning a structured document hierarchy. When preparing smaller units for retrieval, the system can include their higher-level headings. These units, with their surrounding context retained, are commonly called chunks. A team building a search service over equipment manuals can use chunks that retain the relationship between section headings and paragraphs. The patent explains a way to represent blocks and relationships numerically; the product documentation describes how structured output and heading context can support practical document workflows. These are separate sources of information. Evaluate the relationships as well as the recognized text Start an evaluation by defining the intended task. If the application needs a procedure to remain associated with its heading, check both character recognition and assignment to the correct section. If it displays illustrations, check whether each caption is associated with the corresponding figure. Choose concrete comparison documents: manuals repeating the same sentence under different headings, pages with two columns, and pages where a caption is separated from its photograph. Recording the layout information retained at input, the intermediate blocks and the evidence returned in a search result can help locate errors. A technical description should similarly distinguish text recognition, block creation, relationship processing and downstream use. Explaining which information is introduced at each stage and how it reaches the next computation makes the differences between document-processing methods more concrete. The original figure labels and forms are preserved; the text explains the source material. The inspection manual is an explanatory example. The technical overview is not an exhaustive account of claim scope: claim 1 also includes image suggestion and training based on detecting replacement by images from another document; claims 14 and 19 concern training-system and storage-medium configurations. Product documentation does not establish that Document AI implements every element of the patent. USPTO patent text · US12393768B2 Google Cloud’s official Layout Parser documentation Google Cloud’s official Enterprise Document OCR documentation Naver’s image-translation patent: keeping translated text aligned with a moving camera Baidu’s RAG approach to conflicts between model answers and retrieved knowledge Explore IPLEX’s AI practice and consultation materials to describe document-processing mechanisms and their technical distinctions for patent work. Explore our AI and patent practice Original Tistory article Read the Korean source This article reflects the information available when it was published. Contact us to discuss your circumstances. Discuss this topic ↗ All articles ON THIS PAGE Why correctly recognized words can still lose their context Separate headings, paragraphs and figures into blocks Encode individual blocks, then their document-level relationships Pretraining for reading order and image–text relationships Connect a heading and caption in a two-column inspection manual What structure does Document AI return? Evaluate the relationships as well as the recognized text TALK TO IPLEX Discuss your IP questions We consider your technology and business needs together. ↗ Contact us Newer Filing a trademark in Norway: routes, examination and maintenance ↗ Older AI Times column: how AI agents read computer screens ↗ Related insights Artificial Intelligence 2026.11.19 Matryoshka embeddings for image retrieval: shortlist with a prefix, rerank with the full vector A practical reading of Matryoshka Representation Learning, its nested training objective and the ImageNet-1K retrieval results reported at NeurIPS 2022. ↗ Read article Artificial Intelligence 2026.11.18 ByteDance’s captioning patent: transcribing separate tracks and resolving overlapping display times How US11837234B2 processes selected audio tracks and adjusts caption timing, with a practical distinction between patent disclosure and CapCut’s public guidance. ↗ Read article Artificial Intelligence 2026.11.17 Apple recognized text: connect a phone number or web address to the right action US12001642B2 links recognized text, image coordinates, text types and user input to contextual menus and actions. ↗ Read article

- https://www.iplexlaw.co.kr/en
- https://www.iplexlaw.co.kr/en/blog/category/ai
- https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/12393768
- https://docs.cloud.google.com/document-ai/docs/layout-parse-chunk
- https://docs.cloud.google.com/document-ai/docs/enterprise-document-ocr
- https://www.iplexlaw.co.kr/en/blog/naver-papago-ar-image-translation
- https://www.iplexlaw.co.kr/en/blog/baidu-rag-knowledge-conflict-resolution
- https://www.iplexlaw.co.kr/en/ai-patents
- https://iplexlaw.tistory.com/21
- https://www.iplexlaw.co.kr/ko/blog/google-layout-aware-document-understanding
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog
- https://www.iplexlaw.co.kr/en/blog/google-layout-aware-document-understanding#article-21-en-0
- https://www.iplexlaw.co.kr/en/blog/google-layout-aware-document-understanding#article-21-en-1
- https://www.iplexlaw.co.kr/en/blog/google-layout-aware-document-understanding#article-21-en-2
- https://www.iplexlaw.co.kr/en/blog/google-layout-aware-document-understanding#article-21-en-3
- https://www.iplexlaw.co.kr/en/blog/google-layout-aware-document-understanding#article-21-en-4
- https://www.iplexlaw.co.kr/en/blog/google-layout-aware-document-understanding#article-21-en-5
- https://www.iplexlaw.co.kr/en/blog/google-layout-aware-document-understanding#article-21-en-6
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog/norway-trademark-filing-guide
- https://www.iplexlaw.co.kr/en/blog/aitimes-anthropic-screen-understanding
- https://www.iplexlaw.co.kr/en/blog/matryoshka-prefix-vector-image-retrieval
- https://www.iplexlaw.co.kr/en/blog/matryoshka-prefix-vector-image-retrieval
- https://www.iplexlaw.co.kr/en/blog/bytedance-track-based-caption-timing
- https://www.iplexlaw.co.kr/en/blog/bytedance-track-based-caption-timing
- https://www.iplexlaw.co.kr/en/blog/apple-recognized-text-contextual-actions
- https://www.iplexlaw.co.kr/en/blog/apple-recognized-text-contextual-actions
