1. Translation arrives after the scene has moved
Imagine pointing a phone at an exhibition sign in English. By the time its Korean translation is ready, a slight movement of the hand may have shifted or tilted the sign. A correct translation placed at the old coordinates is difficult to read in context. Naver’s US patent US12197883B2 addresses this timing and positioning problem through an augmented-reality image-translation pipeline. Papago’s public product descriptions provide a useful example of the user experience, while the patent explains a technical approach; those materials do not by themselves establish how every current product feature is implemented.
2. Selecting the input from buffered frames
A video consists of successive frames. The disclosed process temporarily stores several recent frames in a buffer and selects one that meets predetermined criteria. Brightness, sharpness, the presence of sufficient text or edges and the extent of image processing may inform that choice. The newest frame is therefore not necessarily the most suitable input. A brief camera shake may obscure one frame while nearby frames remain readable. Buffer size, selection criteria and the timing of the next processing cycle are design decisions alongside the translation model itself.
3. Recognising text and defining the translation region
Optical character recognition converts text represented by image pixels into machine-readable characters. Translating those characters into another language is a separate step. The disclosure describes neural-network-based recognition and the use of an external image-translation device. The system also identifies the region in which translated text will appear, using features associated with the text and its surroundings. On a sign containing both a heading and a timetable, retaining that spatial relationship prevents the two translations from becoming confused. Figure 9 sets out the sequence: store frames, select a suitable frame, translate, determine the translation region and render it in another frame.
4. Mapping the result onto the current view
A sign that was centred in the reference frame may move sideways or appear as a slanted quadrilateral in a later view. Distinctive keypoints, such as corners and surrounding visual patterns, help establish correspondences between the two frames. A feature-transformation model then maps the translation region to its new position. One disclosed example uses a homography, a projective transformation useful for relating views of a plane. Rendering composites the translated region onto the later frame. Separating the translated content from its display geometry allows processing to continue while the camera view changes.
5. Matching or tracking according to elapsed time
The patent distinguishes frames separated by a longer interval from those captured closer together. For the longer interval, feature matching can extract keypoints and descriptors in both frames, compare them and estimate a transformation. For the shorter interval, feature tracking can follow the movement of established points using optical flow. These are different ways to update the display region, rather than different translation models. In the fictional sign example, “TODAY’S PROGRAM” is translated into Korean and moved to the sign’s new position after the camera shifts. The illustration simplifies the reference points for explanation; it is not a measured product test.
6. Refreshing a result that is no longer current
A previously prepared translation should not remain indefinitely after the camera moves to another sign. Figure 10 illustrates a time-based loop: after rendering, the processor checks whether a predetermined interval has elapsed. If not, the region is maintained for subsequent display; if it has, the process returns to storing and selecting a new frame and performing translation again. Other disclosed examples may withhold a result when the interval or motion difference is excessive and start a new cycle. This rule assesses whether a result is still appropriate for the current view. It does not assess the linguistic accuracy of the translation itself.
7. Implications for AI product and patent analysis
Naver’s Papago materials describe image and real-time AR translation, and Naver Cloud’s Image Translation documentation describes recognising image text and returning text or an image containing the translated text. For developers, the relevant questions extend to input selection, changes in the view during processing and the conditions for recomputing the display region. Tests can distinguish a stationary sign, slow camera movement, an oblique view and a change of target, recording the source frame, display frame and refresh decision. From a patent perspective, the technical contribution must be considered as the claimed combination of processing steps, not merely the broad idea of translating a sign. The original figures and the fictional example help explain that combination without treating the example as a performance benchmark.
Read the original Tistory article
Explore our AI patent practice