# ActiveLLM: asking a language model which 32 examples to label

How ActiveLLM separates example selection from classifier training, and what its few-shot results, exceptions and evaluation limits actually show.

Source: https://www.iplexlaw.co.kr/en/blog/activellm-few-shot-example-selection

HOME / NEWS & INSIGHTS NEWS & INSIGHTS ActiveLLM: asking a language model which 32 examples to label How ActiveLLM separates example selection from classifier training, and what its few-shot results, exceptions and evaluation limits actually show. AI 2026.11.12 published IPLEX 13 min read When only a few examples can be labelled, the choice of examples can matter as much as their number. ActiveLLM assigns that choice to a large language model, then trains a separate classifier on the selected, labelled data. Its contribution concerns the organisation of learning, not a claim that an LLM must perform every subsequent prediction. Cold starts and a separate selection model Active learning directs annotation towards examples expected to help training. Conventional uncertainty-based strategies can struggle initially because the classifier has too little labelled data to estimate uncertainty reliably; repeated training during selection can also delay annotation. ActiveLLM uses an instruction-following LLM as a query model, without requiring the downstream classifier’s parameters or probability estimates. Figure 1: separate LLM selection, human annotation and BERT training — Bayer, Lutz & Reuter (2026), CC BY 4.0. The paper by Markus Bayer, Justin Lutz and Christian Reuter of PEASEC, Technical University of Darmstadt, appeared in Transactions of the Association for Computational Linguistics, volume 14, pages 1–22, in January 2026 (DOI 10.1162/TACL.a.63). Here, few-shot means training a downstream model with a small labelled set, rather than simply placing examples in an inference prompt. From candidate texts to classifier training The LLM receives a bounded candidate list, instructions describing the selection task, the required number of examples and an output format. It returns example identifiers; the chosen texts are labelled and then used to train a BERT-like classifier, which is evaluated separately. The LLM is not required at every later classification call. A list of 200 candidates and a selection of 32 examples represent different budgets. Major experiments present 100 or 200 candidates, rather than an entire dataset of thousands of texts. Existing dataset labels simulate the human annotator in the experiments. The findings therefore test selection quality, not superior automatic labelling or the elimination of human annotation errors. Prompt design: more explanation is not always better On CTI, selecting 32 from 300 candidates gave F1 scores of 0.6781 for random selection, 0.7214 for B2 with a simple step-by-step instruction, and 0.6271 for B3, which asked for an explanation for every example. C2, including annotation guidelines, reached 0.7329. The comparisons vary prompt components; they do not test one prompt containing every possible instruction. Figure 2: prompt specifying role, candidates, selection size and output format — Bayer, Lutz & Reuter (2026), CC BY 4.0. Because suitable annotation guidelines are often unavailable for public benchmarks, the broader evaluation uses B2 without extra selection advice or guidelines. Clear selection size and output requirements still matter. Generated explanations should not be treated as a faithful record of the model’s internal reasoning, and a longer answer is not evidence of a better selection. Experimental budgets and metrics Prompt development uses CTI, while general classification and sentence-pair evaluation use GLUE; CTI and AG News are evaluated on test sets, GLUE on validation sets. Prompt comparisons average 25 runs, combining five data orders with five model initialisations. Major subsequent results average five initialisations with one data order. Table 3 reports BERT-base accuracy after training on 32 examples, including accuracy for CoLA; it must not be confused with CTI F1 or conventional CoLA reporting elsewhere. With 50, 100, 200, 300 and 400 CTI candidates, the reported F1 values are 0.7120, 0.7438, 0.7695, 0.7214 and 0.6632 respectively, with 32 examples selected each time. Two hundred is best in that experiment, not a universal optimum. Conventional active-learning baselines repeatedly train and can access the full pool, whereas ActiveLLM selects a batch from a subset. This compares complete configurations, not one isolated algorithmic change under identical conditions. Large improvements, alongside clear exceptions For SST-2, random selection yields 56.03% accuracy and ActiveGPT4 with 200 candidates yields 73.14%: a difference of 17.11 percentage points, not a 17.11% relative gain. For WNLI, Mistral Large reaches 52.96% versus 35.77%, a 17.19-point difference. Yet GPT-4o on RTE scores 48.88% against the random baseline of 50.83%; Gemini Ultra on SST-2 scores 49.08% against 56.03%; GPT-3.5 on QNLI matches the baseline at 55.11%. Table 3: GLUE accuracy of BERT-base trained on 32 selected examples — Bayer, Lutz & Reuter (2026), CC BY 4.0. Figure 5 compares standalone selection, established few-shot methods and their combinations. On CTI, ActiveGPT4, ADAPET and their combination score 76.95, 59.31 and 75.95 on an F1 scale of 100. On SST-2, ActiveGPT4, PERFECT and the combination achieve 73.14%, 48.85% and 53.10% accuracy. The standalone differences are 17.64 F1 points and 24.29 percentage points. On AG News, SetFit rises from 79.57% to 81.09% when combined, while standalone ActiveGPT4 is 67.33%. ADAPET and SetFit use BERT-base; PERFECT uses RoBERTa-Large. The mixed tasks, metrics and architectures do not form a single ranking. Figure 5: standalone selection, few-shot methods and combinations by task — Bayer, Lutz & Reuter (2026), CC BY 4.0. Selection time and the remaining costs Table 4 gives conventional strategies’ time, in minutes, to select 32 examples over six active-learning iterations. BALD on QQP produced no result after 48 hours; its missing entry is not a zero accuracy score. By omitting repeated classifier training during selection and inspecting only a subset, ActiveLLM is reported to achieve query times of seconds. The paper does not provide a fully controlled, side-by-side accounting of every LLM’s end-to-end latency and API charges. A precise speedup or cost-saving multiplier cannot be inferred from that table. Deployment planning must include request charges, input length, data transfer conditions and the later classifier workload. Scaling beyond the smallest datasets In experiments selecting 25 examples per round up to 300, the advantage is most apparent below roughly 100 examples; later performance approaches random selection. Repeating the identifiers of previously selected examples retains history without reproducing all their text. Using an initial set of 50 ActiveGPT4 examples improves LC and PE, but the same benefit is not reproduced for EKM and BALD. Figure 7: LC, PE, EKM and BALD with 50 initial ActiveGPT4 examples — Bayer, Lutz & Reuter (2026), CC BY 4.0. The evidence mainly concerns BERT-family classifiers, not all generative-model training. The prompt was developed chiefly for GPT-4, rather than independently optimised for each LLM. Benchmark contamination cannot be excluded. Long candidates, hidden service instructions, model versions and settings affect reproducibility. Sending source texts to the selection LLM still raises confidentiality questions even when the eventual classifier runs separately. Reading the architecture from a patent perspective The technical problem is how to choose valuable training examples before a reliable labelled set exists. The proposed information flow connects an instruction-following LLM, a candidate batch, selection constraints, returned identifiers, annotation and downstream training. Separating the selector from the trained classifier is central; candidate size, identifier formatting and selection-history reminders are additional design choices. Observed effects depend on the combination of model, prompt, dataset and annotation budget. They support considering large models at the data-selection stage and show gains in specific few-shot settings, while preserving the reported failures and diminishing gains at larger budgets. This architectural analysis does not establish patent novelty or patentability. The SST-2 gain is a useful example of the contribution, not a universal performance promise. The original figure labels and forms are preserved; the text explains the source material. Paper reviewed: ActiveLLM (TACL 2026) Explore our AI and patent practice Read the Korean source This article reflects the information available when it was published. Contact us to discuss your circumstances. Discuss this topic ↗ All articles ON THIS PAGE Cold starts and a separate selection model From candidate texts to classifier training Prompt design: more explanation is not always better Experimental budgets and metrics Large improvements, alongside clear exceptions Selection time and the remaining costs Scaling beyond the smallest datasets Reading the architecture from a patent perspective TALK TO IPLEX Discuss your IP questions We consider your technology and business needs together. ↗ Contact us Newer Employee invention compensation: when does the limitation period begin? ↗ Older Lost in the Middle: why the position of an answer-bearing document matters ↗ Related insights Artificial Intelligence 2026.11.19 Matryoshka embeddings for image retrieval: shortlist with a prefix, rerank with the full vector A practical reading of Matryoshka Representation Learning, its nested training objective and the ImageNet-1K retrieval results reported at NeurIPS 2022. ↗ Read article Artificial Intelligence 2026.11.18 ByteDance’s captioning patent: transcribing separate tracks and resolving overlapping display times How US11837234B2 processes selected audio tracks and adjusts caption timing, with a practical distinction between patent disclosure and CapCut’s public guidance. ↗ Read article Artificial Intelligence 2026.11.17 Apple recognized text: connect a phone number or web address to the right action US12001642B2 links recognized text, image coordinates, text types and user input to contextual menus and actions. ↗ Read article

- https://www.iplexlaw.co.kr/en
- https://www.iplexlaw.co.kr/en/blog/category/ai
- https://aclanthology.org/2026.tacl-1.1/
- https://www.iplexlaw.co.kr/en/ai-patents
- https://www.iplexlaw.co.kr/ko/blog/activellm-few-shot-example-selection
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog
- https://www.iplexlaw.co.kr/en/blog/activellm-few-shot-example-selection#article-1954-en-0
- https://www.iplexlaw.co.kr/en/blog/activellm-few-shot-example-selection#article-1954-en-1
- https://www.iplexlaw.co.kr/en/blog/activellm-few-shot-example-selection#article-1954-en-2
- https://www.iplexlaw.co.kr/en/blog/activellm-few-shot-example-selection#article-1954-en-3
- https://www.iplexlaw.co.kr/en/blog/activellm-few-shot-example-selection#article-1954-en-4
- https://www.iplexlaw.co.kr/en/blog/activellm-few-shot-example-selection#article-1954-en-5
- https://www.iplexlaw.co.kr/en/blog/activellm-few-shot-example-selection#article-1954-en-6
- https://www.iplexlaw.co.kr/en/blog/activellm-few-shot-example-selection#article-1954-en-7
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog/brunch-case-1955
- https://www.iplexlaw.co.kr/en/blog/lost-in-the-middle-evidence-position
- https://www.iplexlaw.co.kr/en/blog/matryoshka-prefix-vector-image-retrieval
- https://www.iplexlaw.co.kr/en/blog/matryoshka-prefix-vector-image-retrieval
- https://www.iplexlaw.co.kr/en/blog/bytedance-track-based-caption-timing
- https://www.iplexlaw.co.kr/en/blog/bytedance-track-based-caption-timing
- https://www.iplexlaw.co.kr/en/blog/apple-recognized-text-contextual-actions
- https://www.iplexlaw.co.kr/en/blog/apple-recognized-text-contextual-actions
