Language Models as Stochastic Processes

  1. Jonathan Cyriax Brast 1
  2. Jörg Schäfer 1
  3. Sara Balderas-Díaz 2
  1. 1 Frankfurt University of Applied Sciences, Frankfurt, Germany
  2. 2 Departamento de Ingeniería Informática, University of Cadiz, Av. Universidad de Cádiz, 10, Puerto Real, 11519, Cádiz, Spain
Proceedings:
Tenth Spanish-German Symposium on Applied Computer Science (SGSOACS 2024) (SGSOACS 2024)

Publisher: Department of Computer Science and Engineering, University of Cadiz

ISBN: 978-84-89867-50-5

Year of publication: 2024

Pages: 31-36

Type: Conference paper

GOOGLE SCHOLAR

Abstract

Language models in causal inference mode predict a series ofconditional discrete probability distributions over tokens when appliedon given textual data. The given text can be interpreted as realizationof the distribution over text provided by the language model.We present a mapping of language models and text to stochastic processes using information theory and derive probability bounds for thehypothesis that the language model correctly predicts the informationfor the next token.This relates to anomaly detection, which asks the question if a samplewas generated by another method in contrast to its assumed distribution.We also show early results that suggest AI detection as one possibleapplication, derived from the stochastic process construction.