A Bayesian classifier

Zalessky, B.; Lukashevich, P.

doi:10.1090/S0094-9000-09-00759-5

A Bayesian classifier

Authors: B. A. Zalessky and P. V. Lukashevich
Translated by: N. Semenov
Journal: Theor. Probability and Math. Statist. 78 (2009), 23-35
MSC (2000): Primary 62C10; Secondary 90Bxx
DOI: https://doi.org/10.1090/S0094-9000-09-00759-5
Published electronically: August 4, 2009
MathSciNet review: 2446846
Full-text PDF Free Access

Abstract | References | Similar Articles | Additional Information

Abstract:

We consider a new Bayesian classifier for the classification of multidimensional observations $X_1,\dots ,X_n$ of $\mathbb {R}^k$ if the learning sample is known. We assume that the data are generated by two disjoint bounded sets $\Omega _0,\Omega _1\subset \mathbb {R}^k$ and each vector $X_i$ of the sample is a result of the observation after one of the sets $\Omega _\ell$, $\ell =0,1$, with a random error. In other words, we assume that a priori the Bayesian probability $\mu$ is given on the set $\Omega =\Omega _0\cup \Omega _1$ and that every vector of observations $X_i$ has the density \[ g_\ell (x)=q_\ell \int _{\Omega _\ell }f(x,y) \mu (dy),\qquad \ell =0,1, \] where the function $f(x,y)$ is a probability density for all $y\in \Omega$ and $q_\ell ^{-1}=\mu (\Omega _\ell )$.

The maximum a posteriori probability estimators $\widehat {\Omega }_{\ell ,n}$, $\ell =0,1$, for the sets $\Omega _\ell$, $\ell =0,1$, are constructed with the help of the learning sample. Under natural assumptions imposed on $\Omega _0$ and $\Omega _1$, we show that the estimators converge to some sets (possibly different from $\Omega _0$ and $\Omega _1$). If the mean frequencies $\pi _\ell$ of observations of the classes $\Omega _\ell$ are equal to $\mu (\Omega _\ell )$, $\ell =0,1$, then the estimators are consistent in the sense that $\widehat {\Omega }_{\ell ,n} \stackrel {n\rightarrow \infty }{\longrightarrow }\Omega _{\ell }$, $\ell =0,1$. We also discuss some results of numerical experiments showing the applicability of our classifier for solving the problems of the statistical classification.

References

S. A. Aivazyan, B. M. Buchshtaber, I. S. Enyukov, and L. D. Meshalkin, Applied Statistics: Classification and Reducing of Dimension, Finansy i Statistika, Moscow, 1989. (Russian)

A. A. Borovkov, Matematicheskaya statistika, “Nauka”, Moscow, 1984 (Russian). Otsenka parametrov. Proverka gipotez. [Estimation of parameters. Testing of hypotheses]. MR 782295

Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone, Classification and regression trees, Wadsworth Statistics/Probability Series, Wadsworth Advanced Books and Software, Belmont, CA, 1984. MR 726392

L. Breiman, Random Forests, Technical report, Department of Statistics, University of California, Berkeley, CA, 1999.

Vladimir Vapnik, Estimation of dependences based on empirical data, Springer Series in Statistics, Springer-Verlag, New York-Berlin, 1982. Translated from the Russian by Samuel Kotz. MR 672244

Vladimir N. Vapnik, Statistical learning theory, Adaptive and Learning Systems for Signal Processing, Communications, and Control, John Wiley & Sons, Inc., New York, 1998. A Wiley-Interscience Publication. MR 1641250

S. Haykin, Neural Networks: A Comprehensive Foundation, Wiley, New York, 2005.

E. E. Zhuk and Yu. S. Kharin, Stability in the Cluster Analysis of Multivariate Data, Belgosuniversitet, Minsk, 1998. (Russian)

Shelemyahu Zacks, The theory of statistical inference, John Wiley & Sons, Inc., New York-London-Sydney, 1971. Wiley Series in Probability and Mathematical Statistics. MR 0420923

È. Leman, Teoriya tochechnogo otsenivaniya, Teoriya Veroyatnosteĭ i Matematicheskaya Statistika [Probability Theory and Mathematical Statistics], vol. 43, “Nauka”, Moscow, 1991 (Russian). Translated from the English by Yu. V. Prokhorov. MR 1143059

G. Matheron, Random sets and integral geometry, John Wiley & Sons, New York-London-Sydney, 1975. With a foreword by Geoffrey S. Watson; Wiley Series in Probability and Mathematical Statistics. MR 0385969

V. V. Mottl′ and I. B. Muchnik, Skrytye markovskie modeli v strukturnom analize signalov, Fiziko-Matematicheskaya Literatura, Moscow, 1999 (Russian, with Russian summary). MR 1778152

J. Pfanzagl, On the measurability and consistency of minimum contrast estimates, Metrika 14 (1969), 249–273.

D. Forsyth and J. Ponce, Computer Vision. A Modern Approach, Prentice Hall, New York, 2002.

Keinosuke Fukunaga, Introduction to statistical pattern recognition, 2nd ed., Computer Science and Scientific Computing, Academic Press, Inc., Boston, MA, 1990. MR 1075415

M. I. Schlesinger and V. Hlavac, Ten Lectures on Statistical and Structural Pattern Recognition, Springer-Verlag, Berlin, 2002.