Abstract
The problem of optimally approximating a function with a linear expansion over a redundant dictionary of waveforms is NP-hard. The greedy matching pursuit algorithm and its orthogonalized variant produce suboptimal function expansions by iteratively choosing dictionary waveforms that best match the function’s structures. A matching pursuit provides a means of quickly computing compact, adaptive function approximations.
Numerical experiments show that the approximation errors from matching pursuits initially decrease rapidly, but the asymptotic decay rate of the errors is slow. We explain this behavior by showing that matching pursuits are chaotic, ergodic maps. The statistical properties of the approximation errors of a pursuit can be obtained from the invariant measure of the pursuit. We characterize these measures using group symmetries of dictionaries and by constructing a stochastic differential equation model.
We derive a notion of the coherence of a signal with respect to a dictionary from our characterization of the approximation errors of a pursuit. The dictionary elements slected during the initial iterations of a pursuit correspond to a function’s coherent structures. The tail of the expansion, on the other hand, corresponds to a noise which is characterized by the invariant measure of the pursuit map.
When using a suitable dictionary, the expansion of a function into its coherent structures yields a compact approximation. We demonstrate a denoising algorithm based on coherent function expansions.
Similar content being viewed by others
References
S. Chen, S. A. Billings, W. Luo (1989):Orthogonal least squares methods and their application to non-linear system identification. Internat. J. Control,50:1873–1896.
L. Cohen (1989):Time-frequency distributions: A review. Proc. IEEE,77:941–979.
P. Collet, J. P. Eckmann (1980): Iterated Maps on the Interval as Dynamical Systems. Boston: Birkhauser.
T. H. Cormen, C. E. Leiserson, R. L. Rivest (1991): Introduction to Algorithms. New York: McGraw-Hill.
I. Daubechies (1991): Ten Lectures on Wavelets. CBMS-NSF Series in Appl. Math. Philadelphia: SIAM.
G. Davis (1994): Adaptive Nonlinear Approximations. PhD dissertation, Department of Mathematics, New York University.
G. Davis, S. Mallat, Z. Zhang (1994):Adaptive time-frequency decompositions. Optical Engrg.,33:2183–2191.
R. L. Devaney (1989): An Introduction to Chaotic Dynamical Systems. New York: Addison-Wesley.
R. A. DeVore, B. Jawerth, B. J. Lucier (1992):Image compression through wavelet transform coding. IEEE Trans. Inform. Theory,38:719–746.
R. A. DeVore, B. Jawerth, V. Popov (1992):Compression of wavelet decompositions. Amer. J. Math.,114:737–785.
D. L. Donoho (1993):Wavelet shrinkage and W.V.D.: A 10-minute tour. In: Progress in Wavelet Analysis and Applications (Y. Meyer, S. Roques, eds.). Gif-sur-Yvette, France: Editions Frontieres, pp. 109–128.
C. W. Gardiner (1985): Handbook of Stochastic Methods. New York: Springer-Verlag.
M. R. Garey, D. S. Johnson (1979): Computers and Intractability: A Guide to the Theory of NP-Completeness. New York: W. H. Freeman.
P. R. Halmos (1956): Lectures on Ergodic Theory. Tokyo: Mathematical Society of Japan.
L. K. Jones (1987):On a conjecture of Huber concerning the convergence of projection pursuit regression. Ann. Statist.,15:880–882.
A. Lasota, M. Mackey (1985): Probabilistic Properties of Deterministic Systems. New York: Cambridge University Press.
S. Mallat, Z. Zhang (1993):Matching pursuit with time-frequency dictionaries. IEEE Trans. Signal Processing,41:3397–3415.
Y. C. Pati, R. Rezaiifar, P. S. Krishnaprasad (1993):Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. Proceedings of the 27th Annual Asilomar Conference on Signals, Systems, and Computers, pp. 40–44. IEEE Computer Society Press.
S. Qian, D. Chen (1994):Signal representation via adaptive normalized gaussian functions. IEEE Trans. Signal Processing,36:1–11.
M. Reed, B. Simon (1972): Methods of Modern Mathematical Statistics, Vol. 1. New York: Academic Press.
N. Saito (1994):Simultaneous noise suppression and signal compression using a library of orthonormal bases and the minimum description length criterion. Wavelets in Geophysics (Efi Foufoula-Georgiou, Praveen Kumar, eds.). Academic Press, San Diego.
L. Blum, M. Shub, S. Smale (1989):On a theory of computation and complexity over the real numbers: NP-completeness, recursive functions, and universal machines. Bull. Amer. Math. Soc.,21:1–46.
P. Walters (1975): Ergodic theory—Introductory Lectures. New York: Springer-Verlag.
Z. Zhang (1993): Matching Pursuit. PhD dissertation, New York University.
Author information
Authors and Affiliations
Additional information
Communicated by Albert Cohen.
Rights and permissions
About this article
Cite this article
Davis, G., Mallat, S. & Avellaneda, M. Adaptive greedy approximations. Constr. Approx 13, 57–98 (1997). https://doi.org/10.1007/BF02678430
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF02678430