Local differentiability of distance functions

Authors:
R. A. Poliquin, R. T. Rockafellar and L. Thibault

Journal:
Trans. Amer. Math. Soc. **352** (2000), 5231-5249

MSC (1991):
Primary 49J52, 58C06, 58C20; Secondary 90C30

DOI:
https://doi.org/10.1090/S0002-9947-00-02550-2

Published electronically:
June 9, 2000

MathSciNet review:
1694378

Full-text PDF

Abstract | References | Similar Articles | Additional Information

Abstract: Recently Clarke, Stern and Wolenski characterized, in a Hilbert space, the closed subsets for which the distance function is continuously differentiable everywhere on an open ``tube'' of uniform thickness around . Here a corresponding local theory is developed for the property of being continuously differentiable outside of on some neighborhood of a point . This is shown to be equivalent to the prox-regularity of at , which is a condition on normal vectors that is commonly fulfilled in variational analysis and has the advantage of being verifiable by calculation. Additional characterizations are provided in terms of being locally of class or such that is convex around for some . Prox-regularity of at corresponds further to the normal cone mapping having a hypomonotone truncation around , and leads to a formula for by way of . The local theory also yields new insights on the global level of the Clarke-Stern-Wolenski results, and on a property of sets introduced by Shapiro, as well as on the concept of sets with positive reach considered by Federer in the finite dimensional setting.

**[1]**J. M. Borwein and J. R. Giles,*The proximal normal formula in Banach space*, Trans. Amer. Math. Soc.**302**(1987), 371-381. MR**88m:49013****[2]**F. H. Clarke, R. J Stern and P. R. Wolenski,*Proximal smoothness and the lower-**property*, J. Convex Analysis,**2**(1995), 117-144. MR**96j:49014****[3]**J.-P. Vial,*Strong and weak convexity of sets and functions*, Math. Ops. Res.**8**(1983), 231-259. MR**84m:90107****[4]**A. S. Shapiro,*Existence and differentiability of metric projections in Hilbert spaces*, SIAM J. Optimization**4**(1994), 130-141. MR**94m:90111****[5]**H. Federer,*Curvature measures*, Trans. Amer. Math. Soc.**93**(1959), 418-491. MR**22:961****[6]**R. A. Poliquin and R. T. Rockafellar,*Prox-regular functions in variational analysis*, Trans. Amer. Math. Soc.**348**(1996), 1805-1838. MR**96h:49039****[7]**R. A. Poliquin and R. T. Rockafellar,*Generalized Hessian properties of regularized nonsmooth functions*, SIAM J. Optimization**6**(1996), 1121-1137. MR**97j:49025****[8]**R. A. Poliquin and R. T. Rockafellar,*Second-order nonsmooth analysis in nonlinear programming*, Recent Advances in Nonsmooth Optimization (D. Du, L. Qi and R. Womersley, eds.), World Scientific Publishers, 1995, pp. 322-350. MR**98e:49043****[9]**R. T. Rockafellar and R. J.-B. Wets,*Variational Analysis*, Springer-Verlag, 1997.MR**98m:49001****[10]**C. Combari, A. Elhilali Alaoui, A. Levy, R. A. Poliquin and L. Thibault,*Convex composite functions in Banach spaces and the primal-lower-nice property*, Proc. Amer. Math. Soc.**126**(1998), 3701-3708. MR**99b:49016****[11]**R. A. Poliquin,*Integration of subdifferentials of nonconvex functions*, Nonlinear Anal. Th. Meth. Appl.**17**(1991), 385-398. MR**92j:49008****[12]**R. A. Poliquin,*An extension of Attouch's Theorem and its application to second-order epi-differentiation of convexly composite functions*, Trans. Amer. Math. Soc.**332**(1992), 861-874. MR**93a:49013****[13]**A. Levy, R. A. Poliquin and L. Thibault,*Partial extension of Attouch's theorem with applications to proto-derivatives of subgradient mappings*, Trans. Amer. Math. Soc.**347**(1995), 1269-1294. MR**95k:49035****[14]**L. Thibault and D. Zagrodny,*Integration of subdifferentials of lower semicontinuous functions*, J. Math. Anal. Appl.**189**(1995), 33-58. MR**95i:49032****[15]**R. Correa, A. Jofré and L. Thibault,*Characterization of lower semicontinuous convex functions*, Proc. Amer. Math. Soc.**116**(1992), 61-72. MR**92k:49027****[16]**E. Asplund,*Cebysev sets in Hilbert spaces*, Trans. Amer. Math. Soc.**144**(1969), 235-240. MR**40:6238****[17]**J.-P. Aubin and I. Ekeland,*Applied Nonlinear Analysis*, Wiley-Interscience, 1984. MR**87a:58002****[18]**K.-S. Lau,*Almost Chebyshev subsets in reflexive Banach spaces*, Indiana Univ. Math. J.**27**(1978), 791-795. MR**58:23286****[19]**J. M. Borwein and H. M. Strojwas,*Proximal analysis and boundaries of closed sets in Banach spaces II: Applications*, Canad. J. Math.**39**(1987), 428-472. MR**88f:46034****[20]**F. H. Clarke,*Optimization and Nonsmooth Analysis*, SIAM Publications, Philadelphia, 1990 (originally published in 1983). MR**85m:49002**; MR**91e:49001****[21]**E. Asplund and R. T. Rockafellar,*Gradients of convex functions*, Trans. Amer. Math. Soc.**121**(1968), 31-47. MR**39:1968****[22]**T. S. Motzkin,*Sur quelques propriétés caractéristiques des ensembles convexes*, Att. R. Acad. Lincei, Rend.**21**(1935), 562-567.**[23]**V. Klee,*Convexity of Chebyshev sets*, Math. Annalen**142**(1961), 292-304. MR**22:12367****[24]**J.-B. Hiriart-Urruty,*Ensembles de Tchebychev vs. ensembles convexes: l'état de la situation vu par l'analyse convexe non lisse*, Ann. Sci. Math. Que.**22**(1998), 47-62. MR**99f:49018****[25]**R. R. Phelps,*Convex Functions, Monotone Operators and Differentiability*, Lecture Notes in Mathematics, Vol. 1364 2nd edition, Springer-Verlag, 1993. MR**94f:46055**

Retrieve articles in *Transactions of the American Mathematical Society*
with MSC (1991):
49J52,
58C06,
58C20,
90C30

Retrieve articles in all journals with MSC (1991): 49J52, 58C06, 58C20, 90C30

Additional Information

**R. A. Poliquin**

Affiliation:
Department of Mathematical Sciences, University of Alberta, Edmonton, Alberta, Canada T6G 2G1

Email:
rene.poliquin@ualberta.ca

**R. T. Rockafellar**

Affiliation:
Department of Mathematics 354350, University of Washington, Seattle, Washington 98195-4350

Email:
rtr@math.washington.edu

**L. Thibault**

Affiliation:
Laboratoire d’Analyse Convexe, Université Montpellier II, 34095 Montpellier, France

Email:
thibault@math.univ-montp2.fr

DOI:
https://doi.org/10.1090/S0002-9947-00-02550-2

Keywords:
Variational analysis,
distance functions,
single-valued projections,
proximal normals,
prox-regularity,
proximal smoothness,
primal-lower-nice functions,
hypomonotone mappings,
monotone mappings

Received by editor(s):
June 17, 1997

Received by editor(s) in revised form:
June 10, 1998

Published electronically:
June 9, 2000

Additional Notes:
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada under grant OGP41983 for the first author, by the National Science Foundation under grant DMS–9500957 for the second author, and by NATO under grant CRG 960360 for the third author. The authors are grateful for useful discussions with C. Combari, and for helpful comments made by the referee.

Article copyright:
© Copyright 2000
American Mathematical Society