Local differentiability of distance functions

Authors:
R. A. Poliquin, R. T. Rockafellar and L. Thibault

Journal:
Trans. Amer. Math. Soc. **352** (2000), 5231-5249

MSC (1991):
Primary 49J52, 58C06, 58C20; Secondary 90C30

Published electronically:
June 9, 2000

MathSciNet review:
1694378

Full-text PDF Free Access

Abstract | References | Similar Articles | Additional Information

Abstract: Recently Clarke, Stern and Wolenski characterized, in a Hilbert space, the closed subsets for which the distance function is continuously differentiable everywhere on an open ``tube'' of uniform thickness around . Here a corresponding local theory is developed for the property of being continuously differentiable outside of on some neighborhood of a point . This is shown to be equivalent to the prox-regularity of at , which is a condition on normal vectors that is commonly fulfilled in variational analysis and has the advantage of being verifiable by calculation. Additional characterizations are provided in terms of being locally of class or such that is convex around for some . Prox-regularity of at corresponds further to the normal cone mapping having a hypomonotone truncation around , and leads to a formula for by way of . The local theory also yields new insights on the global level of the Clarke-Stern-Wolenski results, and on a property of sets introduced by Shapiro, as well as on the concept of sets with positive reach considered by Federer in the finite dimensional setting.

**[1]**J. M. Borwein and J. R. Giles,*The proximal normal formula in Banach space*, Trans. Amer. Math. Soc.**302**(1987), no. 1, 371–381 (English, with French summary). MR**887515**, 10.1090/S0002-9947-1987-0887515-5**[2]**F. H. Clarke, R. J. Stern, and P. R. Wolenski,*Proximal smoothness and the lower-𝐶² property*, J. Convex Anal.**2**(1995), no. 1-2, 117–144. MR**1363364****[3]**Jean-Philippe Vial,*Strong and weak convexity of sets and functions*, Math. Oper. Res.**8**(1983), no. 2, 231–259. MR**707055**, 10.1287/moor.8.2.231**[4]**Alexander Shapiro,*Existence and differentiability of metric projections in Hilbert spaces*, SIAM J. Optim.**4**(1994), no. 1, 130–141. MR**1260410**, 10.1137/0804006**[5]**Herbert Federer,*Curvature measures*, Trans. Amer. Math. Soc.**93**(1959), 418–491. MR**0110078**, 10.1090/S0002-9947-1959-0110078-1**[6]**R. A. Poliquin and R. T. Rockafellar,*Prox-regular functions in variational analysis*, Trans. Amer. Math. Soc.**348**(1996), no. 5, 1805–1838. MR**1333397**, 10.1090/S0002-9947-96-01544-9**[7]**R. A. Poliquin and R. T. Rockafellar,*Generalized Hessian properties of regularized nonsmooth functions*, SIAM J. Optim.**6**(1996), no. 4, 1121–1137. MR**1416532**, 10.1137/S1052623494279316**[8]**René Poliquin and Terry Rockafellar,*Second-order nonsmooth analysis in nonlinear programming*, Recent advances in nonsmooth optimization, World Sci. Publ., River Edge, NJ, 1995, pp. 322–349. MR**1460008****[9]**R. Tyrrell Rockafellar and Roger J.-B. Wets,*Variational analysis*, Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 317, Springer-Verlag, Berlin, 1998. MR**1491362****[10]**C. Combari, A. Elhilali Alaoui, A. Levy, R. Poliquin, and L. Thibault,*Convex composite functions in Banach spaces and the primal lower-nice property*, Proc. Amer. Math. Soc.**126**(1998), no. 12, 3701–3708. MR**1451793**, 10.1090/S0002-9939-98-04324-X**[11]**René A. Poliquin,*Integration of subdifferentials of nonconvex functions*, Nonlinear Anal.**17**(1991), no. 4, 385–398. MR**1123210**, 10.1016/0362-546X(91)90078-F**[12]**René A. Poliquin,*An extension of Attouch’s theorem and its application to second-order epi-differentiation of convexly composite functions*, Trans. Amer. Math. Soc.**332**(1992), no. 2, 861–874. MR**1145732**, 10.1090/S0002-9947-1992-1145732-5**[13]**A. B. Levy, R. Poliquin, and L. Thibault,*Partial extensions of Attouch’s theorem with applications to proto-derivatives of subgradient mappings*, Trans. Amer. Math. Soc.**347**(1995), no. 4, 1269–1294. MR**1290725**, 10.1090/S0002-9947-1995-1290725-3**[14]**Lionel Thibault and Dariusz Zagrodny,*Integration of subdifferentials of lower semicontinuous functions on Banach spaces*, J. Math. Anal. Appl.**189**(1995), no. 1, 33–58. MR**1312029**, 10.1006/jmaa.1995.1003**[15]**R. Correa, A. Jofré, and L. Thibault,*Characterization of lower semicontinuous convex functions*, Proc. Amer. Math. Soc.**116**(1992), no. 1, 67–72. MR**1126193**, 10.1090/S0002-9939-1992-1126193-4**[16]**Edgar Asplund,*Čebyšev sets in Hilbert space*, Trans. Amer. Math. Soc.**144**(1969), 235–240. MR**0253023**, 10.1090/S0002-9947-1969-0253023-7**[17]**Jean-Pierre Aubin and Ivar Ekeland,*Applied nonlinear analysis*, Pure and Applied Mathematics (New York), John Wiley & Sons, Inc., New York, 1984. A Wiley-Interscience Publication. MR**749753****[18]**Ka Sing Lau,*Almost Chebyshev subsets in reflexive Banach spaces*, Indiana Univ. Math. J.**27**(1978), no. 5, 791–795. MR**0510772****[19]**J. M. Borwein and H. M. Strójwas,*Proximal analysis and boundaries of closed sets in Banach space. II. Applications*, Canad. J. Math.**39**(1987), no. 2, 428–472. MR**899844**, 10.4153/CJM-1987-019-4**[20]**Frank H. Clarke,*Optimization and nonsmooth analysis*, Canadian Mathematical Society Series of Monographs and Advanced Texts, John Wiley & Sons, Inc., New York, 1983. A Wiley-Interscience Publication. MR**709590**

F. H. Clarke,*Optimization and nonsmooth analysis*, 2nd ed., Classics in Applied Mathematics, vol. 5, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1990. MR**1058436****[21]**E. Asplund and R. T. Rockafellar,*Gradients of convex functions*, Trans. Amer. Math. Soc.**139**(1969), 443–467. MR**0240621**, 10.1090/S0002-9947-1969-0240621-X**[22]**T. S. Motzkin,*Sur quelques propriétés caractéristiques des ensembles convexes*, Att. R. Acad. Lincei, Rend.**21**(1935), 562-567.**[23]**Victor Klee,*Convexity of Chevyshev sets*, Math. Ann.**142**(1960/1961), 292–304. MR**0121633****[24]**Jean-Baptiste Hiriart-Urruty,*Ensembles de Tchebychev vs. ensembles convexes: l’état de la situation vu via l’analyse convexe non lisse*, Ann. Sci. Math. Québec**22**(1998), no. 1, 47–62 (French, with English and French summaries). MR**1626398****[25]**Robert R. Phelps,*Convex functions, monotone operators and differentiability*, 2nd ed., Lecture Notes in Mathematics, vol. 1364, Springer-Verlag, Berlin, 1993. MR**1238715**

Retrieve articles in *Transactions of the American Mathematical Society*
with MSC (1991):
49J52,
58C06,
58C20,
90C30

Retrieve articles in all journals with MSC (1991): 49J52, 58C06, 58C20, 90C30

Additional Information

**R. A. Poliquin**

Affiliation:
Department of Mathematical Sciences, University of Alberta, Edmonton, Alberta, Canada T6G 2G1

Email:
rene.poliquin@ualberta.ca

**R. T. Rockafellar**

Affiliation:
Department of Mathematics 354350, University of Washington, Seattle, Washington 98195-4350

Email:
rtr@math.washington.edu

**L. Thibault**

Affiliation:
Laboratoire d’Analyse Convexe, Université Montpellier II, 34095 Montpellier, France

Email:
thibault@math.univ-montp2.fr

DOI:
http://dx.doi.org/10.1090/S0002-9947-00-02550-2

Keywords:
Variational analysis,
distance functions,
single-valued projections,
proximal normals,
prox-regularity,
proximal smoothness,
primal-lower-nice functions,
hypomonotone mappings,
monotone mappings

Received by editor(s):
June 17, 1997

Received by editor(s) in revised form:
June 10, 1998

Published electronically:
June 9, 2000

Additional Notes:
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada under grant OGP41983 for the first author, by the National Science Foundation under grant DMS–9500957 for the second author, and by NATO under grant CRG 960360 for the third author. The authors are grateful for useful discussions with C. Combari, and for helpful comments made by the referee.

Article copyright:
© Copyright 2000
American Mathematical Society