Skip to main content
Log in

Efficient Global Optimization of Expensive Black-Box Functions

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

In many engineering optimization problems, the number of function evaluations is severely limited by time or cost. These problems pose a special challenge to the field of global optimization, since existing methods often require more function evaluations than can be comfortably afforded. One way to address this challenge is to fit response surfaces to data collected by evaluating the objective and constraint functions at a few points. These surfaces can then be used for visualization, tradeoff analysis, and optimization. In this paper, we introduce the reader to a response surface methodology that is especially good at modeling the nonlinear, multimodal functions that often occur in engineering. We then show how these approximating functions can be used to construct an efficient global optimization algorithm with a credible stopping rule. The key to using response surfaces for global optimization lies in balancing the need to exploit the approximating surface (by sampling where it is minimized) with the need to improve the approximation (by sampling where prediction error may be high). Striking this balance requires solving certain auxiliary problems which have previously been considered intractable, but we show how these computational obstacles can be overcome.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Alexandrov, N. M., Dennis, Jr., J. E., Lewis, R. M. and Torczon, V. (1998), A trust-region framework for managing the use of approximation models in optimization, Structural Optimization 15: 16–23.

    Google Scholar 

  2. Androulakis, I. P., Maranas, C. D. and Floudas, C. A. (1995), αBB: a global optimization method for general constrained nonconvex problems, Journal of Global Optimization 7: 337–363.

    Google Scholar 

  3. Aslett, R., Buck, R. J., Duvall, S. G., Sacks, J. and Welch, W. J. (1998), Circuit optimization via sequential computer experiments: design of an output buffer, Applied Statistics 47: 31–48.

    Google Scholar 

  4. Betro, B. (1991), Bayesian methods in global optimization, Journal of Global Optimization 1: 1–14.

    Google Scholar 

  5. Boender, C. G. E. and Romeijn, H. E. (1995), Stochastic methods, in R. Horst and P. M. Pardalos, eds., Handbook of Global Optimization, pp. 829–869, Kluwer Academic Publishers, Dordrecht/Boston/London.

    Google Scholar 

  6. Booker, A. J., Conn, A. R., Dennis, J. E., Frank, P. D., Trossett, M. and Torczon, V. (1995), Global modeling for optimization, Boeing Information and Support Services, Technical Report ISSTECH–95-032.

  7. Booker, A. J., Dennis, J. E., Frank, P. D., Serafini, D. B. and Torczon, V. (1997), Optimization using surrogate objectives on a helicopter test example, Boeing Shared Services Group, Technical Report SSGTECH–97-027.

  8. Box, G. E. P., Hunter, W. G. and Hunter, J. S. (1978), Statistics for Experimenters, John Wiley, New York.

    Google Scholar 

  9. Cox, D. D. and John. S. (1997), SDO: A statistical method for global optimization, in N. Alexandrov and M. Y. Hussaini, eds., Multidisciplinary Design Optimization: State of the Art, pp. 315–329, SIAM, Philadelphia.

    Google Scholar 

  10. Cressie. N. (1989), Geostatistics, The American Statistician 43: 197–202. See also the comment on this article by G. Wahba and the reply by N. Cressie (1990), in The American Statistician 44: 255–258.

    Google Scholar 

  11. Cressie, N. (1990), The origins of kriging, Mathematical Geology 22: 239–252.

    Google Scholar 

  12. Cressie, N. (1993), Statistics for Spatial Data, John Wiley, New York.

    Google Scholar 

  13. Currin, C., Mitchell, T., Morris, M. and Ylvisaker, D. (1991), Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments, Journal of the American Statistical Association 86: 953–963.

    Google Scholar 

  14. Dixon, L. C. W. and Szego, G. P. (1978), The global optimisation problem: an introduction, in L. C. W. Dixon and G. P. Szego (eds.), Towards Global Optimisation, Vol. 2, pp. 1–15. North Holland, Amsterdam.

    Google Scholar 

  15. Eby, D., Averill, R. C., Punch III, W. F. and Goodman, E. D. (1998), Evaluation of injection island GA performance on flywheel design optimization, in I. C. Parmee (ed.), Adaptive Computing in Design and Manufacture, Springer Verlag.

  16. Elder IV, J. F. (1992), Global R d optimization when probes are expensive: the GROPE algorithm. Proceedings of the 1992 IEEE International Conference on Systems, Man, and Cybernetics, Vol. 1, pp. 577–582, Chicago.

  17. Gao, F., Sacks, J. and Welch, W. J. (1996), Predicting urban ozone levels and trends with semiparametric modeling, Journal of Agricultural, Biological, and Environmental Statistics 1: 404–425.

    Google Scholar 

  18. Koehler, J. and Owen, A. (1996), Computer experiments, in S. Ghosh and C. R. Rao (eds.), Handbook of Statistics, 13: Design and Analysis of Experiments, pp. 261–308, Elsevier, Amsterdam.

    Google Scholar 

  19. Kushner, H. J. (1964), A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise, Journal of Basic Engineering 86: 97–106.

    Google Scholar 

  20. Locatelli, M. (1997), Bayesian algorithms for one-dimensional global optimization, Journal of Global Optimization 10: 57–76.

    Google Scholar 

  21. Matheron, G. (1963), Principles of geostatistics, Economic Geology 58: 1246–1266.

    Google Scholar 

  22. McKay, M. D., Conover, W. J. and Beckman, R. J. (1979), A comparison of three methods for selecting values of input variables in the analysis of output from a computer code, Technometrics 21: 239–245.

    Google Scholar 

  23. Mockus, J. (1994), Application of Bayesian approach to numerical methods of global and stochastic optimization, Journal of Global Optimization 4: 347–365.

    Google Scholar 

  24. Mockus, J., Tiesis, V. and Zilinskas, A. (1978), The application of Bayesian methods for seeking the extremum, in L. C. W. Dixon and G. P. Szego (eds.), Towards Global Optimisation, Vol.2, pp. 117–129. North Holland, Amsterdam.

    Google Scholar 

  25. Morris, M. D., Mitchell, T. J. and Ylvisaker, D. (1993), Bayesian design and analysis of computer experiments: use of derivatives in surface prediction, Technometrics 35: 243–255.

    Google Scholar 

  26. Parzen, E. (1963), A new approach to the synthesis of optimal smoothing and prediction systems, in R. Bellman, (ed.), Mathematical Optimization Techniques, pp. 75–108, University of California Press, Berkeley.

    Google Scholar 

  27. Perttunen, C. (1991), A computational geometric approach to feasible region division in constrained global optimization, Proceedings of the 1991 IEEE Conference on Systems, Man, and Cybernetics, Vol. 1, pp. 585–590.

    Google Scholar 

  28. Press, W. H., Flannery, B. P., Teukolsky, S. A. and Vetterling, W. T. (1993), Numerical Recipes in FORTRAN, Cambridge University Press.

  29. Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. P. (1989), Design and analysis of computer experiments (with discussion), Statistical Science 4: 409–435.

    Google Scholar 

  30. Sacks, J. and Ylvisaker, D. (1970), Statistical designs and integral approximation, in R. Pyke (ed.), Proceedings of the Twelfth Biennial Seminar of the Canadian Mathematical Congress, pp. 115–136, Canadian Mathematical Congress, Montreal.

    Google Scholar 

  31. Schonlau, M. (1997), Computer experiments and global optimization, Ph. D. Thesis, University of Waterloo, Waterloo, Ontario, Canada.

    Google Scholar 

  32. Schonlau, M., Welch, W. J. and Jones, D. R. (1998), Global versus local search in constrained optimization of computer models, to appear in N. Flournoy, W. F. Rosenberger and W. K. Wong (eds.), New Developments and Applications in Experimental Design, Institute of Mathematical Statistics. Also available as Technical Report RR–97-11, Institute for Improvement in Quality and Productivity, University of Waterloo, Waterloo, Ontario, Canada, December 1997.

  33. Stuckman, B. E. (1988), A global search method for optimizing nonlinear systems, IEEE Transactions on Systems, Man, and Cybernetics 18: 965–977.

    Google Scholar 

  34. Theil, H. (1971), Principles of Econometrics, John Wiley, New York.

    Google Scholar 

  35. Ver Hoef, J. M. and Cressie, N. (1993), Multivariable spatial prediction, Mathematical Geology 25: 219–240.

    Google Scholar 

  36. Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J. and Morris, M. D. (1992), Screening, predicting, and computer experiments, Technometrics 34: 15–25.

    Google Scholar 

  37. Zilinskas, A. (1992), A review of statistical models for global optimization, Journal of Global Optimization 2: 145–153.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jones, D.R., Schonlau, M. & Welch, W.J. Efficient Global Optimization of Expensive Black-Box Functions. Journal of Global Optimization 13, 455–492 (1998). https://doi.org/10.1023/A:1008306431147

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1008306431147

Navigation