Streaming generalized cross entropy. (English) Zbl 1493.94013
Summary: We propose a new method to combine adaptive processes with a class of entropy estimators for the case of streams of data. Starting from a first estimation obtained from a batch of initial data, model parameters are estimated at each step by combining the prior knowledge with the new observation (or a block of observations). This allows to extend the maximum entropy technique to a dynamical setting, also distinguishing between entropic contributions of the signal and the error. Furthermore, it provides a suitable approximation of standard GME problems when the exacted solutions are hard to evaluate. We test this method by performing numerical simulations at various sample sizes and batch dimensions. Moreover, we extend this analysis exploring intermediate cases between streaming GCE and standard GCE, i.e., considering blocks of observations of different sizes to update the estimates, and incorporating collinearity effects as well. The role of time in the balance between entropic contributions of signal and errors is further explored considering a variation of the Streaming GCE algorithm, namely Weighted Streaming GCE. Finally, we discuss the results: In particular, we highlight the main characteristics of this method, the range of application, and future perspectives.
MSC:
| 94A17 | Measures of information, entropy |
References:
| [1] | Amusa, L.; Zewotir, T.; North, D., Examination of entropy balancing technique for estimating some standard measures of treatment effects: a simulation study, Electron J Appl Stat Anal, 12, 2, 491-507 (2019) |
| [2] | Angelelli, M., Tropical limit and micro-macro correspondence in statistical physics, J Phys A: Math Theor, 50, 415202 (2017) · Zbl 1378.82032 |
| [3] | Angelelli, M.; Konopelchenko, B., Zeros and amoebas of partition functions, Rev Math Phys, 30, 9, 1850015 (2018) · Zbl 1447.51029 |
| [4] | Bagya Lakshmi, H.; Gallo, M.; Srinivasan, RM, Comparison of regression models under multi-collinearity, Electron J Appl Stat Anal, 11, 1, 340-368 (2018) |
| [5] | Berger, AL; Della Pietra, VJ; Della Pietra, S., A maximum entropy approach to natural language processing, Comput Linguist, 22, 1, 39-71 (1996) |
| [6] | Bertsekas, DP, Combined primal-dual and penalty methods for constrained minimization, SIAM J Control, 13, 3, 521-544 (1975) · Zbl 0269.90044 |
| [7] | Ciavolino, E.; Al-Nasser, AD, Comparing generalised maximum entropy and partial least squares methods for structural equation models, J Nonparametr Stat, 21, 8, 1017-1036 (2009) · Zbl 1175.62003 |
| [8] | Ciavolino, E.; Calcagnì, A., Generalized cross entropy method for analysing the SERVQUAL model, J Appl Stat, 42, 3, 520-534 (2014) · Zbl 1514.62492 |
| [9] | Ciavolino, E.; Calcagnì, A., A generalized maximum entropy (GME) estimation approach to fuzzy regression model, Appl Soft Comput, 38, 51-63 (2016) |
| [10] | Ciavolino, E.; Carpita, M., The GME estimator for the regression model with a composite indicator as explanatory variable, Qual Quant, 49, 3, 955-965 (2014) |
| [11] | Ciavolino, E.; Dahlgaard, JJ, Simultaneous equation model based on the generalized maximum entropy for studying the effect of management factors on enterprise performance, J Appl Stat, 36, 7, 801-815 (2009) · Zbl 1473.62384 |
| [12] | Ciavolino, E.; Carpita, M.; Al-Nasser, AD, Modelling the quality of work in the italian social co-operatives combining NPCA-RSM and SEM-GME approaches, J Appl Stat, 42, 1, 161-179 (2014) · Zbl 1514.62493 |
| [13] | Cover, TM; Thomas, JA, Elements of information theory (2006), Hoboken, NJ: Wiley, Hoboken, NJ · Zbl 1140.94001 |
| [14] | Crooks, GE, Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences, Phys Rev E, 60, 3, 2721-2726 (1999) |
| [15] | Daum, F., Nonlinear filters: beyond the Kalman filter, IEEE Aerosp Electron Syst Mag, 20, 8, 57-69 (2005) |
| [16] | Dewar, R., Maximum entropy production as an inference algorithm that translates physical assumptions into macroscopic predictions: don’t shoot the messenger, Entropy, 11, 4, 931-944 (2009) |
| [17] | Feynman, RP, Statistical mechanics: a set of lectures, advanced books classics (revised edition) (1998), Boulder, CO: Westview Press, Boulder, CO |
| [18] | Gladyshev, G., Thermodynamic theory of the evolution of living beings (1997), Hauppauge: Nova Science Pub. Inc, Hauppauge |
| [19] | Golan, A.; Erickson, G.; Rychert, JT; Smith, CR, Maximum entropy, likelihood and uncertainty, Maximum entropy and Bayesian methods. Boise, Idaho, USA, 1997: proceedings of the 17th international workshop on maximum entropy and Bayesian methods of statistical analysis, vol 98 of fundamental theories of physics (1998), Dordrecht: Springer, Dordrecht · Zbl 0912.62008 |
| [20] | Golan, A., Information and entropy econometrics—a review and synthesis, Found Trends Econ, 2, 1-2, 1-145 (2007) |
| [21] | Golan, A., Foundations of info-metrics: modeling, inference, and imperfect information (2018), New York, NY: Oxford University Press, New York, NY · Zbl 1383.62003 |
| [22] | Golan, A.; Judge, G.; Miller, D., Maximum entropy econometrics: robust estimation with limited data (1996), Chichester: Wiley, Chichester · Zbl 0884.62126 |
| [23] | Holzinger A, Hörtenhuber M, Mayer C, Bachler M, Wassertheurer S, Pinho AJ, Koslicki D (2014) On entropy-based data mining. In: Holzinger A, Jurisica I (eds) Interactive knowledge discovery and data mining in biomedical informatics. Springer Nature, pp 209-226. 10.1007/978-3-662-43968-5_12 |
| [24] | Jarzynski, C., Nonequilibrium equality for free energy differences, Phys Rev Lett, 78, 14, 2690-2693 (1997) |
| [25] | Jaynes, ET, Probability theory—the logic of science (2003), Cambridge: Cambridge University Press, Cambridge · Zbl 1045.62001 |
| [26] | Khinchin, AI, Mathematical foundations of information theory (1957), Grove: Dover, Grove · Zbl 0088.10404 |
| [27] | Landau, LD; Lifschitz, EM, Statistical physics, vol 5 of course of theoretical physics (1980), Oxford: Butterworth-Heinemann, Oxford |
| [28] | Papalia, R. Bernardini; Ciavolino, E., GME estimation of spatial structural equations models, J Classif, 28, 1, 126-141 (2011) |
| [29] | Pukelsheim, F., The three sigma rule, Am Stat, 48, 2, 88-91 (1994) |
| [30] | Simon, D., Optimal state estimation (2006), Hoboken, NJ: Wiley, Hoboken, NJ |
| [31] | Solomonoff RJ (1964) A formal theory of inductive inference, parts i, ii. Inf Control 7(1, 2):1-2, 224-254. 10.1016/s0019-9958(64)90131-7 · Zbl 0258.68045 |
| [32] | Widrow, B.; Winter, R., Neural nets for adaptive filtering and adaptive pattern recognition, Computer, 21, 3, 25-39 (1988) |
| [33] | Wu, X., A weighted generalized maximum entropy estimator with a data-driven weight, Entropy, 11, 4, 917-930 (2009) · Zbl 1179.91209 |
| [34] | Xu, X.; He, H.; Hu, D., Efficient reinforcement learning using recursive least-squares methods, J Artif Intell Res, 16, 259-292 (2002) · Zbl 0994.68123 |
| [35] | Zanetti, R., Recursive update filtering for nonlinear estimation, IEEE Trans Autom Control, 57, 6, 1481-1490 (2012) · Zbl 1369.93663 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
