×

Model checking differentially private properties. (English) Zbl 1512.68157

Summary: With the explosion of digital data collected from social apps, privacy protection regulations have been issued by almost all countries. Differential privacy is proposed as a successful technique to make use of these data, without leaking personal private data at the same time. In this paper, we investigate logical reasoning for differential privacy properties and propose model checking algorithms. We introduce the branching time temporal logic dpCTL* to specify differentially private properties. Several mechanisms in differential privacy are formalized as Markov chains or Markov decision processes. In our framework, we show that subtle privacy conditions are specified by dpCTL*. In order to verify privacy properties automatically, model checking problems are investigated. We develop a model checking algorithm for Markov chains. Model checking dpCTL* properties on Markov decision processes however is shown to be undecidable. Therefore, we propose a sound algorithm to verify these properties in Markov decision processes. We implement our model checking algorithm into a tool, based on the PRISM language, as well as show the experimental results and a case study of stream processing in differential privacy under the essential fairness assumptions.

MSC:

68Q60 Specification and verification (program logics, model checking, etc.)
03B44 Temporal logic
60J20 Applications of Markov chains and discrete-time Markov processes on general state spaces (social mobility, learning theory, industrial processes, etc.)
68P27 Privacy of data
90C40 Markov and semi-Markov decision processes
Full Text: DOI

References:

[1] Fung, B. C.M.; Wang, K.; Chen, R.; Yu, P. S., Privacy-preserving data publish: a survey of recent developments, ACM Comput. Surv., 42, 4, Article 14 pp. (2010)
[2] Dwork, C.; Roth, A., The algorithmic foundations of differential privacy, Found. Trends Theor. Comput. Sci., 9, 3-4, 211-407 (2014) · Zbl 1302.68109
[3] Ji, Z.; Lipton, Z. C.; Elkan, C., Differential privacy and machine learning: a survey and review (2014), CoRR
[4] (2016), WWDC, Engineering privacy for your users
[5] (2021), Apple, Protecting your identity
[6] Ding, B.; Kulkarni, J.; Yekhanin, S., Collecting telemetry data privately, (NIPS (2017)), 3574-3583
[7] Dwork, C.; McSherry, F.; Nissim, K.; Smith, A., Calibrating noise to sensitivity in private data analysis, (Halevi, S.; Rabin, T., TCC. TCC, LNCS, vol. 3876 (2006), Springer), 265-284 · Zbl 1112.94027
[8] Dwork, C., Differential privacy, (Bugliesi, M.; Preneel, B.; Sassone, V.; Wegener, I., ICALP. ICALP, LNCS, vol. 4052 (2006), Springer), 1-12 · Zbl 1133.68330
[9] Mironov, I., On significance of the least significant bits for differential privacy, (Yu, T.; Danezis, G.; Gligor, V. D., ACM CCS (2012)), 650-661
[10] Tang, J.; Korolova, A.; Bai, X.; Wang, X.; Wang, X., Privacy loss in apple’s implementation of differential privacy on MacOS 10.12 (2017), CoRR
[11] Manna, Z.; Pnueli, A., The Temporal Logic of Reactive and Concurrent Systems - Specification (1992), Springer
[12] Baier, C.; Katoen, J.-P., Principles of Model Checking (2008), The MIT Press · Zbl 1179.68076
[13] Puterman, M. L., Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley Series in Probability and Statistics, vol. 594 (2005), John Wiley & Sons, Inc. · Zbl 1184.90170
[14] Tschantz, M. C.; Kaynar, D.; Datta, A., Formal verification of differential privacy for interactive systems, (Mathematical Foundations of Programming Semantics, vol. 276 (2011)), 61-79 · Zbl 1342.68212
[15] Alvim, M. S.; Andrés, M. E.; Chatzikokolakis, K.; Degano, P.; Palamidessi, C., On the information leakage of differentially-private mechanisms, J. Comput. Secur., 23, 4, 427-469 (2015)
[16] Gazeau, I.; Miller, D.; Palamidessi, C., Preserving differential privacy under finite-precision semantics, Theor. Comput. Sci., 655, 92-108 (2016) · Zbl 1419.94036
[17] Winograd-Cort, D.; Haeberlen, A.; Roth, A.; Pierce, B. C., A framework for adaptive differential privacy, (Proceedings of the ACM on Programming Languages, vol. 1 (2017)), Article 10 pp.
[18] Gaboardi, M.; Haeberlen, A.; Hsu, J.; Narayan, A.; Pierce, B. C., Linear dependent types for differential privacy, (POPL (2013)), 357-370 · Zbl 1301.68111
[19] Reed, J.; Pierce, B. C., Distance makes the types grow stronger: a calculus for differential privacy, (ICFP (2010), ACM), 157-168 · Zbl 1323.68254
[20] Zhang, D.; Kifer, D., LightDP: towards automating differential privacy proofs, (POPL (2017), ACM), 888-901 · Zbl 1380.68136
[21] Wang, Y.; Ding, Z.; Wang, G.; Kifer, D.; Zhang, D., Proving differential privacy with shadow execution, (PLDI (2019)), 655-669
[22] Wang, Y.; Ding, Z.; Kifer, D.; Zhang, D., Checkdp: an automated and integrated approach for proving differential privacy or finding precise counterexamples, (CCS (2020)), 919-938
[23] Barthe, G.; Fong, N.; Gaboardi, M.; Grégoire, B.; Hsu, J.; Strub, P.-Y., Advanced probabilistic couplings for differential privacy, (CCS (2016), ACM), 55-67
[24] Barthe, G.; Gaboardi, M.; Gregoire, B.; Hsu, J.; Strub, P.-Y., Proving differential privacy via probabilistic couplings, (LICS (2016), IEEE) · Zbl 1401.68184
[25] Barthe, G.; Köpf, B.; Olmedo, F.; Zanella-Béguelin, S., Probabilistic relational reasoning for differential privacy, (POPL (2012), ACM), 97-110 · Zbl 1321.68182
[26] Barthe, G.; Danezis, G.; Grégoire, B.; Kunz, C.; Zanella-Béguelin, S., Verified computational differential privacy with applications to smart metering, (CSF (2013), IEEE), 287-301
[27] Barthe, G.; Gaboardi, M.; Arias, E. J.G.; Hsu, J.; Kunz, C.; Strub, P. Y., Proving differential privacy in hoare logic, (CSF (2014), IEEE), 411-424
[28] Barthe, G.; Gaboardi, M.; Arias, E. J.G.; Hsu, J.; Roth, A.; Strub, P., Higher-order approximate relational refinement types for mechanism design and differential privacy, (POPL (2015), ACM), 68-79 · Zbl 1346.68058
[29] Barthe, G.; Farina, G. P.; Gaboardi, M.; Arias, E. J.G.; Gordon, A.; Hsu, J.; Strub, P.-Y., Differentially private Bayesian programming, (CCS (2016), ACM), 68-79
[30] Ding, Z.; Wang, Y.; Wang, G.; Zhang, D.; Kifer, D., Detecting violations of differential privacy, (Backes, M.; Wang, X., CCS (2018), ACM), 475-489
[31] Bichsel, B.; Gehr, T.; Drachsler-Cohen, D.; Tsankov, P.; Vechev, M., Dp-finder: Finding differential privacy violations by sampling and optimization, 508-524 (2018)
[32] Bichsel, B.; Steffen, S.; Bogunovic, I.; Vechev, M., Dp-sniper: black-box discovery of differential privacy violations using classifiers, (2021 IEEE Symposium on Security and Privacy (SP) (2021)), 391-409
[33] Gaboardi, M.; Nissim, K.; Purser, D., The complexity of verifying loop-free programs as differentially private, (ICALP, vol. 168 (2020)), Article 129 pp.
[34] Barthe, G.; Chadha, R.; Jagannath, V.; Sistla, A.; Viswanathan, M., Deciding differential privacy for programs with finite inputs and outputs, 141-154 (2020) · Zbl 1507.68111
[35] Chatzikokolakis, K.; Gebler, D.; Palamidessi, C.; Xu, L., Generalized bisimulation metrics, (Baldan, P.; Gorla, D., CONCUR 2014 - Concurrency Theory (2014), Springer: Springer Berlin Heidelberg, Berlin, Heidelberg), 32-46 · Zbl 1417.68123
[36] Chistikov, D.; Murawski, A. S.; Purser, D., Bisimilarity distances for approximate differential privacy, (Lahiri, S. K.; Wang, C., Automated Technology for Verification and Analysis (2018), Springer International Publishing: Springer International Publishing Cham), 194-210 · Zbl 1517.68108
[37] Chistikov, D.; Murawski, A. S.; Purser, D., Asymmetric distances for approximate differential privacy, (Fokkink, W.; van Glabbeek, R., 30th International Conference on Concurrency Theory (CONCUR 2019). 30th International Conference on Concurrency Theory (CONCUR 2019), Leibniz International Proceedings in Informatics (LIPIcs), vol. 140 (2019), Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik: Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik Dagstuhl, Germany), Article 10 pp.
[38] Chistikov, D.; Kiefer, S.; Murawski, A. S.; Purser, D., The big-O problem for labelled Markov chains and weighted automata, (Konnov, I.; Kovács, L., 31st International Conference on Concurrency Theory (CONCUR 2020). 31st International Conference on Concurrency Theory (CONCUR 2020), Leibniz International Proceedings in Informatics (LIPIcs), vol. 171 (2020), Schloss Dagstuhl-Leibniz-Zentrum für Informatik: Schloss Dagstuhl-Leibniz-Zentrum für Informatik Dagstuhl, Germany), Article 41 pp. · Zbl 07559497
[39] Chadha, R.; Sistla, A. P.; Viswanathan, M., On linear time decidability of differential privacy for programs with unbounded inputs, (2021 36th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS) (2021)), 1-13
[40] Barthe, G.; Chadha, R.; Krogmeier, P.; Sistla, A. P.; Viswanathan, M., Deciding accuracy of differential privacy schemes, (Proc. ACM Program. Lang., 5 (POPL) (Jan. 2021))
[41] Liu, D.; Wang, B.; Zhang, L., Model checking differentially private properties, (Ryu, S., APLAS. APLAS, LNCS, vol. 11275 (2018), Springer), 394-414 · Zbl 1519.68137
[42] Hahn, E. M.; Li, Y.; Schewe, S.; Turrini, A.; Zhang, L., IscasMC: a web-based probabilistic model checker, (FM. FM, Lecture Notes in Computer Science, vol. 8442 (2014), Springer), 312-317
[43] Ghosh, A.; Roughgarden, T.; Sundararajan, M., Universally utility-maximizing privacy mechanisms, (STOC (2009), ACM: ACM New York, NY, USA), 351-360 · Zbl 1304.94060
[44] Ghosh, A.; Roughgarden, T.; Sundararajan, M., Universally utility-maximizing privacy mechanisms, SIAM J. Comput., 41, 6, 1673-1693 (2012) · Zbl 1271.68102
[45] Hansson, H.; Jonsson, B., A logic for reasoning about time and reliability, Form. Asp. Comput., 6, 5, 512-535 (1994) · Zbl 0820.68113
[46] Bianco, A.; de Alfaro, L., Model checking of probabalistic and nondeterministic systems, (FSTTCS. FSTTCS, LNCS, vol. 1026 (1995), Springer), 499-513 · Zbl 1354.68167
[47] Courcoubetis, C.; Yannakakis, M., The complexity of probabilistic verification, J. ACM, 42, 4, 857-907 (1995) · Zbl 0885.68109
[48] Couvreur, J.-M.; Saheb, N.; Sutre, G., An optimal automata approach to LTL model checking of probabilistic systems, (LPAR. LPAR, LNCS, vol. 2850 (2003)), 361-375 · Zbl 1273.68224
[49] Baier, C.; Kiefer, S.; Klein, J.; Klüppelholz, S.; Müller, D.; Worrell, J., Markov chains and unambiguous Büchi automata, (CAV. CAV, LNCS, vol. 9779 (2016), Springer), 23-42 · Zbl 1411.68051
[50] Rabin, M., Probabilistic automata, Inf. Control, 6, 3, 230-245 (1963) · Zbl 0182.33602
[51] Paz, A., Introduction to Probabilistic Automata, Computer Science and Applied Mathematics (1971), Academic Press, Inc.: Academic Press, Inc. Orlando, FL, USA · Zbl 0234.94055
[52] Tzeng, W., A polynomial-time algorithm for the equivalence of probabilistic automata, SIAM J. Comput., 21, 2, 216-227 (1992) · Zbl 0755.68075
[53] Kwiatkowska, M.; Norman, G.; Parker, D., PRISM 4.0: verification of probabilistic real-time systems, (Gopalakrishnan, G.; Qadeer, S., CAV. CAV, LNCS, vol. 6806 (2011), Springer), 585-591
[54] Dwork, C.; Naor, M.; Pitassi, T.; Rothblum, G. N., Differential privacy under continual observation, (STOC (2010), ACM: ACM New York, NY, USA), 715-724 · Zbl 1293.68096
[55] Dwork, C.; Naor, M.; Pitassi, T.; Rothblum, G. N.; Yekhanin, S., Pan-private streaming algorithms, (Innovations in Computer Science (2010)), 66-80
[56] Liu, D.; Wang, B.; Zhang, L., Verifying Pufferfish Privacy in Hidden Markov Models, (VMCAI (2022), Springer-Verlag: Springer-Verlag Berlin, Heidelberg), 174-196 · Zbl 1498.68104
[57] Fu, C.; Hahn, E. M.; Li, Y.; Schewe, S.; Sun, M.; Turrini, A.; Zhang, L., EPMC Gets Knowledge in Multi-agent Systems, (VMCAI (2022), Springer-Verlag: Springer-Verlag Berlin, Heidelberg), 93-107
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.