Statistical methods for network surveillance. (English) Zbl 1400.62344
Summary: The term network surveillance is defined in general terms and illustrated with many examples. Statistical methodologies that can be used as tools for network surveillance are discussed. Details for 3 illustrative examples that address network security, surveillance for data network failures, and surveillance of email traffic flows are presented. Some open areas of research are identified.
MSC:
| 62P30 | Applications of statistics in engineering and industry; control charts |
| 62M07 | Non-Markovian processes: hypothesis testing |
| 90B25 | Reliability, availability, maintenance, inspection in operations research |
Keywords:
change point; control charts; dynamic networks; reliability; security; network surveillanceReferences:
| [1] | FuY, JeskeDR. SPC methods for non‐stationary correlated count data with application to network surveillance. Appl Stoch Models Bus Ind. 2014;30(6):708‐722. · Zbl 07880631 |
| [2] | MoodyJ, MuchaPJ. Portrait of political party polarization. Netw Sci. 2013;1(1):119‐121. |
| [3] | TaylorIW, LindingR, Warde‐FarleyD, et al. Dynamic modularity in protein interaction networks predicts breast cancer outcome. Nat Biotechnol. 2009;27:199‐204. |
| [4] | DebarH, DacierM, WespiA. Toward a taxonomy of intrusion detection systems. Comput Netw. 1999;31(8):805‐822. |
| [5] | KentS. On the trial of intrusions into information systems. IEEE Spectr. 2000;37(12):52‐56. |
| [6] | ScarfoneK, MellP. Guide to Intrusion Detection and Prevention Systems (IDPS). Gaithersburg, MD: National Institute of Standards and Technology; 2007. NIST SP 800‐94. |
| [7] | TartakovskyAG. Rapid detection of attacks in computer networks by quickest changepoint detection methods. In: AdamsN (ed.), HeardN (ed.), eds. Data Analysis for Network Cyber‐Security. London, UK: Imperial College Press; 2014:33‐70. |
| [8] | TartakovskyAG, VeeravalliVV. Asymptotically optimal quickest change detection in distributed sensor systems. Seq Anal. 2008;27(4):441‐475. · Zbl 1247.93014 |
| [9] | TartakovskyAG, RozovskiiBL, ShahK. A nonparametric multichart CUSUM test for rapid intrusion detection. Paper presented at: Joint Statistical Meetings; 2005; Minneapolis, MN. |
| [10] | TartakovskyAG, RozovskiiBL, BlažekR, KimH. Detection of intrusions in information systems by sequential change‐point methods. Stat Methodol. 2006;3(3):252‐340. · Zbl 1248.94032 |
| [11] | MirkovicJ, DietrichS, DittrichD, ReiherP. Internet Denial of Service: Attack and Defense Mechanisms. Upper Saddle River, NJ: Prentice Hall; 2005. |
| [12] | ChengC‐M, KungHT, TanK‐S. Use of spectral analysis in defense against DoS attacks. Paper presented at: IEEE Global Communications Conference; 2002; Taipei, Taiwan. |
| [13] | BarfordB, KlineJ, PlonkaD, RonA. A signal analysis of network traffic anomalies. Paper presented at: Internet Measurement Workshop; 2002; Marseille, France. |
| [14] | HussainA, HeidemannJ, PapadopoulosC. A framework for classifying denial of service attacks. Paper presented at: ACM SIGCOMM Conference; 2003; Karlsruhe, Germany. |
| [15] | LakhinaA, CrovellaM, DiotC. Mining anomalies using traffic feature distributions. Paper presented at: ACM SIGCOMM Conference; 2005; Philadelphia, PA. |
| [16] | PartridgeC, CousinsD, JacksonAW, KrishnanR, SaxenaT, StrayerWT. Using signal processing to analyze wireless data traffic. Paper presented at: ACM Workshop on Wireless Security; 2002; Atlanta, GA. |
| [17] | HuangP, FeldmannA, WillingerW. A non‐intrusive, wavelet‐based approach to detecting network performance problems. Paper presented at: ACM SIGCOMM Internet Measurement Workshop; 2001; San Francisco, CA. |
| [18] | LiL, LeeG. DDoS attack detection and wavelets. Paper presented at: 12th International Conference on Computer Communications and Networks; 2003; San Francisco, CA. |
| [19] | HeX, PapadopoulosC, HeidemannJ, MitraU, RiazU. Remote detection of bottleneck links using spectral and statistical methods. Comput Netw. 2009;53(3):279‐298. · Zbl 1181.68026 |
| [20] | BartlettG, HeidemannJ. PapadopoulosC. Understanding passive and active service discovery. Paper presented at: 6th AMC SIGCOMM Conference on Internet Measurement Conference; 2007; San Diego, CA. |
| [21] | HussainA, HeidemannJ, PapadopoulosC. Identification of repeated denial of service attacks. Paper presented at: IEEE Conference on Computer Communications; 2006; Barcelona, Spain. |
| [22] | MirkovicJ, ReiherP, PapadopoulosC, et al. Testings a collaborative DDoS defense in a red team/blue team exercise. IEEE Trans Comput. 2008;57(8):1098‐1112. |
| [23] | MitraU, HeidemannJ, OrtegaA, PapadopoulosC. Detecting and identifying malware: a new signal processing goal. IEEE Signal Process Mag. 2006;23(5):107‐111. |
| [24] | MarchetteD. Computer Intrusion Detection and Network Monitoring: A Statistical View‐point. New York, NY: Springer; 2001. · Zbl 1063.62160 |
| [25] | TartakovskyAG. Hybrid Intrusion Detection System Integrating Anomaly and Signature Intrusion Detection Methods. Rolling Hills Estates, CA: Argo Science Corp; 2010. Phase I Final Technical Report. |
| [26] | TartakovskyAG, PolunchenkoAS. Decentralized quickest change detection in distributed sensor systems with applications to information assurance and counter terrorism. Paper presented at: 13th Annual Army Conference on Applied Statistics; 2007; Houston, TX. |
| [27] | TartakovskyAG, VeeravalliV. Change‐point detection in multichannel and distributed systems with applications. In: MukhopadhyayN (ed.), DattaS (ed.), ChattopadhyayS (ed.), eds. Applications of Sequential Methodologies. New York, NY: Marcel Dekker; 2004. |
| [28] | TartakovskyAG, RozovskiiBL, BlažekR, KimH. A novel approach to detection of intrusions in computer networks via adaptive sequential and batch‐sequential change‐point detection methods. IEEE Trans Signal Process. 2006;54(9):3372‐3382. · Zbl 1373.68144 |
| [29] | TartakovskyAG, PolunchenkoAS, SokolovG. Efficient computer network anomaly detection by changepoint detection methods. IEEE J Sel Top Signal Process. 2013;7(1):4‐11. |
| [30] | TartakovskyAG, NikiforovI, BassevilleM. Sequential Analysis: Hypothesis Testing and Changepoint Detection. Boca Raton, FL: CRC Press; 2014. |
| [31] | PolunchenkoAS, TartakovskyAG, MukhopadhhyayN. Nearly optimal change‐point detection with an application to cybersecurity. Seq Anal. 2010;31(3):409‐435. · Zbl 1274.62515 |
| [32] | AntonatosS, AnagnostakisKG, MarkatosEP. PolychronakisM. Performance analysis of content matching intrusion detection systems. Paper presented at: Symposium on Applications and the Internet; 2004; Tokyo, Japan. |
| [33] | XinidisK, CharitakisI, AntonatosS, AnagnostakisKG, MarkatosEP. An active splitter architecture for intrusion detection and prevention. IEEE Trans Dependable Secure Comput. 2006;3(1):31‐44. |
| [34] | DiBenedettoS, MasseyD, PapadopoulosC, WalshP. Analyzing the Aftermath of the McColo Shutdown. Paper presented at: Workshop on Trust and Security in the Future Internet in conjunction with SAINT 2009; 2009; Seattle, WA. |
| [35] | FeatherFW, SiewiorekR, MaxionD. Fault detection in ethernet networks using anomaly signature matching. Paper presented at: ACM SIGCOMM ’93 Conference on Communications Architectures, Protocols and Applications; 1993; San Francisco, CA. |
| [36] | BrutlagJD. Aberrant behavior detection in time series for network monitoring. Paper presented at: 14th Systems Administration Conference; 2000; New Orleans, LA. |
| [37] | ThottanM, JiC. Adaptive thresholding for proactive network problem detection. Paper presented at: IEEE 3rd International Workshop on Systems Management. 1998; Newport, RI. |
| [38] | CaoJ, ChenA, BuT, BuvaneswariA. Monitoring time‐varying network streams using state‐space models. Paper presented at: IEEE INFOCOM. 2009; Rio de Janeiro, Brazil. |
| [39] | JeskeDR, Montes de OcaV, BischoffW, MarvastiM. CUSUM techniques for timeslot sequences with applications to network surveillance. Comput Stat Data Anal. 2009;53:4332‐4344. · Zbl 1301.90014 |
| [40] | Montes de OcaV, JeskeDR, ZhangQ, RendonC, MarvastiM. A CUSUM changepoint detection algorithm for non‐stationary sequences with application to network surveillance. J Softw Syst. 2010;83:1288‐1298. |
| [41] | LambertD, LiuC. Adaptive thresholds: monitoring streams of network counts. J Am Stat Assoc. 2006;101(473):78‐88. · Zbl 1118.62388 |
| [42] | MarvastiM, JeskeDR. Nonparametric method for determination of anomalous event states in complex systems exhibiting non‐stationarity. US patent 8,275,563. September 25, 2012. |
| [43] | GluhovskyI, HoffmanAJ, LeeHM, YashchinE. System and method of predicting future behavior of a battery of end‐to‐end probes to anticipate and prevent computer network performance degradation. US patent 7,081,823. July 25, 2006. |
| [44] | LeeY, NelderJA, NohM. H‐likelihood: problems and solutions. Stat Comput. 2007;17(1):49‐55. |
| [45] | Jones‐FarmerLA, WoodallWH, SteinerSH, ChampCW. An overview of phase I analysis for process improvement and monitoring. J Qual Technol. 2014;46(3:265‐280. |
| [46] | JeskeDR. Determining the phase 1 study sample size to control the accuracy of the conditional in‐control ARL of a normal‐theory CUSUM. Qual Reliab Eng Int. 2016;32:2499‐2504. |
| [47] | ZhouY, LiJ, JeskeDR. A wavelet‐based nonparametric CUSUM control chart for Autocorrelated processes with applications to network surveillance. Paper presented at: Joint Statistical Meetings; 2017; Baltimore, MD. |
| [48] | WilsonJD, StevensNT, WoodallWH. Modeling and estimating change in temporal networks via a dynamic degree corrected stochastic block model. 2016. https://arxiv.org/abs/1605.04049 |
| [49] | KarrerB, NewmanME. Stochastic block models and community structure in networks. Phys Review E. 2011;83(1). https://doi.org/10.1103/PhysRevE.83.016107 · doi:10.1103/PhysRevE.83.016107 |
| [50] | PeelL, ClausetA. Detecting change points in the large‐scale structure of evolving networks. Paper presented at: 29th AAAI Conference on Artificial Intelligence; 2015; Austin, TX. |
| [51] | BhamidiS, JinJ, NobelAB. Change point detection in network models: Preferential attachment and long‐range dependence. 2015. https://arxiv.org/abs/1508.02043 |
| [52] | SavageD, ZhangX, YuX, ChouP, WangQ. Anomaly detection in online social networks. Soc Netw. 2014;39:62‐70. |
| [53] | RanshousS, ShenS, KoutraD, HarenbergS, FaloutsosC, SamatovaNF. Anomaly detection in dynamic networks: a survey. Wiley Interdiscip Rev Comput Stat. 2015;7(3):223‐247. · Zbl 07912769 |
| [54] | BinduPV, ThilagamPS. Mining social networks for anomalies: methods and challenges. J Netw Comput Appl. 2016;68:213‐229. |
| [55] | WoodallWH, ZhaoMJ, PaynabarK, SparksR, WilsonJD. An overview and perspective on social network monitoring. IISE Trans. 2017;49(3):354‐365. |
| [56] | McCullohI, CarleyKM. Detecting change in longitudinal social networks. J Soc Struct. 2011;12:1‐37. |
| [57] | PriebeCE, ConroyJM, MarchetteDJ, ParkY. Scan statistics on Enron graphs. Comput Math Organ Theory. 2005;11(3):229‐247. · Zbl 1086.68562 |
| [58] | SparksR, Wilson, JD. Monitoring communication outbreaks among an unknown team of actors in dynamic networks. 2016. https://arxiv.org/abs/1606.09308 |
| [59] | WeißCH. Controlling processes of Poisson counts. Qual Reliab Eng Int. 2007;23(6):741‐754. |
| [60] | WeißCH. EWMA monitoring of correlated processes of Poisson counts. Qual Technol Quant Manag. 2009;6(2):137‐153. |
| [61] | SparksRS, KeighleyT, MuscatelloD. Improving EWMA plans for detecting unusual increases in Poisson counts. Adv Decis Sci. 2009;2009:1‐16. · Zbl 1175.62117 |
| [62] | SparksRS, CarterC, GrahamPL, et al. A strategy for understanding the sources of variation in syndromic surveillance for bioterrorism and public health incidence. IIE Trans. 2010;42:613‐631. |
| [63] | ZhouQ, ZouC, WangZ, JiangW. Likelihood‐based EWMA charts for monitoring Poisson count data with time‐varying sample sizes. J Am Stat Assoc. 2012;107(499):1049‐1062. · Zbl 1395.62378 |
| [64] | ZengD, ChangW, ChenH. A comparative study of spatio‐temporal hotspot analysis techniques in security informatics. Paper presented at: 7th International IEEE Conference on Intelligent Transportation Systems; 2004; Washington, WA. |
| [65] | KimY, O’KellyM. A bootstrap based space-time surveillance model with an application to crime occurrences. J Geogr Syst. 2008;10(2):141‐165. |
| [66] | NeillDB. Expectation‐based scan statistics for monitoring spatial time series data. Int J Forecasting. 2009;25(3):498‐517. |
| [67] | NakayaT, YanoK. Visualizing crime clusters in a space‐time cube: an exploratory data‐analysis approach using space‐time kernel density estimation and scan statistics. Trans GIS. 2010;14(3):223‐239. |
| [68] | Mei, Y. Quickest detection in censoring sensor networks. Paper presented at: IEEE International Symposium on Information Theory; 2011; Saint Petersburg, Russia. |
| [69] | AzarnoushB, PaynabarK, BekkiJ, RungerG. Monitoring temporal homogeneity in attributed network streams. J Qual Technol. 2016;48:28‐43. |
| [70] | MontgomeryDC. Introduction to statistical quality control. 7th ed. New York, NY: John Wiley & Sons; 2013. · Zbl 1266.62003 |
| [71] | WoodallWH, MontgomeryDC. Research issues and ideas in statistical process control. J Qual Technol. 1999;31(4):376‐386. |
| [72] | WoodallWH, MontgomeryDC. Some current directions in the theory and application of statistical process monitoring. J Quality Technology. 2014;46(1):78‐94. |
| [73] | PorterMA, OnnelaJ‐P, MuchaPJ. Communities in networks. Notices Am Math Soc. 2009;56:1082‐1097. · Zbl 1188.05142 |
| [74] | FortunatoS. Community detection in graphs. Phys Rep. 2010;486(3):75‐174. |
| [75] | YuL, WoodallWH, TsuiKL. Detecting node propensity changes in dynamic degree corrected stochastic block models. Paper presented at: 5th International Symposium on Statistical Process Monitoring; 2017; Seoul, South Korea. |
| [76] | PonsP, LatapyM. Computing communities in large networks using random walks. Paper presented at: International Symposium on Computer and Information Sciences; 2005; Istanbul, Turkey. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
