Abstract
This study aimed to improve the labeling of objects inside images in the crowdsourcing process. Images are one of the most widely used types of data on the internet. Controlling and refining images through the crowdsourcing process is time-consuming and tedious because of their quality and nature. Gamification of data collection was presented as a solution to review, categorize images, and motivate people. Because participant motivation might influence the quality of the output data, we used gamification elements to manage user interaction in this study. The proposed method has a great effect on improving the quality of output data by considering various challenges such as motivation, financial costs, and delays. The proposed algorithm calculates the average of the points specified by each user and then compares it with the average of the total correct answers. In the end, the proposed algorithm uses this comparison to decide whether to accept or reject the answer. In this research, the LabelMe, Flickr, and VOC2012 datasets were used. Implementing the proposed method in a real context showed that the proposed design improved the image labeling accuracy, which was increased by 11.3% compared to the previous methods. In this experiment, the people who interacted the most generated the most accurate data.
Similar content being viewed by others
Data availability
The dataset generated during the current study are available from the corresponding author on reasonable request.
References
Adelsman RM, Whinston AB (1977) Sophisticated voting with information for two voting functions. J Econ Theory 15(1):145–159
Allahverdi R, Bastanfard A, Akbarzadeh D (2012) Sasanian coins classification using discrete cosine transform. In: The 16th CSI international symposium on artificial intelligence and signal processing (AISP 2012). IEEE, pp 278–282
Amirkhani D, Bastanfard A (2019) Inpainted image quality evaluation based on saliency map features. In: 2019 5th Iranian conference on signal processing and intelligent systems (ICSPIS), pp 1–6. https://doi.org/10.1109/ICSPIS48872.2019.9066140
Amirkhani D, Bastanfard A (2021) An objective method to evaluate exemplar-based inpainted images quality using Jaccard index. Multimed Tools Appl 80:26199–26212. https://doi.org/10.1007/s11042-021-10883-3
Antin J (2012) Gamification is not a dirty word. Interactions 19(4):14–16
Aparicio AF, Vela FL, Sánchez JL, Montes JL (2012) Analysis and application of gamification. In: Proceedings of the 13th international conference on interacción persona-ordenador, pp 1–2
Bagley KS (2012) Conceptual mile markers to improve time-to-value for exploratory search sessions. University of Massachusetts Lowell
Bastanfard A, Amirkhani D (2021) Improving the accuracy of the annotation algorithm in pattern-based tennis game video. In: 2021 29th Iranian conference on electrical engineering (ICEE). IEEE, pp 493–497
Bastanfard A, Aghaahmadi M, Fazel M, Moghadam M (2009) Persian viseme classification for developing visual speech training application. In: Pacific-rim conference on multimedia. Springer, Berlin, Heidelberg, pp 1080–1085
Bastanfard A, Jafari S, Amirkhani D (2019) Improving tracking soccer players in shaded playfield video. In: 2019 5th Iranian conference on signal processing and intelligent systems (ICSPIS). IEEE, pp 1–8
Bastanfard A, Amirkhani D, Mohammadi M (2022) Toward image super-resolution based on local regression and nonlocal means. Multimed Tools Appl 81:23473–23492. https://doi.org/10.1007/s11042-022-12584-x
Berengueres J, Alsuwairi F, Zaki N, Ng T (2013) Gamification of a recycle bin with emoticons. In: 2013 8th ACM/IEEE international conference on human-robot interaction (HRI). IEEE, pp 83–84
Bhatti UA, Huang M, Wang H, Zhang Y, Mehmood A, Di W (2018) Recommendation system for immunization coverage and monitoring. Human Vaccin Immunother 14(1):165–171
Bhatti UA, Huang M, Wu D, Zhang Y, Mehmood A, Han H (2019) Recommendation system using feature extraction and pattern recognition in clinical care systems. Enterp Inf Syst 13(3):329–351
Bhatti SS, Gao X, Chen G (2020) General framework, opportunities and challenges for crowdsourcing techniques: a comprehensive survey. J Syst Softw 167:110611
Bhatti UA, Yu Z, Chanussot J, Zeeshan Z, Yuan L, Luo W, Nawaz SA, Bhatti MA, Ain QU, Mehmood A (2021) Local similarity-based spatial–spectral fusion hyperspectral image classification with deep CNN and Gabor filtering. IEEE Trans Geosci Remote Sens 60:1–15
Bhatti UA, Zeeshan Z, Nizamani MM, Bazai S, Yu Z, Yuan L (2022) Assessing the change of ambient air quality patterns in Jiangsu Province of China pre-to post-COVID-19. Chemosphere 288:132569
Bhatti UA, Yu Z, Hasnain A, Nawaz SA, Yuan L, Wen L, Bhatti MA (2022) Evaluating the impact of roads on the diversity pattern and density of trees to improve the conservation of species. Environ Sci Pollut Res 29(10):14780–14790
Bista SK, Nepal S, Colineau N, Paris C (2012) Using gamification in an online community. In: 8th international conference on collaborative computing: networking, applications and Worksharing (CollaborateCom). IEEE, pp 611–618
Bista SK, Nepal S, Paris C (2012) Engagement and cooperation in social networks: do benefits and rewards help? In: 2012 IEEE 11th international conference on trust, security and privacy in computing and communications. IEEE, pp 1405–1410
Blohm, I. J.M. Leimeister, Design of IT-based enhancing services for motivational support and behavioral change. 2013.
Borzsony S, Kossmann D, Stocker K (2001) The skyline operator. In: Proceedings 17th international conference on data engineering. IEEE
Bradley RA, Terry ME (1952) Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39(3/4):324–345
Busa-Fekete R, Szorenyi B, Cheng W, Weng P, Hüllermeier E (2013) Top-k selection based on adaptive sampling of noisy preferences. In: International conference on machine learning. PMLR, pp 1094–1102
Cramer H, Rost M, Holmquist LE (2011) Performing a check-in: emerging practices, norms and conflicts; in location sharing using foursquare. In: Proceedings of the 13th international conference on human computer interaction with mobile devices and services, pp 57–66
Csikszentmihalyi M, Csikzentmihaly M (1990) Flow: The psychology of optimal experience. Vol. 1990. Harper & Row New York
Davidson SB, Khanna S, Milo T, Roy S (2013) Using the crowd for top-k and group-by queries. In: Proceedings of the 16th international conference on database theory, pp 225–236
Deci EL, Koestner R, Ryan RM (1999) A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychol Bull 125(6):627
Demartini G, Difallah DE, Cudré-Mauroux P (2012) Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In: Proceedings of the 21st international conference on world wide web, pp 469–478
Deterding S (2011) Situated motivational affordances of game elements: a conceptual model. In Gamification: using game design elements in non-gaming contexts, a workshop at CHI 2011 10, 1979742.1979575
Elo AE (1978) The rating of chessplayers, past and present. Arco Pub
Eriksson B (2013) Learning to top-k search using pairwise comparisons. In: Artificial intelligence and statistics. PMLR, pp 265–273
Fan J et al (2014) A hybrid machine-crowdsourcing system for matching web tables. In: 2014 IEEE 30th international conference on data engineering. IEEE
Fang Y et al (2016) Effective result inference for context-sensitive tasks in crowdsourcing. In: International conference on database systems for advanced applications. Springer
Feige U et al (1994) Computing with noisy information. SIAM J Comput 23(5):1001–1018
Frith JH (2012) Constructing location, one check-in at a time: examining the practices of foursquare users. North Carolina State University
Gåsland MM (2011) Game mechanic based e-learning: A case study (Master thesis, Instituttor datateknikk og informasjonsvitenskap)
Ghadiyaram D, Bovik AC (2015) Massive online crowdsourced study of subjective and objective picture quality. IEEE Trans Image Process 25(1):372–387
Ghallab M, Nau D, Traverso P (2004) Automated planning: theory and practice. Elsevier
Gnauk B, Dannecker L, Hahmann M (2012) Leveraging gamification in demand dispatch systems. In: Proceedings of the 2012 joint EDBT/ICDT workshops, pp 103–110
Gokhale C, Das S, Doan A, Naughton JF, Rampalli N, Shavlik J, Zhu X (2014) Corleone: hands-off crowdsourcing for entity matching. In: Proceedings of the 2014 ACM SIGMOD international conference on management of data, pp 601–612
Grote A, Schaadt NS, Forestier G, Wemmert C, Feuerhake F (2018) Crowdsourcing of histological image labeling and object delineation by medical students. IEEE Trans Med Imaging 38(5):1284–1294
Groz B, Milo T (2015) Skyline queries with noisy comparisons. In: Proceedings of the 34th ACM sigmod-sigact-sigai symposium on principles of database systems, pp 185–198
Gruenheid A et al (2012) Crowdsourcing entity resolution: when is a= b? Technical Report/ETH Zurich. Department of Computer Science, p 785
Guin TD-L et al (2012) Myths and realities of respondent engagement in online surveys. Int J Mark Res 54(5):613–633
Guo S, Parameswaran A, Garcia-Molina H (2012) So who won? Dynamic max discovery with the crowd. In: Proceedings of the 2012 ACM SIGMOD international conference on management of data, pp 385–396
Hajarian M, Bastanfard A, Mohammadzadeh J, Khalilian M (2019) A personalized gamification method for increasing user engagement in social networks. Soc Netw Anal Min 9(1):1–14
Hajarian M, Bastanfard A, Mohammadzadeh J, Khalilian M (2019) SNEFL: social network explicit fuzzy like dataset and its application for Incel detection. Multimed Tools Appl 78:33457–33486. https://doi.org/10.1007/s11042-019-08057-3
Heikinheimo H, Ukkonen A (2013) The crowd-median algorithm. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol 1, pp 69–77
Herbrich R, Minka T, Graepel T (2006) TrueSkill™: a Bayesian skill rating system. Adv Neural Inf Proces Syst 19
Hu L, Qian Y, Chen J, Shi X, Zhang J, Mao S (2017) Photo crowdsourcing based privacy-protected healthcare. IEEE Trans Sustain Comput 4(2):168–177
Jeffery SR, Franklin MJ, Halevy AY (2008) Pay-as-you-go user feedback for dataspace systems. In: Proceedings of the 2008 ACM SIGMOD international conference on management of data, pp 847–860
Jiang X et al (2011) Statistical ranking and combinatorial Hodge theory. Math Program 127(1):203–244
Kaplan H et al (2013) Answering planning queries with the crowd. Proc VLDB Endow 6(9):697–708
Kapp KM (2012) The gamification of learning and instruction: game-based methods and strategies for training and education. John Wiley & Sons
Khan AR, Garcia-Molina H (2014) Hybrid strategies for finding the max with the crowd: technical report. Stanford InfoLab
Lee H, Doh YY (2012) A study on the relationship between educational achievement and emotional engagement in a gameful interface for video lecture systems. In: 2012 international symposium on ubiquitous virtual reality. IEEE
Li G, Deng D, Wang J, Feng J (2011) Pass-join: a partition-based method for similarity joins. arXiv preprint arXiv:11117171
Li G, Zheng Y, Fan J, Wang J, Cheng R (2017) Crowdsourced data management: overview and challenges. In: Proceedings of the 2017 ACM international conference on Management of Data, pp 1711–1716
Liu Y, Alexandrova T, Nakajima T (2011) Gamifying intelligent environments. In: Proceedings of the 2011 international ACM workshop on ubiquitous meta user interfaces, pp 7–12
Lofi C, El Maarry K, Balke WT (2013) Skyline queries in crowd-enabled databases. In: Proceedings of the 16th international conference on extending database technology, pp 465–476
Lofi C, El Maarry K, Balke W-T (2013) Skyline queries over incomplete data-error models for focused crowd-sourcing. In: International conference on conceptual modeling. Springer
Malone TW (1981) Toward a theory of intrinsically motivating instruction. Cogn Sci 5(4):333–369
Malone TW (1982) Heuristics for designing enjoyable user interfaces: lessons from computer games. In: Proceedings of the 1982 conference on human factors in computing systems, pp 63–68
Marcus A, Wu E, Karger D, Madden S, Miller R (2011) Human-powered sorts and joins. arXiv preprint arXiv:11096881
Marcus A, Karger D, Madden S, Miller R, Oh S (2012) Counting with the crowd. Proc VLDB Endow 6(2):109–120
Mason AD, Michalakidis G, Krause PJ (2012) Tiger nation: empowering citizen scientists. In: 2012 6th IEEE international conference on digital ecosystems and technologies (DEST). IEEE
Massung E, Coyle D, Cater KF, Jay M, Preist C (2013) Using crowdsourcing to support pro-environmental community activism. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 371–380
McDaniel R, Lindgren R, Friskics J (2012) Using badges for shaping interactions in online learning environments. In: 2012 IEEE international professional communication conference. IEEE
McGonigal J, Broken RI (2011) Why games make us better and how they can change the world. The Penguin Group
Mekler ED et al (2013) Do points, levels and leaderboards harm intrinsic motivation? An empirical analysis of common gamification elements. In: Proceedings of the first international conference on gameful design, research, and applications
Minoofam SA, Bastanfard A (2010) Square Kufic pattern formation by asynchronous cellular automata. In: International conference on cellular automata. Springer, Berlin, Heidelberg, pp 79–82
MirMashhouri A, Bastanfard A, Amirkhani D (2022) Collecting a database for emotional responses to simple and patterned two-color images. Multimed Tools Appl 81(13):18935–18953
Morneau RA, Van Herreweghe WG, Little JW, Lefebvre DB (2012) Energy company perspective on virtual worlds/3-D immersive environments. In: SPE intelligent energy international. OnePetro
Movahedi Z, Bastanfard A (2021) Toward competitive multi-agents in polo game based on reinforcement learning. Multimed Tools Appl 80:26773–26793. https://doi.org/10.1007/s11042-021-10968-z
Mozafari B et al (2014) Scaling up crowd-sourcing to very large datasets: a case for active learning. Proc VLDB Endow 8(2):125–136
Negahban S, Oh S, Shah D (2012) Iterative ranking from pair-wise comparisons. In: Advances in neural information processing systems
Nguyen QV, Nguyen TT, Miklós Z, Aberer K, Gal A, Weidlich M (2014) Pay-as-you-go reconciliation in schema matching networks. In: 2014 IEEE 30th international conference on data engineering. IEEE, pp 220–231
Nicholson S (2012) A user-centered theoretical framework for meaningful gamification. In: Paper presented at the games+ learning+ society. 8.0, Madison, USA
Parameswaran A, Sarma AD, Garcia-Molina H, Polyzotis N, Widom J (2011) Human-assisted graph search: it's okay to ask questions. arXiv preprint arXiv:11033102
Parameswaran AG, Garcia-Molina H, Park H, Polyzotis N, Ramesh A, Widom J (2012) Crowdscreen: algorithms for filtering data with humans. In: Proceedings of the 2012 ACM SIGMOD international conference on Management of Data, pp 361–372
Park H, Widom J (2014) Crowdfill: collecting structured data from the crowd. In: Proceedings of the 2014 ACM SIGMOD international conference on management of data, pp 577–588
Passos EB et al (2011) Turning real-world software development into a game. In: 2011 Brazilian symposium on games and digital entertainment. IEEE
Pfeiffer T, Gao X, Chen Y, Mao A, Rand D (2012) Adaptive polling for information aggregation. In: Proceedings of the AAAI conference on artificial intelligence vol 26, no 1, pp 122–128
Pomerol J-C, Barba-Romero S (2012) Multicriterion decision in management: principles and practice. Vol. 25. Springer Science & Business Media
Rahm E, Bernstein PA (2001) A survey of approaches to automatic schema matching. VLDB J 10(4):334–350
Rudinac S, Larson M, Hanjalic A (2013) Learning crowdsourced user preferences for visual summarization of image collections. IEEE Trans Multimed 15(6):1231–1243
Ryan RM, Deci EL (2000) Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 55(1):68
Sakamoto M, Nakajima T, Alexandrova T (2012) Value-based design for gamifying daily activities. In: International conference on entertainment computing. Springer
Sarawagi S, Bhamidipaty A (2002) Interactive deduplication using active learning. In: Proceedings of the eighth ACM SIGKDD international conference on knowledge discovery and data mining, pp 269–278
Sarma AD et al (2014) Crowd-powered find algorithms. In: 2014 IEEE 30th international conference on data engineering. IEEE
Sarma AD et al (2015) Jellybean: crowd-powered image counting algorithms. HCOMP
Seaborn K, Fels DI (2015) Gamification in theory and action: a survey. Int J Human-Comput Stud 74:14–31
Skinner BF (1965) Science and human behavior. Simon and Schuster
Takbiri Y, Bastanfard A, Amini A (2023) A gamified approach for improving the learning performance of K-6 students using Easter eggs. Multimed Tools Appl 82:20683–20701. https://doi.org/10.1007/s11042-023-14356-7
Thom J, Millen D, DiMicco J (2012) Removing gamification from an enterprise SNS. In: Proceedings of the ACM 2012 conference on computer supported cooperative work, pp 1067–1070
Trushkowsky B et al (2013) Crowdsourced enumeration queries. In: 2013 IEEE 29th international conference on data engineering (ICDE). IEEE
Tseng W-Y, Chen K-H, Huang J-W (2019) Crowdsourced object-labeling based on a game-based mobile application. Multimed Tools Appl 78(13):18137–18168
Venetis P, Garcia-Molina H, Huang K, Polyzotis N (2012) Max algorithms in crowdsourcing environments. In: Proceedings of the 21st international conference on world wide web, pp 989–998
Verroios V, Garcia-Molina H (2015) Entity resolution with crowd errors. In: 2015 IEEE 31st international conference on data engineering. IEEE
Vesdapunt N, Bellare K, Dalvi N (2014) Crowdsourcing algorithms for entity resolution. Proc VLDB Endow 7(12):1071–1082
Wang J, Kraska T, Franklin MJ, Feng J (2012) Crowder: crowdsourcing entity resolution. arXiv preprint arXiv:12081927
Wang J, Li G, Feng J (2012) Can we beat the prefix filtering? An adaptive framework for similarity join and search. In: Proceedings of the 2012 ACM SIGMOD international conference on management of data, pp 85–96
Wang J, Li G, Kraska T, Franklin MJ, Feng J (2013) Leveraging transitive relations for crowdsourced joins. In: Proceedings of the 2013 ACM SIGMOD international conference on Management of Data, pp 229–240
Wang S, Xiao X, Lee CH (2015) Crowd-based deduplication: an adaptive approach. In: Proceedings of the 2015 ACM SIGMOD international conference on Management of Data, pp 1263–1277
Wauthier F, Jordan M, Jojic N (2013) Efficient ranking from pairwise comparisons. In: International conference on machine learning. PMLR, pp 109–117
Whang SE, McAuley J, Garcia-Molina H (2012) Compare me maybe: crowd entity resolution interfaces. Stanford InfoLab
Whang SE, Lofgren P, Garcia-Molina H (2013) Question selection for crowd entity resolution. Proc VLDB Endow 6(6):349–360
Witt M, Scheiner CW, Robra-Bissantz S (2011) Gamification of online idea competitions: insights from an explorative case. In: GI-Jahrestagung, p 392
Yan T, Kumar V, Ganesan D (2010) Crowdsearch: exploiting crowds for accurate real-time image search on mobile phones. In: Proceedings of the 8th international conference on Mobile systems, applications, and services, pp 77–90
Zhang CJ, Chen L, Jagadish HV, Cao CC (2013) Reducing uncertainty of schema matching via crowdsourcing. Proc VLDB Endow 6(9):757–768
Zhang X, Li G, Feng J (2016) Crowdsourced top-k algorithms: an experimental evaluation. Proc VLDB Endow 9(8):612–623
Zichermann G, Cunningham C (2011) Gamification by design: implementing game mechanics in web and mobile apps. O'Reilly Media, Inc.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interests
Authors declare that there is no Conflict of Interests or Competing of Interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Bastanfard, A., Shahabipour, M. & Amirkhani, D. Crowdsourcing of labeling image objects: an online gamification application for data collection. Multimed Tools Appl 83, 20827–20860 (2024). https://doi.org/10.1007/s11042-023-16325-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-16325-6