Discovering the principles of crowd work - Wikipedia and beyond

Systems such as Wikipedia or Linux involve hundreds of thousands of contributors with different knowledge, views, and motivations directly interacting with and building on each others’ work. As such these systems violate simple “wisdom of crowds” assumptions of independent judgments and automatic aggregation. To understand how these systems function, we draw on social science theories about how people organize in the offline world to find effective algorithms for crowd work. For example, using parallel computing to analyze the entire history of Wikipedia, we discovered that adding more contributors to an article on average actually decreases article quality, rather than improving it as we (and many others) expected. To address these issues we have drawn from concepts in organizational behavior such as group identification, motivation, and coordination to identify social algorithms that lead to effectively combining the efforts of many contributors. Our models have been validated in thousands of online communities, and extended to understand domains such as massively collaborative scientific discovery (e.g., the Polymath Projects) and crowdsourcing task markets (e.g., Amazon’s Mechanical Turk).

Improving crowd sensemaking

Building on these principles, we are developing tools to increase the ability of the crowd to understand each others’ work and effectively contribute. As the size and complexity of crowd-built artifacts grow, so do challenges to workers’ ability to effectively contribute to them. For example, despite early exponential growth, Wikipedia has been losing members, due in part to the difficultly of adding to articles that thousands have edited before. We are developing intelligent interfaces for Wikipedia combining visualization and machine learning that enable users to quickly understand thousands of differing viewpoints; predict conflict across millions of pages; facilitate interactions between editors; and help newcomers understand how to improve their contributions. We are now partnering with the Wikimedia Foundation to create a novel research methodology for Wikipedia that enables researchers to design, deploy and field test new interfaces without interfering with the ongoing function of the live site. Building on this experience, we are also developing new distributed sensemaking systems in the context of scientific discovery to enable greater interdisciplinary insights (cognitiveatlas.org).

Enabling new crowd abilities - Crowdsourcing complex work

We are developing ways to accomplish complex, interdependent and creative tasks in crowdsourcing markets such as Amazon’s Mechanical Turk, which have previously been limited to simple, independent, and objective tasks. Some examples include using the crowd for article writing, poetry translation, and science journalism.

Quality control for crowdsourcing
Enabling more complex and creative work incurs new challenges, for example maintaining quality control for subjective tasks which may have many valid answers or that are difficult to evaluate. Our initial work in this area developed new task design methods based on the idea of making believable invalid answers more effortful than good faith responses, which reduced cheating by a factor of 20 and more than doubled time-on-task. More recently , we have developed machine learning systems that can predict crowd workers’ quality based only on how they do the work (e.g., scrolling, mouse movements) without knowledge of their actual output, enabling quality control for subjective tasks where traditional approaches such as gold standards or worker agreement are impossible.

Coordination and crowdsourcing
Another key challenge is coordinating many small micro-contributions to create a complex, interdependent artifact, such as writing an article or coding software. For example, imagine trying to write a coherent article with a hundred contributors in which each only provides a few minutes of work. We have developed systems inspired by distributed computing models such as Google’s MapReduce which manage the dependencies between contributors to produce crowd-written articles rated better than individually-produced ones and as good as Wikipedia articles. In collaboration with professional journalists, we are now experimenting with crowdsourcing the science journalism process (e.g., mybossisarobot.com), with the dual benefits of involving citizens as participants in the research process and increasing the amount of scientific information consumable by the general public.

Augmenting scientific discovery and insight

Collaborative ontology building
The rapid growth of medical and neuroscientific data require structured ontologies which can help people find and make use of large data stores. However, traditional bioinformatics approaches to ontology building are insufficient for many domains where rapid advances mean that knowledge is both dynamic and distributed among many sources. Other drawbacks include high participation costs, limited longevity, and adoption issues. In contrast, we are developing a system for ontology building that combines the bazaar-like aspects of emerging collaborative online paradigms such as Wikipedia with cathedral-like support for structured data. The goal of this research is to enable researchers to make robust cross-disciplinary inferences (such as tying together cognitive, behavioral, and neuroimaging data) in a system that brings together knowledge from many researchers while maintaining low participation costs. A prototype of the system can be found at cognitiveatlas.org.

Literature discovery
The recent rapid advancement of science has been accompanied by an explosion in the accompanying scientific literature. Nowadays to make a significant advance it seems necessary to study a single area of science for many years; yet some of the most important discoveries are made by bridging between fields. In order to help people more quickly understand a new field we are developing tools that combine visualization and large scale data mining to help researchers build mental models of information spaces.

Collaborative knowledge mapping
Increasingly specialized knowledge is required to comprehend our complex world. Often, making sense of consumer choice, scientific advancements, and social issues involves frequently changing information, rapid learning, and competing viewpoints. This project aims to design a web-based knowledge building environment for the collaborative creation of knowledge maps. Knowledge maps are interactive visualizations that capture and convey the information, argumentation, and perspective that underlies people's informal theories about the empirical world.

Publications

Sorry, this is rather out of date right now. See CV for the most up to date list of publication

Balakrishnan, A., Fussell, S., Kiesler, S., Kittur, A. (2010). Pitfalls of Information Access with Visualizations in Remote Collaborative Analysis. CSCW 2010: Proceedings of the ACM Conference on Computer-Supported Cooperative Work. New York: ACM Press.

Kittur, A., & Kraut, R. E. (2010). Beyond Wikipedia: Conflict and coordination in online production groups. CSCW 2010: Proceedings of the ACM Conference on Computer-Supported Cooperative Work. New York: ACM Press.

Miller, E., Seppa, C., Kittur, A., Sabb, F., Poldrack, R. A. (2010). The Cognitive Atlas: Employing Interaction Design Processes to Facilitate Collaborative Ontology Creation. WWW 2010, Workshop on the Future of the Web for Collaborative Science.

Halfaker, A., Kittur, A., Kraut, R. E., Riedl, J. (2009). Quality, Experience and Ownership in WikiWork. WikiSym 2009: The International Symposium on Wikis and Open Collaboration. New York: ACM Press.

Kittur, A., Pendleton, B., Kraut, R. E. (2009). Herding the Cats: The Influence of Groups in Coordinating Peer Production. WikiSym 2009: The International Symposium on Wikis and Open Collaboration. New York: ACM Press.

Kittur, A., Lee, B., Kraut, R. E. (2009). Coordination in Collective Intelligence: The Role of Team Structure and Task Interdependence. CHI 2009: Proceedings of the ACM Conference on Human-factors in Computing Systems. New York: ACM Press.

Kittur, A., Suh, B., Chi, E. (2009). What's in Wikipedia? Mapping Topics and Conflict Using Collaboratively Annotated Category Links. CHI 2009: Proceedings of the ACM Conference on Human-factors in Computing Systems. New York: ACM Press.

Kittur, A., Chau, D., Hong, J., Faloutsos, C. (2009). Supporting Ad Hoc Sensemaking: Integrating Cognitive, HCI, and Data Mining Approaches. CHI 2009, Workshop on Sensemaking. Boston, MA.

Kittur, A., Kraut, R. E. (2008). Harnessing the Wisdom of Crowds in Wikipedia: Quality Through Coordination. CSCW 2008: Proceedings of the ACM Conference on Computer-Supported Cooperative Work. New York: ACM Press. (Best paper nomination)

Kittur, A., Chi, E., Suh, B. (2008). Can You Ever Trust a Wiki? Impacting Perceived Trustworthiness in Wikipedia. CSCW 2008: Proceedings of the ACM Conference on Computer-Supported Cooperative Work. New York: ACM Press.(Best paper award)

Kittur, A., Chi, E., Suh, B. (2008). Crowdsourcing User Studies With Mechanical Turk. CHI 2007: Proceedings of the ACM Conference on Human-factors in Computing Systems. New York: ACM Press

Suh, B., Chi, E., Kittur, A., Pendleton, B. (2008). Lifting the Veil: Improving Accountability and Social Transparency in Wikipedia with WikiDashboard.CHI 2007: Proceedings of the ACM Conference on Human-factors in Computing Systems. New York: ACM Press

Suh, B., Chi, E., Pendleton, B. A., Kittur, A. (2007). Us vs. Them: Understanding Social Dynamics in Wikipedia with Revert Graph Visualizations. VAST 2007: IEEE Symposium on Visual Analytics Science and Technology.

Kittur, A., Suh, B., Pendleton, B. A., Chi., E. (2007). He Says, She Says: Conflict and Coordination in Wikipedia. CHI 2007: Proceedings of the ACM Conference on Human-factors in Computing Systems. New York: ACM Press

Kittur, A., Chi, E., Pendleton, B. A., Suh, B., Mytkowicz, T. (2007). Power of the Few vs. Wisdom of the Crowd: Wikipedia and the Rise of the Bourgeoisie. Alt.CHI 2007

Kittur, A., Holyoak, K. J., & Hummel, J. E. (2006). Using Ideal Observers in Higher-order Human Category Learning. Proceedings of the Twenty Eighth Annual Meeting of the Cognitive Science Society. Vancouver, CA.

Kittur, A., Hummel, J. E., & Holyoak, K. J. (2006). Ideals Aren’t Always Typical: Dissociating Goodness-of-Exemplar From Typicality Judgments. Proceedings of the Twenty Eighth Annual Meeting of the Cognitive Science Society. Vancouver, CA.

Green, C., & Kittur, A. (2006). Retrieval-Induced Forgetting in a Multiple-Trace Memory Model. Proceedings of the Twenty Eighth Annual Meeting of the Cognitive Science Society. Vancouver, CA.

Kittur, A., Hummel, J.E., & Holyoak, K.J. (2004). Feature- vs. Relation-Defined Categories: Probab(alistic)ly Not the Same. Proceedings of the Twenty Sixth Annual Meeting of the Cognitive Science Society. Chicago, IL.

Green, C., & Kittur, A. (2004). A Multiple-Trace Memory Model Exhibiting Realistic Retrieval Dynamics. Proceedings of the Twenty Sixth Annual Meeting of the Cognitive Science Society. Chicago, IL.

Hummel, J.E., Holyoak, K.J., Green, C., Doumas, L.A.A., Devnich, D., Kittur, A., & Kalar, D.J. (2004). A Solution to the Binding Problem for Compositional Connectionism. In S.D. Levy & R. Gayler: Compositional Connectionism in Cognitive Science: Papers from the AAAI Fall Symposium. Menlo Park, CA: AAAI Press.

Lee, M. K., Borchelt, D. R., Kim, G., Thinakaran, G., Slunt, H. H., Ratovitski, T., Martin, L. J., Kittur, A., Gandy, S., Levey, A. I., Jenkins, N., Copeland, N., Price, D. L., & Sisodia, S. S. (1997). Hyperaccumulation of FAD-linked presenilin 1 variants in vivo. Nature Medicine, 3, 756-60.

copyright © 2006 niki kittur