Wisdom of the Ancients: the web-comic-epigraph for my dissertation proposal, from XKCD
As of last monday, I have now successfully defended my dissertation proposal. In the context of my doctoral program, that means there is just one more hurdle to climb over to finish. I’m generally rather excited about the project, and would be thrilled to have more input and feedback on it (Designing Online Communities Proposal PDF). I would be happy for any and all comments on it in the comments of this post.
Designing Online Communities: How Designers, Developers, community Managers, And Software Structure Discourse And knowledge Production On The Web
Abstract: Discussion on the web is mediated through layers of software and protocols. As scholars increasingly turn to study communication, learning and knowledge production on the web, it is essential to look below the surface of interaction and consider how site administrators, programmers and designers create interfaces and enable functionality. The managers, administrators and designers of online communities can turn to more than 20 years of technical books for guidance on how to design and structure online communities toward particular objectives. Through analysis of this “how-to” literature, this dissertation intends to offer a point of entry into the discourse of design and configuration that plays an integral role in structuring how learning and knowledge are produced online. The project engages with and interprets “how-to” literature to help study software in a way that respects the tension that exists between the structural affordances of software with the dynamic and social nature of software as a component in social interaction.
What’s Next?
At some point in the next year I will likely defend a completed dissertation. Places do dissertations differently, in my program the idea is that what I just defended is actually the first three chapters of a five chapter dissertation. So, at this point I need to follow through on what I said I would do in my methods section (to create chapter 4, results) and then write up how it connects with the conceptual context section (to create chapter 5, conclusions). So I should be able to grind this out in relatively short order.
At this point, I think this project should be interesting enough to warrant a book proposal. So I’ll likely start exploring putting together a book proposal for it in the next year as well. With that in mind, any suggestions for who might be interested in receiving a proposal on this topic are welcome.
Still not the droid… By Stéfan: Our crowdsourcing conversation is upside down, much like how Calculon is holding these stormtroopers upside down.
Some fantastic work is going on in crowdsourcing the transcription of cultural heritage collections. After some recent thinking and conversation on these projects I want to more strongly and forcefully push a point about this work. This is the same line of thinking I started nearly a year ago in Meaningification and Crowdscafolding: Forget Badges. I’ve come to believe that conversations about the objective of this work, as broadly discussed, are exactly upside down. Transcripts and other data are great, but when done right, crowdsourcing projects are the best way of accomplishing the entire point of putting collections online. I think a lot of the people who work on these projects think this way but we are still in a situation where we need to justify this work by the product instead of justifying it by the process.
Getting transcriptions, or for that matter getting any kind of data or work is a by-product of something that is actually far more amazing than being able to better search through a collection. The process of crowdsourcing projects fulfills the mission of digital collections better than the resulting searches. That is, when someone sits down to transcribe a document they are actually better fulfilling the mission of the cultural heritage organization than anyone who simply stops by to flip through the pages.
Why are we putting cultural heritage collections online again?
There are a range of reasons that we put digital collections online. With that said the single most important reason to do so is to make history accessible and invite students, researchers, teachers, and anyone in the public to explore and connect with our past. Historians, Librarians, Archivists, and Curators who share digital collections and exhibits can measure their success toward this goal in how people use, reuse, explore and understand these objects.
In general, crowdsourcing transcription is first and foremost described as a means by which we can get better data to help better enable the kinds of use and reuse that we want people to make of our collections. In this respect, the general idea of crowdsourcing is described as an instrument for getting data that we can use to make collections more accessible. Don’t get me wrong, crowdsourcing does this. With that said it does so much more than this. In the process of developing these crowdsourcing projects we have stumbled into something far more exciting than speeding up or lowering the costs of document transcription. Far better than being an instrument for generating data that we can use to get our collections more used it is actually the single greatest advancement in getting people using and interacting with our collections. A few examples will help illustrate this.
Increased Use, Deeper Use, Crowdsourcing Civil War Diaries
Last year, the University of Iowa libraries crowdsourced the transcription of a set of civil war diaries. I had the distinct privilege of interviewing Nicole Saylor, the head of Digital Library Services, about the project. From any perspective the project was very successful. They got great transcriptions and they ended up attracting more donors to support their work.
The project also succeeded in dramatically increasing site traffic. As Nicole explained, “On June 9, 2011, we went from about 1000 daily hits to our digital library on a really good day to more than 70,000.” As great as all this is, as far as I’m concerned, the most valuable thing that happened is that when people come to transcribe the diaries they engage with the objects more deeply than they would have if transcription was not an option. Consider this quote from Nicole explaining how one particular transcriptionist interacted with the collection. It is worth quoting her at length;
The transcriptionists actually follow the story told in these manuscripts and often become invested in the story or motivated by the thought of furthering research by making these written texts accessible. One of our most engaged transcribers, a man from the north of England, has written us to say that the people in the diaries have become almost an extended part of his family. He gets caught up in their lives, and even mourns their deaths. He has enlisted one of his friends, who has a PhD in military history, to look for errors in the transcriptions already submitted. “You can do it when you want as long as you want, and you are, literally, making history,” he once wrote us. That kind of patron passion for a manuscript collection is a dream. Of the user feedback we’ve received, a few of my other favorites are: “This is one of the COOLEST and most historically interesting things I have seen since I first saw a dinosaur fossil and realized how big they actually were.” “I got hooked and did about 20. It’s getting easier the longer I transcribe for him because I’m understanding his handwriting and syntax better.” “Best thing ever. Will be my new guilty pleasure. That I don’t even need to feel that guilty about.
The transcriptions are great, they make the content more accessible, but as Nicole explains, “The connections we’ve made with users and their sustained interest in the collection is the most exciting and gratifying part.” This is exactly as it should be! The invitation of crowdsourcing and the event of the project are the most valuable and precious user experiences that a cultural heritage institution can offer its users. It is essential that the project offer meaningful work. These projects invite the public to leave a mark and help enhance the collections. With that said, if the goal is to get people to engage with collections and engage deeply with the past then the transcripts are actually a fantastic byproduct that is created by offering meaningful activities for the public to engage in.
Rationing out Transcription
Part of what prompted this post is a story that Ben Brumfield gave on crowdsourcing transcription at the recent Institute for Museum and Library Services Web Wise conference. It was a great talk, and when they get around to posting it online you should all go watch it. There was one particular moment in the talk that I thought was essential for this discussion.
At one point in a transcription project he noticed that one of his most valuable power users was slowing down on their transcriptions. The user had started to cut back significantly in the time they spent transcribing this particular set of manuscripts. Ben reached out to the user and asked about it. Interestingly, the user responded to explain that they had noticed that there weren’t as many scanned documents showing up that required transcription. For this user, the 2-3 hours they spent each day working on transcriptions was such an important experience, such an important part of their day, that they had decided to cut back and deny themselves some of that experience. The user needed to ration out that experience. It was such an important part of their day that they needed to make sure that it lasted.
At its best, crowdsourcing is not about getting someone to do work for you, it is about offering your users the opportunity to participate in public memory.
Crowdsourcing is better at Digital Collections than Displaying Digital Collections
What crowdsourcing does, that most digital collection platforms fail to do, is offers an opportunity for someone to do something more than consume information. When done well, crowdsourcing offers us an opportunity to provide meaningful ways for individuals to engage with and contribute to public memory. Far from being an instrument which enables us to ultimately better deliver content to end users, crowdsourcing is the best way to actually engage our users in the fundamental reason that these digital collections exist in the first place.
Meaningful Activity is the Apex of User Experience for Cultural Heritage Collections
When we adopt this mindset, the money spent on crowdsourcing projects in terms of designing and building systems, in terms of staff time to manage, etc. is not something that can be compared to the costs of having someone transcribe documents on mechanical turk. Think about it this way, the transcription of those documents is actually a precious resource, a precious bit of activity that would mean the world to someone. It isn’t that any task or obstacle for users to take on will do, for example, if you asked users to transcribe documents that could easily be OCRed the whole thing loses its meaning and purpose. It isn’t about sisyphean tasks, it is about providing meaningful ways for the public to enhance collections while more deeply engaging and exploring them.
Just as Ben’s user rationed out the transcription of those documents we might actually think about crowdsourcing experiences as one of the most precious things we can offer our users. Instead of simply offering them the ability to browse or poke around in digital collections we can invite them to participate. We are in a position to let our users engage in a personal way that is only for them at that moment. Instead of browsing through a collection they literally become a part of our historical record.
The Important Difference between Exploitation-ware and Software for the Soul
Slide from Ruling the World
As a bit of a coda, what is tricky here is that there is (strangely) an important and but somewhat subtle line between exploiting people and giving people the most valuable kinds of experience that we can offer for digital collections. The trick is that gamification is (for the most part) bullshit. You can trick people into doing things with gimmicks, but when you do so you frequently betray their trust and can ruin the innately enjoyable nature of being a part of something that matters to you, in our case, the way that users could deeply interact with and explore the past via your online collections. What sucks about what has happened in the idea of gamification is that it is about the least interesting parts of games. It’s about leaderboards and badges. As Sebastian Deterding has explained, many times and many ways, the best part of games, the things that we should actually try to emulate in a gamification that attempts to be more than pointsification or exploitationware are the part of games that let us participate in something bigger. It is the part of games that invites us to playfully take on big challenges. Be wary of anyone who tries to suggest we should trick people or entice them into this work. We can offer users an opportunity to deeply explore, connect with and contribute to public memory and we can’t let anything get in the way of that.
In Please Write it Down: Design and Research in the Digital Humanities I suggested that there are some valuable ways of thinking about the connections between building/designing and creating knowledge and scholarship. In particular, I suggested that those interested in learning through building in the digital humanities might find some value in work in educational research over the last decade which has tried to define what exactly what a design based research methodology might look like.
This is the first post, in what I imagine might be an ongoing line of thought here, to try to put ideas from design based research in conversation with the digital humanities. As a point of entry, I am going to walk through one emerging genre of writing in design based research, the design narrative. Before getting there, however, I would briefly pause to note that the journal this piece appeared in, Educational Technology Research and Development, is itself an interesting note to the digital humanities. I for one, would love to see a journal in the digital humanities similarly situated as a place for sharing and disseminating R&D knowledge.
The Case of Environmental Detectives
In Environmental Detectives: The Development of an Augmented Reality Platform for Environmental Simulations Eric Klopfer and Kurt Squire offer a summative and reflective report on their work developing the augmented reality game Environmental Detectives. The paper makes some valuable suggestions for how we might better design augmented reality games, but I think its primary strength is as an example of a particularly novel and useful genre of design based research report.
Brenda Bannan-Ritland’s article, The role of design in research: The integrative learning design framework offers a robust framework for thinking through how the design process and the research process can fit together. See her diagram below (don’t get lost in the details). The intellectual work that diagram and her approach offers os to illustrate what happens if you mush together the steps in an array of design processes and research approaches. The diagram illustrates how the features of product development, research design, and user centered design can leaf together.
If you look a the top part of the diagram carefully you will notice that practically every step in this process has an arrow that points over to the publish results box. This is a key concept here, the idea behind design based research is not that the design process is itself a research method, but that throughout the design process there are a series of publishable results and lessons learned that emerge which warrant being refined, shared and communicated. Squire and Klopfer’s article is a great example of the kind of piece one would want to write as a summative result of an extended design research process.
Design Narrative as a Genre of Design Based Research Article
Design based research can generate publishable results in any particular research tradition. You can find interviews, ethnographic approaches, micro ethnographic approaches, case studies, randomized clinical trials, and methods from usability studies like eye tracking used at different points in the design and development process. In short, there are any number of ways to use existing research methods approaches to reflect on and report out results of research in the process of informing design. Part of what is particularly interesting about Klopfer and Squire’s paper is that it represents a somewhat novel mode of research writing, the design narrative.
Drawing from Hoadley’s 2002 piece, Creating context: Design-based research in creating and understanding CSCL, Klopfer and Squire offer a reflective narrative account of their work designing, developing and researching the Environmental Detectives game. Unlike other papers they published, which might report parts of this research in terms of a case study, or the pre-post test scores or the results of a particular evaluative test of the game’s outcomes, this summitive piece serves to reflect on the design process and offer an account of the context and lessons learned in the course of the design process. It is worth reporting on actual structure of the piece.
Review of literature that informed the design: After explaining background on the idea of design narrative Klopfer and Squire offer an account of both the extent literature on augmented reality games and a review of the existing games projects that they looked to which informed their design. This serves to provide the conceptual context that they began from, it sets the reader up to understand exactly where the project started from while also providing information on what theory and knowledge at the time of the projects start looked like.
Retrospective and Reflective Design Narrative: The bulk of the paper then reports out on each phase of their design process. In their particular case they describe six phases of their research, brainstorming, designing the first instantiation, developing a first generation prototype, classroom field trials, classroom implementations, expanding to new contexts, and a sixth phase in which they added customized dynamic events to the game. It is not necessary to go into the details of each section for this review. What matters is to stress that each section begins by explaining how they went about their work in the given phase and reports a bit on what they learned in that phase. What is essential in this approach is that each section explains what worked and didn’t work in any given phase and how exactly Klopfer decided to remedy their approach and design to respond to problems.
As is generally the case with qualitative research, the moments when things don’t go according to plan and exactly how we make sense and work through those moments are generally the most valuable parts of the process. The value in this kind of retrospective account is two-fold. It provides a context for understanding why the game they made does what it does, but more importantly, the design narrative’s primary value is as a guide to other designers on what parts of the design process were particularly valuable. This kind of narrative helps us to refine our ideas not only about this particular design situation, but more broadly about how we can refine our own design practices.
Conclusions and Implications from Reflection:After reporting the design narrative the paper presents a set of technological and pedagogical implications. In much the way that the discussion section and conclusion sections of research reports function, this section attempts to suss out and distill the lessons learned from the work. In their case, they present a range of specific implications for the design of augmented reality games that emerged from their design approach.
The Value of Design Narratives
If you read through their references, you can see that they have published about this work on a few previous occasions. It is not that they are double dipping on publications, instead those other publications report results from subsets of this project, some of the earlier findings, or any of the points in the design process that resulted in interesting findings. This paper is really a summitive report, retracing the design narrative of the entire project.
I see the value of this particular design narrative approach as having two primary values, two values that I think are particularly useful to the still emerging world of the digital humanities. Composing these narratives serves an internal value to designers as part of reflective practice. Sharing these narratives makes the kinds essential tacit knowledge that comes about as part of doing design accessible to others.
Reflective Practice is Best Practice: If you can hold yourself to some sound practices for documenting the stages in your design process (the ideas that you had, how you went about implementing and revising them, and the results), you are in a good position to use that documentation to reflect on your practice. In this sense, the design narrative, the retrospective account of what you did, why you did it, what you learned is an essential piece of doing reflective design practice. When you go back and think through your own process you are not simply reporting on what you learned you are actually making sense out of your trajectory and coming to understand what it is that you actually learned. Like much of qualitative and hermeneutic research, the process of writing is not a process of transmission of knowledge but of the discovery of knowledge. Writing a design narrative is the process by which we come to know and learn from our work.
Making Tacit Practical Design Knowledge Explicit and Available: It is essential that the knowledge developed in the design process is documented and shared. While the individual studies that come out of a design research process provide evidence of the value, or of particular lessons learned in part of a design project, they leave a considerable amount of the bigger picture knowledge off the table. Quite frankly, much of the most essential parts of design are not about explaining that something works, if someone wants to get into design they need access to the deeply pragmatic, heuristic driven, knowledge that develops on over time in the process of design. The design narrative is an essential medium for capturing and disseminating this kind of tacit knowledge.
In short, I would suggest that this particular piece of scholarship serves as a great example of the value of reporting design narratives and an exemplar for others to use as a model for composing their own design narratives.
I am excited to report that i have finished gathering data from my RPG Maker VX community survey and am well on the way toward finishing interviews with a subset of the respondents. For more information about this project see my previous post. At this point I thought I would share a cursory overview of some of the interesting preliminary survey findings. For those survey research junkies out there I should make clear that this survey is part of a qualitative research project. It was developed strictly as a means to gather descriptive data to provide a broader context for analyzing discussions on the site and interviews with community members. For details on the survey methods and response rates jump down to the last section of the post.
This is a community of young people:
Most of the community members are between the ages of 16 and 24, and of those most are between 18 and 22. As I will document through analysis of discussions and interviews the members of this community are developing sophisticated practices for taking and giving criticism as well as working collaboratively. In this space young people are both the teachers and the learners. While critics frequently lament students motivation and hard work it is clear that this communal space is providing a place for young people to cut their teeth as artists, designers, critics, and producers of digital media.
This is a global community:
45% of the sample reports living in the United States. The rest of the group is spread across Europe, South America, and Asia. A majority of community members reported English as their native language (64%) the remaining 36% represent a smattering of other languages, including Spanish, French and Japanese.
These young people are not just playing around
Most of those surveyed have been involved for more than a year and report spending a considerable amount of time each week on writing, design, and art projects for their games. Group members show significantly different amounts of time spent on different parts of projects. Some spend the bulk of their time writing others spend the bulk of their time creating game artwork.
This is a place where young people are first exposed to programing
RPG Maker VX includes a scripting system, Ruby Game Scripting System, which extends the Ruby Computer Programing language. Nearly all (83%) of the community members report that they have used the games scripting system, and 35% of the respondents reported that working with RPG Maker was their first experience with computer code.
These young people strongly identify with hits from the “RPG Cannon”
When asked about their favorite video games participants cited a mixture of current and “classic” games. To get a quick sense of the kinds of games which appeared most frequently, scan the word frequency chart I generated with Wordle bellow. This is just the raw frequency of individual words, but it is easy to see the trends which emerge around some of the most famous super Nintendo role playing games and franchises. The Final Fantasy series, Chrono Trigger, Legend of Zelda, Secret of Mana, Breath of Fire, all appear prominently on respondents lists of favorite games. It is worth keeping in mind that many of these games were original released around or before the majority of these community members were born.
As RPG Maker allows players to make these kinds of games, it makes sense that these kinds of games are also part of their list of favorites. While some might think of the kinds of graphics and formats for games which RPG Maker creates are a weakness of the software, there is good reason to believe that these gamers love for SNES RPGs connects them to a kind of game and experience which they find deeply engaging.
Surveying a community without boundaries:
It is best to develop a survey with a specific population in mind. Part of the difficulty of surveying a diffuse community like the online community associated with the RPG Maker VX site is in defining the boundaries of that community. The site has over 40k members, and during any given visit to the site nearly twice as many non-members are viewing the discussion boards as members. It would be impossible to accurately sample non-members who visit the site, there is no trace of their visits. With that said, instead of setting upfront criteria for who counted as a community member (based on post count, or number of visits, or the length of time they have been involved in the community) I decided to create a sample of individuals who had logged in within the last week. While this will inherently sample more frequently involved users it would also include a sizable segment of other more infrequent visitors. To sample a cross-section of community members in a given week I used the sites member search system to sift through the total number of folks who had logged in over the proceeding week, in this case it was 1740 members. From there I sampled a randomly selected group of 160 members. I have received 85 responses, giving me a respectable 53% response rate.
Limitations with the sample
In accordance with George Mason’s human subjects review boards requirements I did not contact anyone who either did not list their age or listed their age as less than 18. In the process of creating the sample I rejected individuals that fell into these categories. Most individuals did list their age and only 10 of the randomly selected members listed themselves a under age 18.
While the response rage is acceptable, I will note two reasons for why members may not have responded. The community message system has used as a mass emailing system for bots. In many cases potential respondents required me to offer a range if kinds of evidence to demonstrate that I was in fact a human before they would click the link to take the survey. Aside from fear of bots, in two cases I heard from individuals who were uncomfortable taking a survey in English because it was not their native language. This suggests that the survey may not fully capture the international character of the community.
A while back, I wrote a post about a very neat piece of software called RPG Maker. I never really got to building a game with it, but I have become fascinated with the community that has come together around the software. This post begins a series of entries about a research project I have started to explore how this community is scaffolding game players into game makers. In this post I will briefly outline some of the interesting. The image below shows an screen shot from Prelude to Identity, a well received game in the community.
Image from popular RPG Maker Game Prelude of Identity
Daily Composition on the RPG Maker VX boards
Everyday several hundred members of the RPG Maker VX Community read through a new set of project development posts on the community’s forums. In each of these posts amateur game designers, primarily between the ages of 18 and 24, share 500-1000 word game proposals for community critique. These posts include elements of traditional composition, like the proposed games setting, characters, and storyline. They also include elements unique to games as new media, like the proposed game’s mechanics, artwork, and audio. Over the next few days, each of these proposed projects receives extensive feedback from the community. After substantial revision, refinement, development, and continued engagement with the community, some of the community members’ complete their games and share them with the group.
For an example of some of the thoughtful kinds of design and composition that goes into creating game maps see Mr. Moo‘s video of a follow up game Crescendo of Identity.
Short Outline of Project Methods
I have received permission from my schools human subjects review board to explore the community through a diverse set of methods. I have started conducting a survey to get a sense of community members activity, behaviors, and participation. In a few weeks I will start and a set of interviews with community members to get a deeper sense of how members understand their participation and explore some of the various roles they are taking on. My goal is to then use the survey and interviews to help add texture and context to a detailed analysis of community interactions as preserved on the message boards.
I have already started to get back survey results. I am excited to share some of the preliminary information here in the next few weeks.
This week in Clio Wired: Creating History With New Media each of my classmatees has been diligently working on composing a design rationale for each of our projects. Below is my rationalization. You can also view it as this PDF.
Related to this I thought folks might be interested in the slides for the presentation I gave on Playing History at the American Association of History and Computing’s conference over the weekend.
