Tag Archives: research

The Value of Design Narratives: The Case of Environmental Detectives

In Please Write it Down: Design and Research in the Digital Humanities I suggested that there are some valuable ways of thinking about the connections between building/designing and creating knowledge and scholarship.  In particular, I suggested that those interested in learning through building in the digital humanities might find some value in work in educational research over the last decade which has tried to define what exactly what a design based research methodology might look like.

This is the first post, in what I imagine might be an ongoing line of thought here, to try to put ideas from design based research in conversation with the digital humanities. As a point of entry, I am going to walk through one emerging genre of writing in design based research, the design narrative. Before getting there, however, I would briefly pause to note that the journal this piece appeared in, Educational Technology Research and Development, is itself an interesting note to the digital humanities. I for one, would love to see a journal in the digital humanities similarly situated as a place for sharing and disseminating R&D knowledge.

The Case of Environmental Detectives

In Environmental Detectives: The Development of an Augmented Reality Platform for Environmental Simulations Eric Klopfer and Kurt Squire offer a summative and reflective report on their work developing the augmented reality game Environmental Detectives. The paper makes some valuable suggestions for how we might better design augmented reality games, but I think its primary strength is as an example of a particularly novel and useful genre of design based research report. 

Brenda Bannan-Ritland’s article, The role of design in research: The integrative learning design framework offers a robust framework for thinking through how the design process and the research process can fit together. See her diagram below  (don’t get lost in the details). The intellectual work that diagram and her approach offers os to illustrate what happens if you mush together the steps in an array of design processes and research approaches. The diagram illustrates how the features of product development, research design, and user centered design can leaf together. 

If you look a the top part of the diagram carefully you will notice that practically every step in this process has an arrow that points over to the publish results box. This is a key concept here, the idea behind design based research is not that the design process is itself a research method, but that throughout the design process there are a series of publishable results and lessons learned that emerge which warrant being refined, shared and communicated. Squire and Klopfer’s article is a great example of the kind of piece one would want to write as a summative result of an extended design research process.

Design Narrative as a Genre of Design Based Research Article

Design based research can generate publishable results in any particular research tradition. You can find interviews, ethnographic approaches, micro ethnographic approaches,  case studies, randomized clinical trials, and methods from usability studies like eye tracking used at different points in the design and development process. In short, there are any number of ways to use existing research methods approaches to reflect on and report out results of research in the process of informing design. Part of what is particularly interesting about Klopfer and Squire’s paper is that it represents a somewhat novel mode of research writing, the design narrative.

Drawing from Hoadley’s 2002 piece, Creating context: Design-based research in creating and understanding CSCL, Klopfer and Squire offer a reflective narrative account of their work designing, developing and researching the Environmental Detectives game. Unlike other papers they published, which might report parts of this research in terms of a case study, or the pre-post test scores or the results of a particular evaluative test of the game’s outcomes, this summitive piece serves to reflect on the design process and offer an account of the context and lessons learned in the course of the design process. It is worth reporting on actual structure of the piece.

Review of literature that informed the design: After explaining background on the idea of design narrative Klopfer and Squire offer an account of both the extent literature on augmented reality games and a review of the existing games projects that they looked to which informed their design. This serves to provide the conceptual context that they began from, it sets the reader up to understand exactly where the project started from while also providing information on what theory and knowledge at the time of the projects start looked like.

Retrospective and Reflective Design NarrativeThe bulk of the paper then reports out on each phase of their design process. In their particular case they describe six phases of their research, brainstorming, designing the first instantiation, developing a first generation prototype, classroom field trials, classroom implementations, expanding to new contexts, and a sixth phase in which they added customized dynamic events to the game. It is not necessary to go into the details of each section for this review. What matters is to stress that each section begins by explaining how they went about their work in the given phase and reports a bit on what they learned in that phase. What is essential in this approach is that each section explains what worked and didn’t work in any given phase and how exactly Klopfer decided to remedy their approach and design to respond to problems.

As is generally the case with qualitative research, the moments when things don’t go according to plan and exactly how we make sense and work through those moments are generally the most valuable parts of the process. The value in this kind of retrospective account is two-fold. It provides a context for understanding why the game they made does what it does, but more importantly, the design narrative’s primary value is as a guide to other designers on what parts of the design process were particularly valuable. This kind of narrative helps us to refine our ideas not only about this particular design situation, but more broadly about how we can refine our own design practices.

Conclusions and Implications from Reflection: After reporting the design narrative the paper presents a set of technological and pedagogical implications. In much the way that the discussion section and conclusion sections of research reports function, this section attempts to suss out and distill the lessons learned from the work. In their case, they present a range of specific implications for the design of augmented reality games that emerged from their design approach.

The Value of Design Narratives

If you read through their references, you can see that they have published about this work on a few previous occasions. It is not that they are double dipping on publications, instead those other publications report results from subsets of this project, some of the earlier findings, or any of the points in the design process that resulted in interesting findings. This paper is really a summitive report, retracing the design narrative of the entire project.
I see the value of this particular design narrative approach as having two primary values, two values that I think are particularly useful to the still emerging world of the digital humanities. Composing these narratives serves an internal value to designers as part of reflective practice. Sharing these narratives makes the kinds essential tacit knowledge that comes about as part of doing design accessible to others.

Reflective Practice is Best Practice: If you can hold yourself to some sound practices for documenting the stages in your design process (the ideas that you had, how you went about implementing and revising them, and the results), you are in a good position to use that documentation to reflect on your practice. In this sense, the design narrative, the retrospective account of what you did, why you did it, what you learned  is an essential piece of doing reflective design practice. When you go back and think through your own process you are not simply reporting on what you learned you are actually making sense out of your trajectory and coming to understand what it is that you actually learned. Like much of qualitative and hermeneutic research, the process of writing is not a process of transmission of knowledge but of the discovery of knowledge. Writing a design narrative is the process by which we come to know and learn from our work.

Making Tacit Practical Design Knowledge Explicit and Available: It is essential that the knowledge developed in the design process is documented and shared. While the individual studies that come out of a design research process provide evidence of the value, or of particular lessons learned in part of a design project, they leave a considerable amount of the bigger picture knowledge off the table. Quite frankly, much of the most essential parts of design are not about explaining that something works, if someone wants to get into design they need access to the deeply pragmatic, heuristic driven, knowledge that develops on over time in the process of design. The design narrative is an essential medium for capturing and disseminating this kind of tacit knowledge.

In short, I would suggest that this particular piece of scholarship serves as a great example of the value of reporting design narratives and an exemplar for others to use as a model for composing their own design narratives.

Building The Forum: Or, Help Design Trevor’s Dissertation

In a 2009 interview with ReadWriteWeb, Mark O’sullivan, the lead developer of the open source web forum software Vanilla, was asked if Web forums are still relevant in the era of the real-time-web of Facebook and Twitter. His response offers an important point of entry for understanding the under-explored implications of web forum software, “Do a Google search for anything. How many of those search results are from discussion forums?” When asked if the prevalence of discussion forum threads in search results had to do with a querk in page-rank, Google’s system for evaluating the relevance of pages in any given search, he responded “It has to do with people having real discussions and giving real answers.” Yes, he is defending the relevance of his product in the face of a range of social networking platforms. Still, his response is verifiable. In my own experience, I find myself in the middle of a threaded web forum discussion on nearly a daily basis. We consult this collective knowledge-base on a regular basis, but generally know little about the structures, systems, ideologies and theories of end users involved in it’s creation.

In particular, we know little about the design decisions behind the forum software that enable dialog and discussion online. To be sure, the nature of Google’s search algorithm, page rank, and its approach to caching pages, each play important roles in this experience. However, the first step in the process involves the software tools that enable and shape our discourse online. The structure of the conversations that we engage in on online discussion boards, blogs, and other comment driven platforms are shaped (to some extent) by the Mark O’Sullivan’s of the world, the individuals that create, design, implement and hack on the software that makes the web a platform for community discussion, deliberation and dialog.

Beyond the knowledge-base, web forums are increasingly being explored as places where young and old alike are developing valuable skills and knowledge. Ito and others suggest that these kinds of spaces are where many young people are “learning to navigate esoteric domains of knowledge and practice and participating in communities that traffic in these forms of expertise” (2009, p. 28). Similar things have been said about fan fiction forumsvideo game forums, and hip-hop forums.The more we think about software environments, like forumsas learning spaces, the morewe need to understand how their design enables and disables particular kinds of discourse.

Welcome to my Dissertation (I Think)

It looks like this is what I am going to write my dissertation about. I intend to blog my way through this process. I have not yet written my proposal, I have drafted a 40 page sketch of some of the conceptual framework I am thinking through. (Which I will not subject you to), but I do not yet have a formal proposal. I plan to work through my research design on the blog and I would like to invite any and everyone even remotely interested in this topic tocomment, critique and share thoughts about how I am going about this. I am largely thinking about this as a project in software studiesplatform studiesSTS, and distributed cognition.

How to Books and A History of Vanilla Forums

To kick this off I thought I would describe the two primary sets of sources I intend to work with. So far I am focusing primarily on two kinds of source material to develop two kinds of stories.

First I am interested in exploring what how-to and technical guides targeted at people setting up software to create web communities can tell us about tell us about the relationships between users, administrators, and developers.

Second, I am interested in using the content and structure of the vanilla forums development forums, and the archive of those forums available through the internet archive, to explore the relationship between ideologies of users and administrators of the software and the actual software itself.


Together, I hope to use these sources to work toward understanding more about how the developers, and administrators of forum software conceptualize their users, and how those conceptualizations of users do or don’t become embedded in the functionality of the forum software itself.

It’s your turn! Take a moment to Opine

I’m mildly interested in what you think about this topic. I am far more interested in what you think about the sources I am considering. What do you think about how-to books and a case study of an open source forum project as the focus of this study? Are there other sources you would think about looking at to explore this? In subsequent posts I will lay out the books I am thinking about looking at and a justification for why I think Vanilla is a particularly interesting case study for this project, but at this point I am particularly interested in hearing feedback about the over all idea behind this area of study and the general kinds of sources I am talking about.