Stuart Reeves: Notes on research

HCI is in continual crisis mode with regard to peer reviewing. Every year around the time of the CHI conference - or to give it its full, unwieldy and faintly outmoded title The ACM SIGCHI Conference on Human Factors in Computing Systems - there is considerable debate about the reviewing process (see the aside at the bottom of this posting for more detail on the reviewing process itself). No doubt similar conversations take place at all other major HCI and related venues (e.g., CSCW), but it seems that the ones from CHI leak onto the web most readily. Other conferences have comparable review processes (CSCW, DIS, Ubicomp), so the conversation here is relevant beyond CHI.

It seems to me that a lot of these conversations revolve around a number of topics related to the review process itself, such as the relative weighting of reviewer and author power, the accountability of activities of programme (sub)committees, the way reviewers get selected, the rising numbers of submissions to the conferences, and (relatedly) the acceptance rates and submission types. As an author it can be incredibly frustrating if the account of why your paper was rejected is scant on detail (such as reports that your paper was discussed at the committee meeting, but the actual detail of the discussion was left out), and correspondingly as a reviewer it can be annoying to have to review work which does not seem to even hit the lower level of what could be considered worthwhile.

In response to participating in the process of peer review, Jeffery Bardzell recently wrote an interesting blog posting outlining the various ways in which not only practical changes could be made to this reviewing process, but also presented some ways the HCI community could reconceptualise what process it is actually engaging in during peer review. Bardzell presents this in his “Position on Peer Reviewing in HCI” (in three parts), and formed a lengthy response to Antti Oulasvirta’s original blog posting on “Why your paper was rejected” with specific reference to CHI.

Oulasvirta’s blog here is key in understanding the meaning of the exchange and in assessing some of the underlying concerns that are perhaps helping animate the discussion. Oulasvirta, as a participant in a subcommittee of CHI 2012, summarises some of the key problems that result in low scores being awarded to some kinds of paper submitted to CHI. He focussed particularly on what he terms “flaws in empirical work”, including such things as: errors in research strategy (i.e., applying the wrong research tool for the job); basic statistical flaws (interestingly Grounded Theory is included here related to errors in inter-coder reliability metrics); causation issues (e.g., problems with the `implies’ relationship); generalisability and replicability of findings.

Reading the blog post reveals Oulasvirta’s epistemic position on the nature of HCI and its relationship to the natural sciences. While it is not explicitly nailed down anywhere, this position seems to be that HCI (or at least parts of HCI) is a scientific discipline, and as such many submissions to conferences may be shot down on a range of scientific grounds, as outlined above. The notion of HCI as a science is grounded in and guided by popular conceptualisations of ‘what science is / what scientists do’. Specifically these are articulations of what is thought to be the essence of scientific practice, such as: the nature of larger problems is such that they may be broken down into subproblems and reassembled upstream (i.e., scientific reductionism); the mutually reinforcing relationship between theory and experiment, with each elaborating the other - experimentation provides ‘canaries in the mine’ for theory (e.g., see the controversy over string/M theory), and theory directs the ‘what next’ of experimentation; the role of replication and experimentation as separate or separable from context and situatedness. These essences from the natural sciences provide yardsticks with which to measure HCI activity against if it is to be (or remain) a normal scientific activity.

This view of HCI is not new, and often emerges in a range of different guises from HCI research. For instance, Bartneck reports how President of the Association for Computing Machinery (ACM) Stuart Feldman’s CHI 2007 opening speech included the assertion that HCI “is absolutely adherent to the classic scientific method”, which, he said, was “[n]ot a description […] of all the fields in computing”. Ben Shneiderman’s comment on Bardzell’s blog raises this issue again with the comment that “HCI is a scientific discipline”. More recently, the CHI 2012 conference is hosting an ongoing workshop series - RepliCHI - that is concerned explicitly with a cornerstone of the natural sciences, namely, replication. The workshop, expressing very common concerns within the HCI community, states that “it is not uncommon for research to be rejected in CHI for being ‘incremental’”, whereas “[i]n science, it is common for people to try and replicate discoveries, so that the community can confirm new discoveries”. There are other variations on HCI’s scientific status, such as James Landay’s blog posting on “giving up CHI/UIST” (comparable maybe to ‘giving up Christmas’ or ‘giving up on a relationship’?). This is a call for a wider notion of science and where it happens: “I think we have been blinded by the perception that ‘true scientific’ research is only found in controlled experiments and nice statistics”. A now-defunct blog posting by Joshua Kaufman (archive.org link) on a talk by Paul Cairns on ‘HCI as Science’ brings Popper’s conceptualisation of what science is to bear on whether HCI is science (e.g., via Popper’s notion of science as falsifiable). Interestingly one of the commenters argues that HCI is more like engineering: “The Wright brothers constructed a working airplane without knowledge of aerodynamics, fluid mechanics, etc. They just applied tried and true engineering best practices which they learned from working on bicycles (plus a ton of trial and error). In fact, several generations of planes were created before scientists started to grasp any of these principles, and the debate continues today. Yet we fly around in planes. They work, even if we don’t have all the ‘laws’ exactly nailed down.” The unstated assumption being that engineering something that interacts with the physical environment is somehow isomorphic with engineering of things that interact with and around humans (I would argue that they are not).

Bardzell’s blog appears to be a reaction in some ways to this implicit / explicit epistemological position on the status of HCI as a science. The perceived problems surrounding the peer review process mentioned at the start of this discussion are transposed into or driven by such epistemic issues. While it appears that Bardzell does not deny that there are scientific aspects to the reviewing process, rather than arguing for the scope of what is considered to be normal science to be enlarged (as Landay does), he suggests that we must attend to the social features of reviewing as they play an important (and overlooked) role in the development of scientific results. In sum he appears to be arguing that reviewing is by its nature “critique” more than (or perhaps instead of) being part of a rigorous scientific process that upholds the popular notions of ‘what science is / what scientists do’. (In some ways perhaps it is not much different from film criticism?) This argument obviously borrows strongly from constructionist accounts in Science and Technology Studies (STS) and Sociology of Scientific Knowledge (SSK) (e.g., Latour and Woolgar’s work), where the outputs of the work of the natural sciences is understood in terms of local practical circumstances (e.g., physical lab) typically filtered through the lense of external features such as politics, power structures, funding systems, etc. In my view Bardzell is linking reviewing work with a consideration of these sorts of aspects.

There are a great many interesting aspects of this discussion. However, it turns out that a lot of these concerns are not new, and not particularly unique to HCI. For instance, we can compare issues around peer review in HCI with some of the gripes emergent from SIGCHI’s parent community, i.e., SIGGRAPH. Ashikhmin expresses similar concerns about the review process, fashionable topics, results being presented in a normatively scientific manner in order acquire the cache of validity, reviewer power, acceptance rates, and assertions about computer graphics being a ‘scientific endeavour’. As Ashikhmin states “computer graphics is simply not a hard science and current attempts to present it as such are misguided at best […] the success or failure of any specific technique is, quite literally, in the eye of beholder”.

Another key aspect of the perceived problem is probably the way in which HCI mixes communities (which is a good, but challenging thing). A great deal of this discussion probably culture clash, perhaps between those coming at HCI from a cognitive science and / or psychology background and those from arts and humanities backgrounds (who are increasingly being involved in HCI). While those from such a non-‘science’ background may seek to conceptualise HCI in the mould of the way that the natural sciences are conceptualised in STS, so correspondingly those from the ‘labelled sciences’ (computer science, cognitive science, behavioural science, etc.) are perhaps guilty of mythologising of what the actual work practice involved in the natural sciences is like, or perhaps forgetting that HCI’s related fields like psychology have plenty of their own scientific demons (e.g., problems with WEIRD behavioural and brain science). While I would not necessarily want to align with STS, SSK, etc., particularly as they have a tendency to eschew studies of mundane practice and revert to something like structural functionalism (as noted by Mike Lynch in Scientific Practice and Ordinary Action), there is something to be said for the importance of the social features of scientific practice and its impact on the presentation of science. However, there is probably an equal tendency by those from a non-‘science’ background to presume that such social aspects are somehow hidden from scientific practitioners, and that they are unknowing ‘cultural dopes’ going about their work.

A nice example of the work of a discovering science is provided by Eric Livingston in his book Ethnographies of Reason. Through a simple procedure he unpacks the embodied and commonsense ways in which a phenomenal field may be investigated through various instruments employed to uncover it. In this account, natural phenomena are explored through practical action, and subject to everyday mundane practicalities. This leads to another issue: that of the relationship between the phenomena that are being investigated by HCI and the role of those self-same phenomena within HCI. As pointed out here in relation to Kaufman’s blog posting “HCI phenomena are constantly changing, HCI is constantly moving into new domains, redefining itself and absorbing new types of technology. Basically, there are no static phenomena so there can’t be an HCI paradigm. Furthermore, since there is no HCI paradigm, HCI is not a science.” This is perhaps not deep enough, though. In this situation we can consider the standard ethnomethodological critique. HCI’s phenomena of interest are actually the phenomena that enable us to study the phenomena in HCI in the first place. In other words, the commonsense methods of local, situated reasoning that are brought to bear in the work of HCI research often remain largely unexamined in spite of the fact that really this is the phenomena we are examining when we do a study. While not attending to these issues may be okay for the natural sciences, which may suspend interest in such matters in favour of transcendental principles of the natural world, for the human sciences (which HCI regularly brushes up against, being concerned with ‘interaction’), we ignore these issues at our peril.

Finally there is the legitimacy argument. Kaufman’s blog posting suggested that it was a good thing for “HCI [to] be considered a science because science was practically the only measurable form of progress in the 20th century”. Scientism faces us all, especially when it comes to gathering funding and establishing the legitimacy of our research work in HCI. In one view this could be seen as a cynical perspective on science (and perhaps devaluing the label), but on the other hand, we can instead see it as ‘strategic’ and necessary.

[An aside: for those who are not familiar with the CHI reviewing process, a simplified version is as follows. The conference is organised into various subcommittees which cater for different aspects of the field of HCI (this subdivison is of course a massive fudge, but practical). Authors target their paper to a given subcommittee; the subcommittee’s chairs then divvy up the papers between themselves and the associate chairs (ACs), who in turn recruit a number of reviewers (3+) for each of the papers they are AC-ing for. Reviewers do the reviewing, ACs write meta-reviews summing up the points made and scores provided by the reviewers, all of which are then sent to authors. Anonymity is top down and bottom up: authors are anonymised for reviewers, and the reviewers are anonymised for the authors. Reviewers are also anonymous between each other. ACs on the other hand know who the reviewers are. Authors, having received their paper’s reviews have the option of writing a rebuttal if they so wish. After the rebuttal period is over, reviewers can alter their review and score based on the rebuttal. The final set of reviews (sometimes augmented with further reviews for particularly divided reviewers) are then used to work out what papers get discussed at the subcommittee meeting. The ACs and chairs meet up, discuss all the papers they feel need discussing (a very opaque process but generally speaking papers below a certain average score are typically dismissed), and then collectively determine accept / reject decisions for all the papers for their particular subcommittee. Finalised reviews and final paper decisions, often including some account of `what went on’ at the subcommittee meeting, are then sent to authors, who are either left whooping with joy or crying in despair.]

A while back (12th April 2011) I set up and ran a workshop at the Mixed Reality Lab on `display ecologies’. The workshop was an attempt to (1) survey existing work in the field and the state of the art as found in the literature; (2) to determine what research questions - whether they are design, conceptual or infrastructural - need to be answered; and (3) to work out what scope there is for innovation. After the workshop I wrote up a report that documented what was discussed, and what each speaker presented.

An excerpt from the report is below, but you can download the whole thing here.

The main purpose of the workshop was to explore public interactive display ecologies – often called ‘situated displays’. Notions of ‘situated displays’, however, are not wide enough a concept to capture the diversity of configurations and forms of physical display (as well as social aspects of ‘display’ and ‘displaying’) that we both encounter in the literature as well as located within the content of the workshop. Instead, it seems that an ‘ecology’ metaphor affords us greater purchase in understanding the design space.

Ecology evokes a range of useful concepts, such as how ecologies are composed of hybrid elements, their heterogeneity in that they fit together or are made to fit together and yet retain boundaries and seams. Overall, ecology as a metaphor is invoked to suggest the ‘messy’ reality of real-world configurations of display. ‘Ecology’ also implies display systems that exist in an ever-changing environment. It implies that these messy interrelated components might self-organise in some way. Ecology implies things might be brought into play that we didn’t expect - new objects and so on which might interact with the displays somehow but we didn’t expect them to. Ecology implies display elements that mesh more with non-digital resources and a wider understanding of what a ‘display’ is or might be.

My paper “Envisioning ubiquitous computing” is to be published at CHI 2012. You can grab a PDF of it here. Full reference and abstract below.

Reeves, S. Envisioning ubiquitous computing. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, May 2012.

Visions of the future are a common feature of discourse within ubiquitous computing and, more broadly, HCI. ‘Envisioning’, a characteristic future-oriented technique for design thinking, often features as significant part of our research processes in the field. This paper compares, contrasts and critiques the varied ways in which envisionings have been used within ubiquitous computing and traces their relationships to other, different envisionings, such as those of virtual reality. In unpacking envisioning, it argues primarily that envisioning should be foregrounded as a significant concern and interest within HCI. Foregrounding envisioning’s frequent mix of fiction, forecasting and extrapolation, the paper recommends changes in the way we read, interpret and use envisionings through taking into account issues such as context and intended audience.

This paper was developed from work on the Designing the Augmented Stadium project and will be published at CHI 2012. It is available as a PDF from my website.

Morrison, A., McMillan, D., Reeves, S., Sherwood, S., and Chalmers, M. A hybrid mass participation approach to mobile software trials. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, May 2012.

User trials of mobile applications have followed a steady march out of the lab, and progressively further ‘into the wild’, recently involving ‘app store’-style releases of software to the general public. We examine the literature on these mass participation systems and identify a number of reported difficulties, which we aim to address with a hybrid methodology combining a global software release with a concurrent local trial. A phone–based game, World Cup Predictor, was created to explore the uptake and use of ad hoc peer-to-peer networking, and evaluated using our hybrid trial method, combining a small-scale local trial (11 users) with a ‘mass participation’ trial (over 10,000 users). Our hybrid method allows for locally observed findings to be verified, for patterns in globally collected data to be explained and addresses ethical issues raised by the mass participation approach. We note trends in the local trial that did not appear in the larger scale deployment, and which would therefore have led to misleading results were the application trialled using ‘traditional’ methods alone. Based on this study and previous experience, we provide a set of guidelines to researchers working in this area.

The point of this particular posting is to bring to attention some of Alfred Schutz’s writings on the use of ‘futures’. Recently someone pointed me towards it due to my interest in understanding the role of futures in HCI and ubiquitous computing design.

Tucked away at the back of Volume II of Schutz’s Collected Papers is his essay on “Tiresias, or our Knowledge of Future Events”. In this essay, Schutz describes the inherently social ways in which understandings of the future are developed by members of society, as analysed from his phenomenological position. (Reading his work rather reminds me of Goffman’s style—-rigorous and relentlessly detailed, drawing extensively from the phenomenological experience of the everyday, but without an obvious scientific ‘empirical’ derivation.)

One of Schutz’s key concept is the ‘stocks of knowledge’ that are developed by members of society in and through their everyday activities. I say ‘stocks’ because these are locally-rational, situationally-generated sets of knowledges that are relevant to the tasks-at-hand.

In this paper his argument seems to be that these knowledges play a fundamental role in the ways that we anticipate, prophesy, predict, forecast, guess, and intuit what will happen in the future and how we plan for those future events and determine their predicted unfolding. These situationally-relevant stocks of knowledge are developed through streams of past and present experiences. Although they are used as a ‘prediction space’ for the future, the same time, these knowledges are necessarily bound (and can only ever be) to the past. As Schutz states:

All projecting consists in an anticipation of future conduct by way of phantasying. It is, to use Dewey’s pregnant description of deliberation, “a dramatic rehearsal in imagination.” Yet projecting is more than mere phantasying. Projecting is motivated phantasying, motivated by the anticipated supervening intention of carrying out the project. The practicability of carrying out the projected action within the imposed frame of reality of the is an essential characteristic of the project. This refers, however, to our stock of knowledge at hand at the time of projecting. Performability of the projected action means that according to my present knowledge at hand the projected action, at least as to its type, would have been feasible if the action had occurred in the past.

It seems to me that the problem he is raising here is that future anticipations are inherently tied to these past and present stocks of knowledge, that that our understanding of the way the future may possibly develop is also bound to how these knowledges developed, but only in the past. Schutz calls this a kind of “anticipated hindsight”, i.e., that future prediction, foresight, forecasting, etc. is nothing less than hindsight transposed to a future framing. Or in other words, “we cannot expect any event of whose typicality we have had no pre-experience”. Instead:

Once materialized, the state of affairs brought forth by our actions will necessarily have quite other aspects than those projected. In this case foresight is not distinguished from hindsight by the dimension of time in which we place the event.

After the future event has (or has not) transpired, these experiences become part of the present-past stocks of knowledge:

Yet because of their very typicality our anticipations are necessarily more or less empty, and this will be filled in by exactly those features of the event, once it is actualized, that make it a unique individual occurrence.

These knowledges are also social, i.e., for them to have utility they must be shared, and known-in-common by members of society. In this way,

validity of anticipations […] is founded on the assumption that some or all of my fellow-men will find in their stock of knowledge at hand typically similar elements, and that these will determine the motives of their action.

This is all pretty abstract. But it can be understood in terms of how it can pose us with some potential problems when we are thinking about what technologies we can, might, or will build, and what the impacts of those technologies can, might, or will be. Perhaps not ‘problems’ as such, but rather necessary restrictions and limitations of the kinds of technologies we can possibly imagine, and how we treat them should they actually ‘arrive’. In my reading, Schutz’s argument gets us to the point where we appreciate that our anticipations and expectations about future technologies are derived from sets of existing typifications that are fundamentally based in the past. These typifications are ‘empty’, and ‘filled in’ with the specificities and peculiarities should the event actually transpire. In this way all visions of future technologies and societies are, and can only ever be, deeply embedded in the past, and the dimensions of feasibility understood as derived from the past. This is perhaps why many visions of the future are so bland and unadventurous.

Recently in collaboration with Halfman Design, and in coordination with Slovenian organisations SCNR and NUK, we developed and deployed a simple interactive site-specific audioguide at a woodland mass grave memorial (at Kočevski Rog) in Slovenia. ‘Spomenik’ was designed to be very accessible (requiring only standard mobile phone functionality—voice and text), and work in a quite ‘remote’ (in infrastructure terms) forest setting. The audioguide was powered by Tropo, a cloud telephony service. A set of images from Halfman Design explains the experience better than I could, so I have presented some of them here with annotation.

The memorial site we focused on featured a large cave (in which killings took place), and a variety of unofficial and official memorials, including these bell monuments:

The path into the woods consisted of two distinct ‘stations’ on which we placed signs highlighting our audioguide phone number. During our deployment a few groups of Slovenian students visited the site and used the system:

Here is the first station indicating the start of the path into the woods:

Visitors then continue into the woods:

And reach the next station, which is sited by the mass grave:

There is a large collection of unofficial and official memorials at the grave site, including the bells around which we based the phone experience:

Physical visits to the site, and reflections recorded by visitors are made available on a simple website:

We are currently in the process of analysing the interactional aspects of this work.

The scripts that run it are available on my github.

There is an ongoing debate over ethnography in HCI and its attendant fields (e.g., CSCW) that finds its yearly outpouring at CHI (the SIGCHI conference on human factors in computing systems). Three obvious and recent points of reference are Dourish’s “Implications for design” paper in 2006 [1], in 2009 Crabtree et al.’s “Ethnography considered harmful” [2], and most recently, Dourish and Bell’s book Divining a Digital Future (2011) [9]. This debate has been cast more or less as being about both the kind of commitments that ethnographies are conducted with, and how that meshes with—particularly regarding what ethnography does for—design.

The latest edition of the saga is found in Rode’s paper “Reflexivity in digital anthropology”, recently published at CHI 2011 [8] (see Taylor’s paper for an interesting counterpoint [11]).

Broadly speaking, the paper (from my reading) develops the argument that there is a shared, complementary validity to the exercise of different forms of ethnography that we find at venues like CHI. In order to advance this argument, the paper structures its analysis heavily on three characterisations of ethnography developed by van Maanen [12]. Unfortunately for the reader, the paper glosses over the various commitments with which these ethnographies can be exercised (ranging from actor network theory, grounded theory, distributed cognition, interactionist perspectives, and ethnomethodology). Nevertheless, the paper engages in a discussion of van Maanen’s “realist”, “confessional” and “impressionistic” forms of ethnography; as follows:

• Quoting van Maanen, Rode states that “[t]he realist tale is ‘a documentary style focused on minute, sometimes precious, but thoroughly mundane details of everyday life among the people studied,’ which are carefully structured to support authors’ points. Details give the impression of reliability.” [8]
• Whereas, impressionist ethnographies “capture a moment in time from a uniquely individual perspective” [8], although they appear to be primarily to do with scene-setting before more ‘realist’ ethnographic mode of accounts proceeds.
• Apparently “more reflexive” (and therefore presumably ‘better’) are confessional ethnographies, in which ethnographers “engage with the nagging doubts surrounding the study and discuss them textually” [8].

It becomes apparent quite quickly that, of course, amongst what Rode characterises as ‘realist’ covers a range of kinds of ethnographies with all sorts of different commitments. However, part of what is pointed at particularly by citation is the ethnomethodologically-informed strain of ethnography and its progeny that is represented (with varying underlying commitments) regularly at venues like CHI, CSCW, ECSCW, DIS, and so on. So, for the purposes of this discussion, I’ll naively assume Rode is talking in part about ethnomethodologically- and (often) conversation-analysis-informed ethnographic work, such as examples from Heath et al. [3], Bentley et al. [4] and Crabtree et al. [5].

And, although the paper is shielded by ‘why can’t we all get along?’ bookends and an appeal to the complementarity of different ethnographic forms, the meat of the argument is concerned with advancing a critique of this so-called ‘realist’ form of ethnography. Simply, this argument is that ‘realist’ work, through being written to provide “an impression of reliability”, can be deficient thanks to a lack of ‘reflexivity’ (in the cultural anthropological sense [10]) which to varying extents both ‘impressionistic’ and ‘confessional’ ethnographies supply.

What is meant by increased ‘reflexivity’? Arguments for reflexivity demand that ethnographers begin to address more of the context in which they conduct their ethnography, attending to and describing for the reader their personal background, their rapport with ‘informants’, and the way in which ethnographers themselves interpret and make sense of settings (and convert this into a textual version for the ethnographic report) during participant-observation. Of course, Rode (and presumably van Maanen) claims that ‘realist’ ethnography is ‘unreflexive’, and that it “black boxes” much of what is going on within ethnographic study. Lynch expands nicely on this:

According to a common constructionist formula, objective expressions rhetorically delete, disguise or place in a ‘black box’ the tacit knowledge, cultural origins and epistemological limits of knowledge. A theoretical, and even moral, imperative for those who follow the constructionist programme is to open up the ‘black box’ to expose the local origins and cultural limitations that have been deleted. But if an initial statement is held to be unreflexive, then the question is: How much more needs to be added to it before it becomes reflexive?’ [10]

However, here emerges the problem: in contrasting ‘unreflexive’ realist ethnography with these more explicitly reflexive forms, the assumption is that in providing such increased reflexive detail about the “nagging doubts” of the ethnographer, the ethnography will somehow be ‘better’ or ‘truer’, more useful or even more insightful. Although it is doubtful ‘reflexive’ ethnographers would agree with the following statement, the net result boils down to reflexivity being an attempt to provide a privileged position to the ethnographer’s observations, insights and reports from the field. The mechanism through which this is done is by providing increased disclosure about the surrounding context of how the ethnography was done, and related ‘reflexive’ details.

However, although never explicitly stated in this manner within the paper, and glossed as helping us better understand “the real world appropriation of technology”, it’s hard to determine exactly what reflexivity is supposed to do for us besides, grossly, some kind of ‘improvement’. As with many calls to increased reflexivity, the implicit assumption is that a reflexive version is somehow advancing the work of ethnography, although it appears difficult to articulate exactly what it is actually working towards improving. Instead, as Lynch suggests, we can see reflexivity as a method for ethnographers to construct this appearance of a somehow more privileged, accountable position, which is a particularly pressing and relevant concern in the fight for academic survival (as hinted at by Rode’s gripes in the paper about these forms of ethnography encountering trouble during peer review). Lynch argues that notions of reflexivity are typically deployed as “polemical efforts to promote theoretical and methodological advantage in a noisy field” (although to some extent Rode steps back from this argument with the notion of complementarity). Ironically we end up circling back to the earnest desire for increased ‘objectivity’ that reflexive ethnographers began reacting against in the first place—just that it’s ‘objectivity’ by another name, specifically in mobilising the veracity cache associated with doing ‘being reflexive’ in a study.

The bottom line here is that, in the end, there are no guarantees that increased reflexivity does anything, and is not a method for producing a ‘good ethnography’:

its cogency will depend upon what it says about its topic and whether it persuades relevant audiences. Depending on the case, [reflexivity] may come across as insightful, witty, convincing, unconvincing, boring or silly.

[…] attempting to be reflexive takes one no closer to a central source of illumination than attempting to be objective [10]

As Lynch points out, ethnomethodology’s approach to reflexivity provides a radical recalibration of understanding in contrast with the hang-wringing forms of reflexivity routinely found in anthropology, STS, and other social sciences. Instead of viewing reflexivity as something that can be somehow added to an ethnography, we should instead view reflexivity itself as the very core phenomena under investigation within our ethnographies.

Social interaction is intrinsically reflexive, in that the procedures and methods we use in order to make our actions accountable (observably and reportably so) to one another are the selfsame actions that “produce and manage settings of organized everyday affairs” [6, p. 1]. For ethnomethodology, “reflexivity is ubiquitous and unremarkable” [10]. Thus, Livingston argues that we should “pursue reflexive phenomena, not yet knowing and waiting to discover, for any particular domain, what such an interest may refer to […] if […] such phenomena are missing from a study, we know we’re in serious trouble” [7].

There is a wider issue at work here with this argument about reflexivity, however, particularly regarding the wholesale importation of cultural theory (via media studies and so on) into the field of HCI. Indeed, the signature of these new ‘reflexive’ ethnographies increasingly gaining exposure within CHI and other venues involves “gendered, racial and cultural ‘standpoints’ that provide existential conditions for reflexive critiques of dominant discourses” [10].

The application of such theory to ethnographies is problematic, as highlighted by Crabtree et al. [2]. By using theory as a lense, we lose the particularities of the settings we study. In the end, we end up knowing less about them, and ethnographies like these end up being about the analyst, about their academic discussions, and not about routine, everyday, observable and reportable lifeworld of those settings. This is detectable in what Rode states in the paper: “[f]or instance, in my recent work on gender I went to the field having thoroughly read up on feminist theory”.

Such practices run into danger in that the ethnography becomes a discussion around theory, rather the materialities of the setting, resulting in, say, Rode’s feminist theory being merely applied to the latest situation, and ending up with the same old story but with a scenery change. The commitment to this attitude becomes clearer as Rode further states, “impressionist text drawing from months of fieldwork can strengthen theories by grounding it in detailed examples”—i.e., fieldwork becomes a merely servant to elaborating a particular theory. As Hughes et al. point out, “concepts that sociologists routinely deploy […] are abstractions which serve sociological purposes, and only sociological purposes” [13].

So, far from “black boxing” social order, ethnomethodologically-informed so-called ‘realist’ ethnography seeks to practically engage with exploring taken-for-granted aspects of how that social order is constituted. Instead, it is the application of standard sociological theoretical preoccupations and narratives—such as gender, class, power relations or culture—that actually “black box” the material, actual ways in which the sites of anthropological studies are organised and made orderly.

Rant over.

References

1. P. Dourish. Implications for design. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI), pages 541-550, New York, NY, USA, 2006. ACM.

2. A. Crabtree, T. Rodden, P. Tolmie, and G. Button. Ethnography considered harmful. In Proc. CHI ‘09, pages 879-888, New York, NY, USA, 2009. ACM.

3. C. Heath and P. K. Luff. Collaboration and control: Crisis management and multimedia technology in London Underground line control rooms. Journal of Computer Supported Cooperative Work, 1(1-2):69-94, 1992.

4. R. Bentley, J. A. Hughes, D. Randall, T. Rodden, P. Sawyer, D. Shapiro, and I. Sommerville. 1992. Ethnographically-informed systems design for air traffic control. In Proc. CSCW ‘92. ACM, New York, NY, USA, 123-129.

5. A. Crabtree, S. Benford, T. Rodden, C. Greenhalgh, M. Flintham, R. Anastasi, A. Drozd, M. Adams, J. Row-Farr, N. Tandavanitj, and A. Steed. Orchestrating a mixed reality game `on the ground’. In Proc. CHI, pages 391-398, New York, NY, USA, 2004. ACM Press.

6. H. Garfinkel. Studies in Ethnomethodology. Prentice-Hall Press, 1967.

7. E. Livingston. Ethnographies of Reason. Directions in Ethnomethodology and Conversation Analysis. Ashgate, 2008.

8. J. A. Rode. Reflexivity in digital anthropology. In Proc. CHI ‘11, pages 123-132, New York, NY, USA, 2011. ACM.

9. P. Dourish and G. Bell. Divining a Digital Future: Mess and Mythology in Ubiquitous Computing. MIT Press, May 2011.

10. M. Lynch. Against reflexivity as an academic virtue and source of privileged knowledge. Theory, Culture & Society, 17(3):26-54, June 2000.

11. A. Taylor. 2011. Out there. In Proc. CHI ‘11. ACM, New York, NY, USA, 685-694.

12. J. Van Maanen. Tales of the field; on writing ethnography. Chicago UP, 1998.

13. J. Hughes, D. Randall, M. Rouncefield and P. Tolmie. Meetings and the accomplishment of organization. In Ethnomethodology at work, M. Rouncefield and P. Tolmie (eds.), Ashgate, 2011.

The Rider Spoke interactive bicycle-based work I was involved in a few years ago has a publication coming out in Pervasive 2011. Abstract and reference below.

Oppermann, L, Flintham, M., Reeves, S., Benford, S., Greenhalgh, C., Marshall, J., Adams, M., Row Farr, J. and Tandavanitj, N. Lessons from Touring a Location-Based Experience. In Proc. Pervasive 2011, to appear.

Touring location-based experiences is challenging as both content and underlying location-services must be adapted to each new setting. A study of a touring performance called Rider Spoke as it visited three different cities reveals how professional artists developed a novel approach to these challenges in which users drove the co-evolution of content and the underlying location-service as they explored each new city. We show how the artists iteratively developed filtering, survey, visualization and simulation tools and processes to enable them to tune the experience to the local characteristics of each city. Our study reveals how by paying attention to both content and infrastructure issues in tandem the artists were able to create a powerful user experience that has since toured to many different cities.

A paper I helped write on the augmented reality collaborative installation Flypad is to appear in ECSCW 2011. Reference and abstract to be found below.

Flintham, M., Reeves, S., Brundell, P., Glover, T., Benford, S., Rowland, D., Koleva, B., Greenhalgh, C., Adams, M., Tandavanitj, N. and Row Farr, J. Flypad: Designing Trajectories in a Large-Scale Permanent Augmented Reality Installation. In Proc. ECSCW 2011, to appear.

A long-term naturalistic study reveals how artists designed, visitors experienced, and curators and technicians maintained a public interactive artwork over a four year period. The work consisted of a collaborative augmented reality game that ran across eleven networked displays (screens and footpads) that were deployed along a winding ramp in a purpose-built gallery. Reflections on design meetings and documentation show how the artists responded to this architectural setting and addressed issues of personalisation, visitor flow, attracting spectators, linking real and virtual, and accessibility. Observations of visitors reveal that while their interactions broadly followed the artists’ design, there was far more flexible engagement than originally anticipated, especially within visiting groups, while interviews with curators and technicians reveal how the work was subsequently maintained and ultimately reconfigured. Our findings extend discussions of ‘interactional trajectories’ within CSCW, affirming the relevance of this concept to describing collaboration in cultural settings, but also suggesting how it needs to be extended to better reflect group interactions at multiple levels of scale.

A couple of related papers on design, both found in Design Studies journal:

W. Sharrock and R. Anderson. The user as a scenic feature of the design space. Design Studies, 15(1):5-18, 1994.

J. Redström. Towards user design? On the shift from object to user as the subject of design. Design Studies, 27(2):123 - 139, 2006. 

I thought I’d note this down here to save someone else the pain (through ignorance and not reading enough) I went through.

The situation is this: say you have an array of different object types that each implement and extend the same abstract class. Persisting such a list using JDO should be pretty straightforward, although finding out how to do this common task is not hugely documented in the web (apart from here) in a way that is comprehensible for code copy+paste idiots like me.

The particular situation I had involved creating and persisting a List of Items, e.g.:

Container.java:
---
@PersistenceCapable
public class Container
{
    ...
    
    @Persistent
    List items = new ArrayList();
    
    ...

    public Container()
    {
        items.add(new ItemA(4, "hello"));
        items.add(new ItemB(93, "world"));
    }
}

Then calling makePersistentAll() on Container.

Here are the rest of the classes:

Item.java:
---
@PersistenceCapable
@Inheritance(strategy=InheritanceStrategy.NEW_TABLE)
@Discriminator(strategy=DiscriminatorStrategy.CLASS_NAME)
public abstract class Item
{
    @Persistent
    int i;

    public Item(int i)
    {
        this.i = i;
    }

    public int getI() { return i; }
    
    public void setI(int i) { this.i = i; }

    public abstract String toString();
}

ItemA.java
---
@PersistenceCapable
@Inheritance(strategy=InheritanceStrategy.SUPERCLASS_TABLE)
public class ItemA extends Item
{

    @Persistent
    String s;

    public ItemA(int i, String s)
    {
        super(i);
        this.s = s;
    }

    public String toString()
    {
        return ItemA.class.getName() + ": i = " + i + ", s = " + s;
    }

    public void doStuff() { }
}

ItemB.java
---
@PersistenceCapable
@Inheritance(strategy=InheritanceStrategy.SUPERCLASS_TABLE)
public class ItemB extends Item
{
    
    public ItemA(int i, String s)
    {
        super(i);
        this.s = s;
    }

    public String toString()
    {
        return ItemA.class.getName() + ": i = " + i + ", s = " + s;
    }

    public void doSomething() { }
}