Stuart Reeves: Notes on research

On replication in HCI

Mon, 06 May 2013 17:40:00 -0400

At CHI 2013 I attended a 2 day workshop on ‘replication in HCI’ (also known as ‘RepliCHI’, a programmatic concern reflected in various workshops and panel discussions at prior CHI conferences). To summarise the purpose of the workshop I will turn to the workshop proposal abstract, which does a fine job:

The replication of, or perhaps the replicability of, research is often considered to be a cornerstone of scientific progress. Yet unlike many other disciplines, like medicine, physics, or mathematics, we have almost no drive and barely any reason to consider replicating the work of other HCI researchers. Our community is driven to publish novel results in novel spaces using novel designs, and to keep up with evolving technology. The aim of this workshop is to trial a new venue that embodies the plans made in previous SIGs and panels, such that we can begin to give people an outlet to publish experiences of attempting to replicate HCI research, and challenge or confirm its findings.

Broadly speaking the position I presented (to be found elaborated in the paper Christian Greiffenhagen and I submitted) was intended to problematise the assumptions built into this notion of ‘replication’ in HCI. One of the key problems we saw was not necessarily that there is anything at issue in asking HCI to tolerate / involve more replication. Rather, that framing this in terms of making HCI more ‘scientific’ is possibly based on a mythical view of ‘good science’ of which “replicability of published research findings […] is often considered a cornerstone of progress” (see above). Our general position was that sociology of science warns us against holding a mythological view of replication in the natural sciences and then applying this to HCI. Replication of results in, say, physics, serves a highly motivated and particular purpose in working through contested parts of the discipline, rather than being a practice engaged in as a matter of course in order to be seen ‘doing normal science’. Further, the existence of any ‘decisive replication’ becomes problematic in that there is not necessarily any standard for what is considered a valid replication before said replication is attempted.

The workshop’s submissions discussed a range of instances attendees at the workshop considered examples of ‘replication’. But there were a number of nagging issues that continually arose (note that those documented below are not necessarily things voiced by me during the workshop).

1. Classification

There were considerable problems arising from the idea of classifying different ‘types’ of replication (the original workshop proposal identified four: “direct replication”, “replicate and extend”, “conceptual replication” and “applied case studies”). The workshop organisers themselves admitted to a significant amount of argument in even coming up with them.

This replication type became more confusing when considering if the intent of the replication was to replicate original research findings, the original research method, or both. Thus it turned out that the intent of the replicating authors was important for articulating what any given replication is ‘doing’, i.e., in terms of whether it is testing and validating ‘the instrument’ used to generate particular findings (e.g., a methodical procedure), or testing the findings themselves.

Another matter about classification was ‘self-replication’. A few presenters described what were essentially replications of their own prior work, typically with some extension. I found these quite problematic: surely for replication to have any meaning, it must be understood as a fundamentally social, negotiated process? Self-replicating an experiment or trial is not really getting to the heart of what ‘attempting to replicate findings’ is doing for negotiating contested results between members of a research community.

Thus, the important underlying issue for classification schema here is that it was unclear whether a particular set of HCI researcher activities constituted, could be classed as, or could be recognised by others as, a replication at all.

2. Fidelity

In one view, it seemed that the more details unearthed by a replicating author the better: increased fidelity of the replication could be achieved through re-creating the environment in which the original study was conducted. Involving existing authors seemed key, as well as access to their data.

Yet, in another view, it was not clear ‘how much’ of the original circumstances were necessary to be ‘really doing a replication’. In the end a ‘perfect’ replication is obviously intractable, and as such the adequacy of material actions leaving to being seen as ‘doing a replication’, aside from the practical circumstances of its production, are determined through social agreement / disagreement amongst a community of researchers.

3. Incremental versus novel

A common problem seemed to be the argument that paper reviewers were found to be frequently shooting down papers perceived to be replicating prior work with statements similar to “it’s just a replication” or “too incremental” or “not novel enough”. However, within the workshop while attendees did report similar experiences, there were also a number of successfully accepted papers (e.g., actually were appearing in the main conference track).

There are two things to say about this. Firstly, I would argue that the sorts of criticisms described are not just used against the more experimentally-oriented HCI work, but instead is a pervasive phenomenon across (CHI) submission types. So any cultural change to move away from the matters raised by authors attempting to get what are seen as ‘replications’ of prior studies published must also acknowledge that this is part of a wider problem. Secondly, and perhaps more cruelly, it is possible that the reason why authors are having trouble turning papers which exhibit some measure of replication into publications, is simply that, based on the fact that some indeed are being published, the papers are just not very good. However, rather than being a matter of the content not being good enough, I suspect that most of the time it is actually the articulation of the contribution which is causing the problems for acceptance.

Leaving this second point alone for the moment, the first point underlines the importance of good reviewing, but also good articulation of the contribution by authors. By ‘good reviewing’ in this context I mean reviewers that seriously consider the contribution of the work, irrespective of whether it is seen as ‘just a replication’. Considering the contribution is about appreciating that there may be value in ‘going over old ground’. On balance, however, I felt that while it was clear reviewers needed a cultural change, authors need to do more to articulate the contribution of their work too, instead of hiding behind a picture of ‘normal science’ in which ‘doing replication’ is seen as an adequate contribution because ‘that’s what science practice is’.

4. Normal HCI practice

The RepliCHI programme has the potential to actually work against its purpose. This is because in offering a venue for work considered to be ‘replication’, it may well silo such work and through this de-normalise it as part of HCI research practice. Ironic indeed.

5. Ambitions

A final note, which I have to attribute to a conversation with Bob Anderson around this matter of replication. The sorts of ambitions that the RepliCHI programme has (which include political ambitions) can potentially also have epistemological consequences. For instance a political manoeuvre by psychology (say) to adopt the methods and standards of the natural sciences as part of a ‘package’ (i.e., to ‘look like physics’) in order to establish itself politically as a science has had epistemological consequences in terms of debates about replication, the kinds of experimentation which is valid, how results are reported, and so on.

In HCI we should be aware of these consequences and argue that political ambitions as a field can be changed rather than being subject to them.

(Further reading)

Paul Resnick, one of the workshop co-organisers, has written up his own reflections on the event.
I wrote a related blog post a while back called “HCI peer review in crisis: what is science in HCI?”, which discusses the various positions I saw emerging from a set of discussions around review process particularly within the CHI conference.
There is a discussion related to this blog posting to be found (publicly) on Facebook.

ECSCW publication: Understanding Mobile Notification Management in Collocated Groups

Thu, 18 Apr 2013 09:18:00 -0400

We have a paper soon to be published at ECSCW 2013. This is an empirical study of the ways in which collocated groups of people, engaged in a simple photo-based task using mobile phones, coordinated their activities around the receiving of notifications. These notifications were informing them both about the activities of other (distant) groups engaged in the same photo-based task, as well as any located media they encountered (e.g., a pre-existing photograph which were ‘close by’).

Abstract and citation below. A PDF is available.

We present an observational study of how notifications are handled by collocated groups, in the context of a collaborative mobile photo-taking exercise. Interaction analysis of video recordings is used to uncover the methodical ways in which participants manage notifications, establishing and sustaining co-oriented interaction to coordinate action, such as sharing notification contents and deciding on courses of action. Findings highlight how embodied and technological resources are collectively drawn upon in situationally nuanced ways to achieve the management of notifications delivered to cohorts. The insights can be used to develop an understanding of how interruptions are dealt with in other settings, and to reflect on how to support notification management within collocated groups by design.

Joel Fischer, Stuart Reeves, Stuart Moran, Chris Greenhalgh, Steve Benford, and Stefan Rennick Egglestone. Understanding mobile notification management in collocated groups. In European Conference on Computer-Supported Cooperative Work. Springer, September 2013.

EPSRC Fellowship award

Thu, 11 Apr 2013 11:52:08 -0400

I have recently been awarded an EPSRC Early Career Fellowship. The purpose of this five year fellowship (starting later in 2013) is to:

Understand the nature of ‘theory’ and ‘practice’ in HCI research and HCI-related industry work, and unpack the connections between the two;
Conduct empirical studies of the actual work practices found across user experience design, interaction design, and HCI;
Create, iterate and evaluate any supporting tools and techniques which can scaffold two-way relations between theory and practice, research and industry.

Partners include PARC and Microsoft Research.

RepliCHI workshop: Is replication important for HCI?

Fri, 22 Feb 2013 09:33:01 -0500

Christian Greiffenhagen and I have a contribution to the RepliCHI workshop on ‘replication in HCI’ which is running at CHI 2013. I have made the (short) position paper available below (abstract and reference). You can download the PDF here.

Replication is emerging as a key concern within subsections of the HCI community. In this paper, we explore the relevance of science and technology studies (STS), which has addressed replication in various ways. Informed by this literature, we examine HCI’s current relationship to replication and provide a set of recommendations and points of clarification that a replication agenda in HCI should concern itself with.

Christian Greiffenhagen and Stuart Reeves. Is replication important for HCI? In Workshop on replication in HCI (RepliCHI), SIGCHI Conference on Human Factors in Computing Systems (CHI), April 2013.

CHI 2013 publication ‘See Me, Feel Me, Touch Me, Hear Me’: Trajectories and Interpretation in a Sculpture Garden

Fri, 22 Feb 2013 09:27:34 -0500

We have a paper published at CHI 2013 on the design of an interactive guide for a cultural setting (a sculpture park). You can get the PDF here.

Reference and abstract below.

We apply the HCI concept of trajectories to the design of a sculpture trail. We crafted a trajectory through each sculpture, combining textual and audio instructions to drive directed viewing, movement and touching while listening to accompanying music. We designed key transitions along the way to oscillate between moments of social interaction and isolated personal engagement, and to deliver official interpretation only after visitors had been given the opportunity to make their own. We describe how visitors generally followed our trajectory, engaging with sculptures and making interpretations that sometimes challenged the received interpretation. We relate our findings to discussions of sense-making and design for multiple interpretations, concluding that curators and designers may benefit from considering “trajectories of interpretation”.

Lesley Fosh, Steve Benford, Stuart Reeves, Boriana Koleva, and Patrick Brundell. ‘See Me, Feel Me, Touch Me, Hear Me’: Trajectories and interpretation in a sculpture garden. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, April 2013

ACM 'interactions' magazine article: Building the future with envisioning

Tue, 08 Jan 2013 07:15:36 -0500

I have an article titled “Building the future with envisioning” published in the Jan/Feb edition of ACM interactions magazine. This is “the flagship magazine for the ACM’s Special Interest Group on Computer-Human Interaction (SIGCHI), with a global circulation that includes all SIGCHI members”, according to their website. This also means that the article is subscription-only and only a portion is accessible online to those without institutional access or some other subscription arrangement. Thus, if you wish to read it, please email me and I can furnish you with a copy.

Personal and Ubiquitous Computing Special Issue

Tue, 11 Dec 2012 19:43:14 -0500

I’m co-guest editing a special issue of Personal and Ubiquitous Computing in the coming year (2013).

The title is Understanding Performative Interactions in Public Settings, and the special issue has been organised by Julie R. Williamson, Lone Koefoed Hansen, Giulio Jacucci, myself and Ann Light.

Some details below:

Interactive technologies are employed in ever more diverse ways: notably, for this special issue, in a wide variety of settings where interactions are transformed into implicit or deliberate ‘performative’ spectacles. Performative spectacles range from everyday actions such as technology use while in urban spaces like city streets to ‘staged’ or deliberately designed cultural and artistic experiences and they can be experienced both by physically co-located or remote spectators. Viewed as a spectacle, human-computer interaction and particularly the individual users become subjects of public interpretation with everyone watching and making sense of each other’s interactions and intentions. These public interactions can (be purposely designed to) range from ‘invisible’ and hidden to highly visible and expressive, often challenging social norms or even evoking discomfort—physically, emotionally or socially.

While social environments have always offered space for interaction and spectacle, ‘performative’ settings are now being defined, supported and enhanced technologically by ubiquitous and mobile computing, gesture-based computing and sophisticated sensing and sensor networks. The study of human-computer interaction has already seen a growing use of perspectives from performance and theatre studies, which are increasingly brought to bear on the challenges of designing interactive systems that leverage the ‘performative’ aspects of these complex environments. In this view, as designers and researchers in HCI and ubiquitous technologies, we design performative artefacts and experiences whenever we open up new interactive possibilities for settings where others may see, feel or hear our actions.

This special issue aims to highlight the performative aspects of designing for interactions – mobile and immobile – in public spaces. It acknowledges the increasing importance of understanding how we exploit characteristics of spectacle and manage the boundaries and transitions evoked in public ‘performance’. In doing so, it offers the chance to explore how human-computer interaction can be described, designed and evaluated by understanding everyday interactions as potentially ‘performative’, as well as exploring those public settings in which technology is more explicitly used for performance purposes.

For more information on topics and important dates, see the special issue website.

Misconstruing reflexivity (again)

Mon, 30 Jul 2012 20:37:00 -0400

It is often interesting to examine the ways in which prior work has been taken, interpreted and applied in others’ subsequent research and compare it with the original intentions. While I’m sure some might argue, Barthes-like, that once our research writings are out there, then the role for us as authors has ‘died’ at their moment of publication, I tend to disagree with this view.

At CHI 2011 I co-authored a paper (“Into the wild: Challenges and opportunities for field trial methods”) that detailed a study of a field trial of technology (in our case a simple photo-sharing application that was part of the Designing the Augmented Stadium project). In the paper we explore how this popular form of HCI research—i.e., design it, build it, stick it out into the world, study it, report findings—works out in practice.

In part riffing on this paper is a recent (CHI 2012) piece from authors Johnson, Rogers, van der Linden and Bianchi-Berthouze, “Being in the Thick of In-the-wild Studies: The Challenges and Insights of Researcher Participation”. Like us, Johnson et al. have performed a field trial (a very nicely designed interactive haptic / visualisation device for assisting in violin bowing), and in addition to reporting on the field trial findings, they have provided extensive discussion on the nature of the field trial process.

What are field trials, though? We described what field trials were like this:

“Field trials have become a common method for studying the use of novel technologies, widely used in fields (such as HCI, Ubicomp and CSCW) where the interests of investigators goes beyond technical feasibility to exploring user understandings, practices and the eventual uses that systems might be put. What we refer to as trials in this paper go under a range of different names—field experiments, deployments, evaluations, field studies, technical probes—but they share a set of common features. A new system, usually developed by the researchers, is given to a set of users who are asked (often implicitly) to use the system ‘naturally’ outside the laboratory the system was designed in. They then use the system for anything from a few hours to a year, frequently as part of their day to day life, or with the system deployed in their home or workplace. In more experimental trials users’ behaviour may be constrained with tasks set for them to carry out, while other trials eschew such controls and attempt to encourage ‘natural’ use. In some cases trials are run as an extension of experiments, whereas for others a more ethnographic analytic mode is adopted.”

Johnson et al. talk about the role of the researcher in the field trial, unpacking what work was required by the investigator (in this case Johnson herself) during the trial—for instance, as facilitator, as champion, as tech support (see Tolmie & Crabtree’s paper “Deploying Research Technology in the Home”), explainer, etc. This is all quite interesting and potentially useful. However the critical part of such a paper comes with the recommendations it makes in light of this. While most are useful, there is a a point that I feel Johnson et al.’s paper starts to diverge strongly with the ways in which we conceived of a route forwards for field trials. In particular Johnson et al. argue for a ‘reflexive mode’ of approach to doing field trials that extends beyond its existing argued-for application within ethnography (see Rode’s paper “Reflexivity in Digital Anthropology” and my previous blog):

“Although this call for more reflexivity has mainly been addressed towards ethnographers studying existing contexts, it is just as relevant to those doing studies of deployments of new technologies in-the-wild. Perhaps, more so, because deploying a new technology requires hands-on intervention into the daily routines of a setting.”

In other words, Johnson et al. are promoting the notion that ‘doing being reflexive’ when investigating a technology prototype deployed in an ‘in-the-wild’ setting has benefits. Specifically, they argue that it “challeng[es] the investigator to question their own role in the research” and that “[c]urrent methods employed in in-the-wild studies do not normally show this level of reflexivity or consideration for the social and personal context beyond the boundary of the interaction being studied”. But—so what? Why should investigators be more ‘reflexive’ (if they can be)?

To flip back for a moment to our “Into the wild” paper, at its core was an explication of just what the work of the field trial is, and what it does (and can do) for delivering research findings. In particular we explored a bunch of issues relevant in that work: 1) demand characteristics, a concept from psychology that describes, broadly, issues of interactions between experimenters / investigators and participants in field trials impacting the running of the trial (although we were not studying an experiment in the traditional psychological mode per se); 2) the role of lead participants, that is, the ways in which a subset of participants may consistently produce the most interesting results for research reportage; and 3) the interdependence of methods and results, i.e., reconfiguring how field trials are seen with factors like demand characteristics and lead participants as crucial for the research work. Thus, we argue that, instead of seeing these things as ‘problems’ for trials, they are instead some key normative orientations to reconfigure (e.g., that systems trials aren’t just ‘about’ success or failure, that participants can also be seen as co-investigators), and some benefits that should be accounted for (e.g., in terms of research communication, better ‘methods’ sections for papers) which may well also be usefully exploited (e.g., purposefully locating good ‘lead participants’). None of these is about entering a reflexive mode in order to do better field trials—instead it is about simple strategies for delivering greater practical value when doing field trials.

Whereas we were interested in understanding the work of the field trial as a methodical, practical engagement with participants, Johnson et al. seem interested in pursing the idea of ‘being reflexive’ itself as a method for ‘doing better trials’. In part I think the problem stems from a profound confusion between reflexivity and reflection, which are terms used almost interchangeably in Johnson et al.’s paper. While I would expect researchers to be reflective (surely this is their primary activity; which researcher would want to be accused of being ‘unreflective’?), reflexivity on the other hand, as a term, indexes a bunch of quite unrelated potential possibilities. This is where things get sticky. The most likely candidate form of reflexivity is something along the lines of ‘standpoint reflexivity’ that is found in many postmodern accounts of cultural theory. In this way of thinking is referenced by Johnson et al. via Dourish & Bell’s book “Divining a Digital Future” in which they argue for the importance of accounting for “the author’s stance”. In this line of thought, ‘deconstructing’ the author’s stance is seen as part of a commitment to ‘radical’ theory (which normally preoccupies itself with a set of standard sociological interests).

Interestingly, not once in our paper do we mention ‘reflexivity’, most probably because we do not see reflexivity as a specialised activity that is engaged in by researchers (instead, we’d probably point to the reflexivity of accounts; as Mike Lynch puts it “[t]o imagine an unreflexive action would be like imagining a sound without amplitude”). While there is nothing wrong with this ‘reflexive’ approach per se, there is typically little to recommend it in terms of its potential contribution to design in HCI (although I remain open to being convinced).

There is a wider game being played here too, specifically the ingress of ever more theory borrowed from other disciplines and brought into HCI. Lynch argues that “[w]hen we recognize that there is no single coherent division between reflexive and unreflexive discourse, then reflexivity loses its metaphysical aura and (apparent) ideological potency for empowering theories and rallying movements”. Although Johnson et al. have meshed our particular paper within this nexus of various rallying ‘calls to reflexivity’ that have been heard recently in HCI, our approach is anything but.

Placebooks community media toolkit launched at Royal Geographic Society

Wed, 18 Jul 2012 06:21:00 -0400

Placebooks is a community media toolkit, and is based around a web service and a mobile application design to support a range of activities in and around rural communities. Placebooks developed as part of the 18 month Bridging the Rural Divide project, an RCUK-funded project base primarily at the Mixed Reality Lab, University of Nottingham and the FITLab, University of Swansea. The project explored notions of ‘rural computing’ and ways to support rural activities, rural economies and rural life.

The Placebooks platform is currently undergoing handover between ourselves as researchers and the People’s Collection Wales, a Welsh Government initiative that has built a bilingual web service to collect, interpret and display the story of Wales. Placebooks was officially launched in July 2012 at the Royal Geographic Society, presented by members of the project team to an audience that ranged from the BBC and local community groups to academics and research councils. The presentation (pictured below, photo by Giles Lane) was followed by a panel.

Placebooks itself enables users to piece together all sorts of media types from disparate sources into an interactive digital booklet. These media types can be maps, images, text, video, audio, etc. drawn from local storage (e.g., the user uploading a photo from their hard disk), or hoovered up from external web services like Everytrail, DropBox, or the People’s Collection Wales. These digital booklets can be published on the website, shared via a range of means (e.g., social media), and downloaded to a mobile application (Android and iPad versions) as a complete offline ‘package’ for use in-the-field.

An important aspect of a Placebook is that it is described by geometries, meaning that as a whole the Placebook itself may be tagged to a particular location or region. Equally, its constituent parts may also be paired with locations, areas or regions that they relate to. In this way Placebooks provides a rich (and easy) way of creating interactive booklets that are infused with a range of geographic information. The system is generic enough to serve a range of purposes and communities.

The project was PI-ed and CI-ed by Matt Jones, Andy Crabtree and Alan Chamberlain, and was run in coordination with various partners such as Ordnance Survey, the Countryside Council for Wales and towards its end, the National Museum Wales and the People’s Collection Wales.

Placebooks itself was built largely by Kevin Glover (who developed the web interface), myself (server backend stuff) and Mark Davies (mobile application) with additional work on integrating external data sources, such as Everytrail, by Mark Paxton.

The development of the software was driven in a number of ways. In the first instance we used some ethnographic work from Peter Tolmie as a way of shaping ideation. This was coupled with having Alan as researcher situated / embedded in the environment and communities in which we were siting the design. This ‘action research’ approach supported the generation of requirements (specifically, we focussed on the Borth and Ynyslas area). Through a continual process of local participant engagement via Alan we were able to direct and shape the software development process, and perform elements of participatory design. This process was also strongly collaborative within the team in terms of co-designing the whole system to quite a fine level of detail—so although only a few of the team were actually building the software, its design was fundamentally ‘worked through’ as a collaborative process. This approach, coupled with significant user engagement work, meant that a wide range of prospective users were represented in the system design and at the same time had a proxy (Alan) with whom it was possible to critically influence how work progressed.

Confusing computations

Mon, 09 Jul 2012 21:06:00 -0400

As a rule, our world is largely dominated by digital computation. We have probably mostly forgotten what analogue computation looked like and its fundamentally different character to digital computation. Recently I read a CHI paper (with a Best Paper Award) by authors Fernaeus, Jonsson and Tholander, called “Revisiting the Jacquard Loom: Threads of History and Current Patterns in HCI”, published at CHI 2012 which seemed to make just this error of exoticising analogue computations, and then confusing them with digital computations in a problematic manner.

The central conceit of the Fernaeus et al. paper is pretty interesting: an assessment of past manufacturing machinery through the lense of modern HCI in order to ‘report back’ insights for its modern development. There is much to recommend in this paper in terms of its novel approach. It is possible that the Jacquard Loom may well have things to teach us, perhaps about the role of expert operation of machinery. However, while it is a novel way to view modern computing in a different light, if we wish to perform this kind of trick, it is strictly important that extra care is taken to consider the effect of that conceit on the topics that are being drawn into the historical comparison, which here are issues of the digital and the physical, tangible user interfaces, and the actual nature of computation.

The paper abuses (perhaps too strong a word, but the meaning is right) the nature of computation, and its specific qualities. It does this by playing quite loose and fast with the history of computing, and through this, the very nature of the relationship between the loom and digital computing.

We are quickly introduced to the notion that the loom and digital computation have some shared attributes through the common input format of punched cards. The looms were fed punched cards which dictated which threads would get pulled where, and thus resulting in a particular pattern described by the punched cards. Early computers also used punched cards as input devices—however I would argue that this is the point where any similarity or possibility of useful analogy stops since the loom used the cards as analogue instruction, whereas early digital computers used the cards as digital information. While the authors admit that “no actual computation [was] performed using the cards”, Babbage’s difference engine and analytical engine are brought in as relevant. The fact that the analytical engine was Turing complete and digital programmable (albeit mechanical) is not drawn out as a key issue. To have done so would be recognising the substantive categorical difference between analogue and digital computers. Instead we get an implied link between the loom’s design and “designers of new computational artefacts”. This is getting into non sequitur territory. Digital and analogue computation are fundamentally different.

Developing this blurring, we get further examples, including the abacus, side rule and player pianos, each of which is extolled as being significant for the argument because “they are based on ingenious compositions of material and mathematical coding schemes, alongside simple methods of manipulation”. Firstly there is the loose play here of the term ‘coding’ (do they really mean to compare this with programming? The reader is left uncertain). Secondly (and reiterating an earlier point), artefacts such as these are fundamentally incomparable in computational terms with a general digital computer since they are, like all analogue computers, intended to perform one form of ‘computation’ only, and cannot be reconfigured in any general purpose manner.

Evidence of this confusion is also found in a characterisation of how we might view the role / relevancy of punched cards: “The mechanics of the punched cards could be regarded as the birth of the binary representation, making it possible to ‘digitize’ material objects, creating a form of ‘code’ only possible to interpret by running it through a mechanical device.” Further slippage is found with the statement that the loom “clearly executes a form of code” and that we must search for a way of looking at the loom in which the “whole process [may be seen as] programming”. At the same time the authors admit the non-digital nature of the loom, but don’t pause to consider its significance. Could it be that the analogy can only provide so much purchase?

In many ways this confusing approach—the punched cards are seen as ‘digitising’ but at the same time the loom is not ‘digital’—is strangely reminiscent for me of an increasingly popular approach in HCI called ‘defamiliarisation’. The theory here is that so-called ‘defamiliarisation’ provides a technique by which we can attempt to gain novel insight through ‘making strange’. Here the loom offers insight into computation by making it strange through problematising its very foundations. Our sense of the digital, the general purpose computer, the Turing complete artefact, etc. is thrown into disarray. In this logic, ‘making strange’ enables new views of that which is familiar, here mundane digital computation, and therefore grist for the design mill. And the mill in question here is tangible computing.

Continuing to stretch the very concept of ‘code’, the paper goes on to a key discussion point on approaches to tangible interactions: “[t]he [loom’s] code is made visible through a completely transparent physical construction [on punched card], while in modern day computers, the machine code is hidden from user in layers of abstraction”. Instead of being seen as intrinsic to the very nature of digital computation, in the confusions of analogue mechanical devices versus digital computation-based, abstraction somehow, somewhere, has been transformed into a ‘problem’. Even though it is arguably fundamental if we are to use digital computers as a medium for anything.

The paper reaches a closing point that “physicality does not necessarily imply understandability. In a sense, the preoccupation with making machinery understandable is somewhat peculiar to HCI, and can be contrasted to other values such as efficiency and productivity”. But understandability is not in any sense an oppositional value to efficiency and productivity, as it appears to be portrayed here. And, furthermore, HCI has since time immemorial been hugely concerned with efficiency and productivity (ergonomics background, canonical usability studies, huge swathes of CSCW, etc.), and less about ‘understandability’.

In the end, the authors seem beguiled by the complexity of the loom and the skills required to operate it and that it can demonstrate for tangible computing an interface that is complex yet “graspable”. This is ironic, of course, since the Jacquard Loom, through automation, made loom work simpler.

My paper Envisioning Ubiquitous Computing was presented at CHI...

Fri, 01 Jun 2012 10:19:23 -0400

My paper Envisioning Ubiquitous Computing was presented at CHI 2012. However, as I was unable to attend, fragments of this video were used Steve Benford, who kindly gave the paper on my behalf. (Note that I have used a range of images scraped off the web for this presentation and thus the copyright is probably dubious.)

HCI peer review in crisis: what is science in HCI?

Tue, 08 May 2012 09:34:00 -0400

HCI is in continual crisis mode with regard to peer reviewing. Every year around the time of the CHI conference - or to give it its full, unwieldy and faintly outmoded title The ACM SIGCHI Conference on Human Factors in Computing Systems - there is considerable debate about the reviewing process (see the aside at the bottom of this posting for more detail on the reviewing process itself). No doubt similar conversations take place at all other major HCI and related venues (e.g., CSCW), but it seems that the ones from CHI leak onto the web most readily. Other conferences have comparable review processes (CSCW, DIS, Ubicomp), so the conversation here is relevant beyond CHI.

It seems to me that a lot of these conversations revolve around a number of topics related to the review process itself, such as the relative weighting of reviewer and author power, the accountability of activities of programme (sub)committees, the way reviewers get selected, the rising numbers of submissions to the conferences, and (relatedly) the acceptance rates and submission types. As an author it can be incredibly frustrating if the account of why your paper was rejected is scant on detail (such as reports that your paper was discussed at the committee meeting, but the actual detail of the discussion was left out), and correspondingly as a reviewer it can be annoying to have to review work which does not seem to even hit the lower level of what could be considered worthwhile.

In response to participating in the process of peer review, Jeffery Bardzell recently wrote an interesting blog posting outlining the various ways in which not only practical changes could be made to this reviewing process, but also presented some ways the HCI community could reconceptualise what process it is actually engaging in during peer review. Bardzell presents this in his “Position on Peer Reviewing in HCI” (in three parts), and formed a lengthy response to Antti Oulasvirta’s original blog posting on “Why your paper was rejected” with specific reference to CHI.

Oulasvirta’s blog here is key in understanding the meaning of the exchange and in assessing some of the underlying concerns that are perhaps helping animate the discussion. Oulasvirta, as a participant in a subcommittee of CHI 2012, summarises some of the key problems that result in low scores being awarded to some kinds of paper submitted to CHI. He focussed particularly on what he terms “flaws in empirical work”, including such things as: errors in research strategy (i.e., applying the wrong research tool for the job); basic statistical flaws (interestingly Grounded Theory is included here related to errors in inter-coder reliability metrics); causation issues (e.g., problems with the `implies’ relationship); generalisability and replicability of findings.

Reading the blog post reveals Oulasvirta’s epistemic position on the nature of HCI and its relationship to the natural sciences. While it is not explicitly nailed down anywhere, this position seems to be that HCI (or at least parts of HCI) is a scientific discipline, and as such many submissions to conferences may be shot down on a range of scientific grounds, as outlined above. The notion of HCI as a science is grounded in and guided by popular conceptualisations of ‘what science is / what scientists do’. Specifically these are articulations of what is thought to be the essence of scientific practice, such as: the nature of larger problems is such that they may be broken down into subproblems and reassembled upstream (i.e., scientific reductionism); the mutually reinforcing relationship between theory and experiment, with each elaborating the other - experimentation provides ‘canaries in the mine’ for theory (e.g., see the controversy over string/M theory), and theory directs the ‘what next’ of experimentation; the role of replication and experimentation as separate or separable from context and situatedness. These essences from the natural sciences provide yardsticks with which to measure HCI activity against if it is to be (or remain) a normal scientific activity.

This view of HCI is not new, and often emerges in a range of different guises from HCI research. For instance, Bartneck reports how President of the Association for Computing Machinery (ACM) Stuart Feldman’s CHI 2007 opening speech included the assertion that HCI “is absolutely adherent to the classic scientific method”, which, he said, was “[n]ot a description […] of all the fields in computing”. Ben Shneiderman’s comment on Bardzell’s blog raises this issue again with the comment that “HCI is a scientific discipline”. More recently, the CHI 2012 conference is hosting an ongoing workshop series - RepliCHI - that is concerned explicitly with a cornerstone of the natural sciences, namely, replication. The workshop, expressing very common concerns within the HCI community, states that “it is not uncommon for research to be rejected in CHI for being ‘incremental’”, whereas “[i]n science, it is common for people to try and replicate discoveries, so that the community can confirm new discoveries”. There are other variations on HCI’s scientific status, such as James Landay’s blog posting on “giving up CHI/UIST” (comparable maybe to ‘giving up Christmas’ or ‘giving up on a relationship’?). This is a call for a wider notion of science and where it happens: “I think we have been blinded by the perception that ‘true scientific’ research is only found in controlled experiments and nice statistics”. A now-defunct blog posting by Joshua Kaufman (archive.org link) on a talk by Paul Cairns on ‘HCI as Science’ brings Popper’s conceptualisation of what science is to bear on whether HCI is science (e.g., via Popper’s notion of science as falsifiable). Interestingly one of the commenters argues that HCI is more like engineering: “The Wright brothers constructed a working airplane without knowledge of aerodynamics, fluid mechanics, etc. They just applied tried and true engineering best practices which they learned from working on bicycles (plus a ton of trial and error). In fact, several generations of planes were created before scientists started to grasp any of these principles, and the debate continues today. Yet we fly around in planes. They work, even if we don’t have all the ‘laws’ exactly nailed down.” The unstated assumption being that engineering something that interacts with the physical environment is somehow isomorphic with engineering of things that interact with and around humans (I would argue that they are not).

Bardzell’s blog appears to be a reaction in some ways to this implicit / explicit epistemological position on the status of HCI as a science. The perceived problems surrounding the peer review process mentioned at the start of this discussion are transposed into or driven by such epistemic issues. While it appears that Bardzell does not deny that there are scientific aspects to the reviewing process, rather than arguing for the scope of what is considered to be normal science to be enlarged (as Landay does), he suggests that we must attend to the social features of reviewing as they play an important (and overlooked) role in the development of scientific results. In sum he appears to be arguing that reviewing is by its nature “critique” more than (or perhaps instead of) being part of a rigorous scientific process that upholds the popular notions of ‘what science is / what scientists do’. (In some ways perhaps it is not much different from film criticism?) This argument obviously borrows strongly from constructionist accounts in Science and Technology Studies (STS) and Sociology of Scientific Knowledge (SSK) (e.g., Latour and Woolgar’s work), where the outputs of the work of the natural sciences is understood in terms of local practical circumstances (e.g., physical lab) typically filtered through the lense of external features such as politics, power structures, funding systems, etc. In my view Bardzell is linking reviewing work with a consideration of these sorts of aspects.

There are a great many interesting aspects of this discussion. However, it turns out that a lot of these concerns are not new, and not particularly unique to HCI. For instance, we can compare issues around peer review in HCI with some of the gripes emergent from SIGCHI’s parent community, i.e., SIGGRAPH. Ashikhmin expresses similar concerns about the review process, fashionable topics, results being presented in a normatively scientific manner in order acquire the cache of validity, reviewer power, acceptance rates, and assertions about computer graphics being a ‘scientific endeavour’. As Ashikhmin states “computer graphics is simply not a hard science and current attempts to present it as such are misguided at best […] the success or failure of any specific technique is, quite literally, in the eye of beholder”.

Another key aspect of the perceived problem is probably the way in which HCI mixes communities (which is a good, but challenging thing). A great deal of this discussion probably culture clash, perhaps between those coming at HCI from a cognitive science and / or psychology background and those from arts and humanities backgrounds (who are increasingly being involved in HCI). While those from such a non-‘science’ background may seek to conceptualise HCI in the mould of the way that the natural sciences are conceptualised in STS, so correspondingly those from the ‘labelled sciences’ (computer science, cognitive science, behavioural science, etc.) are perhaps guilty of mythologising of what the actual work practice involved in the natural sciences is like, or perhaps forgetting that HCI’s related fields like psychology have plenty of their own scientific demons (e.g., problems with WEIRD behavioural and brain science). While I would not necessarily want to align with STS, SSK, etc., particularly as they have a tendency to eschew studies of mundane practice and revert to something like structural functionalism (as noted by Mike Lynch in Scientific Practice and Ordinary Action), there is something to be said for the importance of the social features of scientific practice and its impact on the presentation of science. However, there is probably an equal tendency by those from a non-‘science’ background to presume that such social aspects are somehow hidden from scientific practitioners, and that they are unknowing ‘cultural dopes’ going about their work.

A nice example of the work of a discovering science is provided by Eric Livingston in his book Ethnographies of Reason. Through a simple procedure he unpacks the embodied and commonsense ways in which a phenomenal field may be investigated through various instruments employed to uncover it. In this account, natural phenomena are explored through practical action, and subject to everyday mundane practicalities. This leads to another issue: that of the relationship between the phenomena that are being investigated by HCI and the role of those self-same phenomena within HCI. As pointed out here in relation to Kaufman’s blog posting “HCI phenomena are constantly changing, HCI is constantly moving into new domains, redefining itself and absorbing new types of technology. Basically, there are no static phenomena so there can’t be an HCI paradigm. Furthermore, since there is no HCI paradigm, HCI is not a science.” This is perhaps not deep enough, though. In this situation we can consider the standard ethnomethodological critique. HCI’s phenomena of interest are actually the phenomena that enable us to study the phenomena in HCI in the first place. In other words, the commonsense methods of local, situated reasoning that are brought to bear in the work of HCI research often remain largely unexamined in spite of the fact that really this is the phenomena we are examining when we do a study. While not attending to these issues may be okay for the natural sciences, which may suspend interest in such matters in favour of transcendental principles of the natural world, for the human sciences (which HCI regularly brushes up against, being concerned with ‘interaction’), we ignore these issues at our peril.

Finally there is the legitimacy argument. Kaufman’s blog posting suggested that it was a good thing for “HCI [to] be considered a science because science was practically the only measurable form of progress in the 20th century”. Scientism faces us all, especially when it comes to gathering funding and establishing the legitimacy of our research work in HCI. In one view this could be seen as a cynical perspective on science (and perhaps devaluing the label), but on the other hand, we can instead see it as ‘strategic’ and necessary.

[An aside: for those who are not familiar with the CHI reviewing process, a simplified version is as follows. The conference is organised into various subcommittees which cater for different aspects of the field of HCI (this subdivison is of course a massive fudge, but practical). Authors target their paper to a given subcommittee; the subcommittee’s chairs then divvy up the papers between themselves and the associate chairs (ACs), who in turn recruit a number of reviewers (3+) for each of the papers they are AC-ing for. Reviewers do the reviewing, ACs write meta-reviews summing up the points made and scores provided by the reviewers, all of which are then sent to authors. Anonymity is top down and bottom up: authors are anonymised for reviewers, and the reviewers are anonymised for the authors. Reviewers are also anonymous between each other. ACs on the other hand know who the reviewers are. Authors, having received their paper’s reviews have the option of writing a rebuttal if they so wish. After the rebuttal period is over, reviewers can alter their review and score based on the rebuttal. The final set of reviews (sometimes augmented with further reviews for particularly divided reviewers) are then used to work out what papers get discussed at the subcommittee meeting. The ACs and chairs meet up, discuss all the papers they feel need discussing (a very opaque process but generally speaking papers below a certain average score are typically dismissed), and then collectively determine accept / reject decisions for all the papers for their particular subcommittee. Finalised reviews and final paper decisions, often including some account of `what went on’ at the subcommittee meeting, are then sent to authors, who are either left whooping with joy or crying in despair.]

Display ecologies

Tue, 21 Feb 2012 12:07:00 -0500

A while back (12th April 2011) I set up and ran a workshop at the Mixed Reality Lab on `display ecologies’. The workshop was an attempt to (1) survey existing work in the field and the state of the art as found in the literature; (2) to determine what research questions - whether they are design, conceptual or infrastructural - need to be answered; and (3) to work out what scope there is for innovation. After the workshop I wrote up a report that documented what was discussed, and what each speaker presented.

An excerpt from the report is below, but you can download the whole thing here.

The main purpose of the workshop was to explore public interactive display ecologies – often called ‘situated displays’. Notions of ‘situated displays’, however, are not wide enough a concept to capture the diversity of configurations and forms of physical display (as well as social aspects of ‘display’ and ‘displaying’) that we both encounter in the literature as well as located within the content of the workshop. Instead, it seems that an ‘ecology’ metaphor affords us greater purchase in understanding the design space.

Ecology evokes a range of useful concepts, such as how ecologies are composed of hybrid elements, their heterogeneity in that they fit together or are made to fit together and yet retain boundaries and seams. Overall, ecology as a metaphor is invoked to suggest the ‘messy’ reality of real-world configurations of display. ‘Ecology’ also implies display systems that exist in an ever-changing environment. It implies that these messy interrelated components might self-organise in some way. Ecology implies things might be brought into play that we didn’t expect - new objects and so on which might interact with the displays somehow but we didn’t expect them to. Ecology implies display elements that mesh more with non-digital resources and a wider understanding of what a ‘display’ is or might be.

CHI 2012 publication: "Envisioning ubiquitous computing"

Fri, 20 Jan 2012 11:27:00 -0500

My paper “Envisioning ubiquitous computing” is to be published at CHI 2012. You can grab a PDF of it here. Full reference and abstract below.

Reeves, S. Envisioning ubiquitous computing. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, May 2012.

Visions of the future are a common feature of discourse within ubiquitous computing and, more broadly, HCI. ‘Envisioning’, a characteristic future-oriented technique for design thinking, often features as significant part of our research processes in the field. This paper compares, contrasts and critiques the varied ways in which envisionings have been used within ubiquitous computing and traces their relationships to other, different envisionings, such as those of virtual reality. In unpacking envisioning, it argues primarily that envisioning should be foregrounded as a significant concern and interest within HCI. Foregrounding envisioning’s frequent mix of fiction, forecasting and extrapolation, the paper recommends changes in the way we read, interpret and use envisionings through taking into account issues such as context and intended audience.

CHI 2012 publication: "A hybrid mass participation approach to mobile software trials"

Thu, 19 Jan 2012 18:38:00 -0500

This paper was developed from work on the Designing the Augmented Stadium project and will be published at CHI 2012. It is available as a PDF from my website.

Morrison, A., McMillan, D., Reeves, S., Sherwood, S., and Chalmers, M. A hybrid mass participation approach to mobile software trials. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, May 2012.

User trials of mobile applications have followed a steady march out of the lab, and progressively further ‘into the wild’, recently involving ‘app store’-style releases of software to the general public. We examine the literature on these mass participation systems and identify a number of reported difficulties, which we aim to address with a hybrid methodology combining a global software release with a concurrent local trial. A phone–based game, World Cup Predictor, was created to explore the uptake and use of ad hoc peer-to-peer networking, and evaluated using our hybrid trial method, combining a small-scale local trial (11 users) with a ‘mass participation’ trial (over 10,000 users). Our hybrid method allows for locally observed findings to be verified, for patterns in globally collected data to be explained and addresses ethical issues raised by the mass participation approach. We note trends in the local trial that did not appear in the larger scale deployment, and which would therefore have led to misleading results were the application trialled using ‘traditional’ methods alone. Based on this study and previous experience, we provide a set of guidelines to researchers working in this area.

Schutz's argument in "Tiresias, or our Knowledge of Future Events"

Mon, 14 Nov 2011 18:09:47 -0500

The point of this particular posting is to bring to attention some of Alfred Schutz’s writings on the use of ‘futures’. Recently someone pointed me towards it due to my interest in understanding the role of futures in HCI and ubiquitous computing design.

Tucked away at the back of Volume II of Schutz’s Collected Papers is his essay on “Tiresias, or our Knowledge of Future Events”. In this essay, Schutz describes the inherently social ways in which understandings of the future are developed by members of society, as analysed from his phenomenological position. (Reading his work rather reminds me of Goffman’s style—-rigorous and relentlessly detailed, drawing extensively from the phenomenological experience of the everyday, but without an obvious scientific ‘empirical’ derivation.)

One of Schutz’s key concept is the ‘stocks of knowledge’ that are developed by members of society in and through their everyday activities. I say ‘stocks’ because these are locally-rational, situationally-generated sets of knowledges that are relevant to the tasks-at-hand.

In this paper his argument seems to be that these knowledges play a fundamental role in the ways that we anticipate, prophesy, predict, forecast, guess, and intuit what will happen in the future and how we plan for those future events and determine their predicted unfolding. These situationally-relevant stocks of knowledge are developed through streams of past and present experiences. Although they are used as a ‘prediction space’ for the future, the same time, these knowledges are necessarily bound (and can only ever be) to the past. As Schutz states:

All projecting consists in an anticipation of future conduct by way of phantasying. It is, to use Dewey’s pregnant description of deliberation, “a dramatic rehearsal in imagination.” Yet projecting is more than mere phantasying. Projecting is motivated phantasying, motivated by the anticipated supervening intention of carrying out the project. The practicability of carrying out the projected action within the imposed frame of reality of the is an essential characteristic of the project. This refers, however, to our stock of knowledge at hand at the time of projecting. Performability of the projected action means that according to my present knowledge at hand the projected action, at least as to its type, would have been feasible if the action had occurred in the past.

It seems to me that the problem he is raising here is that future anticipations are inherently tied to these past and present stocks of knowledge, that that our understanding of the way the future may possibly develop is also bound to how these knowledges developed, but only in the past. Schutz calls this a kind of “anticipated hindsight”, i.e., that future prediction, foresight, forecasting, etc. is nothing less than hindsight transposed to a future framing. Or in other words, “we cannot expect any event of whose typicality we have had no pre-experience”. Instead:

Once materialized, the state of affairs brought forth by our actions will necessarily have quite other aspects than those projected. In this case foresight is not distinguished from hindsight by the dimension of time in which we place the event.

After the future event has (or has not) transpired, these experiences become part of the present-past stocks of knowledge:

Yet because of their very typicality our anticipations are necessarily more or less empty, and this will be filled in by exactly those features of the event, once it is actualized, that make it a unique individual occurrence.

These knowledges are also social, i.e., for them to have utility they must be shared, and known-in-common by members of society. In this way,

validity of anticipations […] is founded on the assumption that some or all of my fellow-men will find in their stock of knowledge at hand typically similar elements, and that these will determine the motives of their action.

This is all pretty abstract. But it can be understood in terms of how it can pose us with some potential problems when we are thinking about what technologies we can, might, or will build, and what the impacts of those technologies can, might, or will be. Perhaps not ‘problems’ as such, but rather necessary restrictions and limitations of the kinds of technologies we can possibly imagine, and how we treat them should they actually ‘arrive’. In my reading, Schutz’s argument gets us to the point where we appreciate that our anticipations and expectations about future technologies are derived from sets of existing typifications that are fundamentally based in the past. These typifications are ‘empty’, and ‘filled in’ with the specificities and peculiarities should the event actually transpire. In this way all visions of future technologies and societies are, and can only ever be, deeply embedded in the past, and the dimensions of feasibility understood as derived from the past. This is perhaps why many visions of the future are so bland and unadventurous.

Pervasive Monuments deployment in Slovenia

Mon, 22 Aug 2011 10:35:00 -0400

Recently in collaboration with Halfman Design, and in coordination with Slovenian organisations SCNR and NUK, we developed and deployed a simple interactive site-specific audioguide at a woodland mass grave memorial (at Kočevski Rog) in Slovenia. ‘Spomenik’ was designed to be very accessible (requiring only standard mobile phone functionality—voice and text), and work in a quite ‘remote’ (in infrastructure terms) forest setting. The audioguide was powered by Tropo, a cloud telephony service. A set of images from Halfman Design explains the experience better than I could, so I have presented some of them here with annotation.

The memorial site we focused on featured a large cave (in which killings took place), and a variety of unofficial and official memorials, including these bell monuments:

The path into the woods consisted of two distinct ‘stations’ on which we placed signs highlighting our audioguide phone number. During our deployment a few groups of Slovenian students visited the site and used the system:

Here is the first station indicating the start of the path into the woods:

Visitors then continue into the woods:

And reach the next station, which is sited by the mass grave:

There is a large collection of unofficial and official memorials at the grave site, including the bells around which we based the phone experience:

Physical visits to the site, and reflections recorded by visitors are made available on a simple website:

We are currently in the process of analysing the interactional aspects of this work.

The scripts that run it are available on my github.

Adam Greenfield’s Everyware is the mediocristan edition. I...

Tue, 02 Aug 2011 18:35:00 -0400

Adam Greenfield’s Everyware is the mediocristan edition. I wonder what an extremistan edition would have looked like.

Studying social organisation with video

Tue, 12 Jul 2011 10:24:00 -0400

Studying social organisation with video:

Notes from a talk I gave on the 7th of July at the Research Squared Conference 2011 on Data Analysis. The abstract is below:

This report was developed from an invited talk I gave for the Research Squared Conference 2011 on Data Analysis held at the University of Loughborough. It examines the role of video within qualitative empirical studies of social conduct and organisation. I focus particularly on the use of video in ethnographies of technology use (mainly because this is my area of interest), and attempt to communicate why video is used to study social interaction. The report provides a broad overview of a particular kind of video use found in various different ethnographic traditions, and particularly focuses on instances within the fields of human-computer interaction, and computer supported cooperative work (e.g., workplace ethnographies of coordination and collaboration around technological environments). I’ll look at how video is used, practically, and what techniques and procedures are often employed. We will also cover briefly some underlying commitments to ethnomethodological or conversation analytic approaches that pervade studies of this kind. Finally I’ll look at what kind of findings you get from using it by covering a number of well-known studies which used video extensively, and consider what kind things you should be aware of when using video, such as various practical matters, caveats, and so on.

Reflections on reflexivity

Thu, 07 Jul 2011 19:02:00 -0400

There is an ongoing debate over ethnography in HCI and its attendant fields (e.g., CSCW) that finds its yearly outpouring at CHI (the SIGCHI conference on human factors in computing systems). Three obvious and recent points of reference are Dourish’s “Implications for design” paper in 2006 [1], in 2009 Crabtree et al.’s “Ethnography considered harmful” [2], and most recently, Dourish and Bell’s book Divining a Digital Future (2011) [9]. This debate has been cast more or less as being about both the kind of commitments that ethnographies are conducted with, and how that meshes with—particularly regarding what ethnography does for—design.

The latest edition of the saga is found in Rode’s paper “Reflexivity in digital anthropology”, recently published at CHI 2011 [8] (see Taylor’s paper for an interesting counterpoint [11]).

Broadly speaking, the paper (from my reading) develops the argument that there is a shared, complementary validity to the exercise of different forms of ethnography that we find at venues like CHI. In order to advance this argument, the paper structures its analysis heavily on three characterisations of ethnography developed by van Maanen [12]. Unfortunately for the reader, the paper glosses over the various commitments with which these ethnographies can be exercised (ranging from actor network theory, grounded theory, distributed cognition, interactionist perspectives, and ethnomethodology). Nevertheless, the paper engages in a discussion of van Maanen’s “realist”, “confessional” and “impressionistic” forms of ethnography; as follows:

Quoting van Maanen, Rode states that “[t]he realist tale is ‘a documentary style focused on minute, sometimes precious, but thoroughly mundane details of everyday life among the people studied,’ which are carefully structured to support authors’ points. Details give the impression of reliability.” [8]
Whereas, impressionist ethnographies “capture a moment in time from a uniquely individual perspective” [8], although they appear to be primarily to do with scene-setting before more ‘realist’ ethnographic mode of accounts proceeds.
Apparently “more reflexive” (and therefore presumably ‘better’) are confessional ethnographies, in which ethnographers “engage with the nagging doubts surrounding the study and discuss them textually” [8].

It becomes apparent quite quickly that, of course, amongst what Rode characterises as ‘realist’ covers a range of kinds of ethnographies with all sorts of different commitments. However, part of what is pointed at particularly by citation is the ethnomethodologically-informed strain of ethnography and its progeny that is represented (with varying underlying commitments) regularly at venues like CHI, CSCW, ECSCW, DIS, and so on. So, for the purposes of this discussion, I’ll naively assume Rode is talking in part about ethnomethodologically- and (often) conversation-analysis-informed ethnographic work, such as examples from Heath et al. [3], Bentley et al. [4] and Crabtree et al. [5].

And, although the paper is shielded by ‘why can’t we all get along?’ bookends and an appeal to the complementarity of different ethnographic forms, the meat of the argument is concerned with advancing a critique of this so-called ‘realist’ form of ethnography. Simply, this argument is that ‘realist’ work, through being written to provide “an impression of reliability”, can be deficient thanks to a lack of ‘reflexivity’ (in the cultural anthropological sense [10]) which to varying extents both ‘impressionistic’ and ‘confessional’ ethnographies supply.

What is meant by increased ‘reflexivity’? Arguments for reflexivity demand that ethnographers begin to address more of the context in which they conduct their ethnography, attending to and describing for the reader their personal background, their rapport with ‘informants’, and the way in which ethnographers themselves interpret and make sense of settings (and convert this into a textual version for the ethnographic report) during participant-observation. Of course, Rode (and presumably van Maanen) claims that ‘realist’ ethnography is ‘unreflexive’, and that it “black boxes” much of what is going on within ethnographic study. Lynch expands nicely on this:

According to a common constructionist formula, objective expressions rhetorically delete, disguise or place in a ‘black box’ the tacit knowledge, cultural origins and epistemological limits of knowledge. A theoretical, and even moral, imperative for those who follow the constructionist programme is to open up the ‘black box’ to expose the local origins and cultural limitations that have been deleted. But if an initial statement is held to be unreflexive, then the question is: How much more needs to be added to it before it becomes reflexive?’ [10]

However, here emerges the problem: in contrasting ‘unreflexive’ realist ethnography with these more explicitly reflexive forms, the assumption is that in providing such increased reflexive detail about the “nagging doubts” of the ethnographer, the ethnography will somehow be ‘better’ or ‘truer’, more useful or even more insightful. Although it is doubtful ‘reflexive’ ethnographers would agree with the following statement, the net result boils down to reflexivity being an attempt to provide a privileged position to the ethnographer’s observations, insights and reports from the field. The mechanism through which this is done is by providing increased disclosure about the surrounding context of how the ethnography was done, and related ‘reflexive’ details.

However, although never explicitly stated in this manner within the paper, and glossed as helping us better understand “the real world appropriation of technology”, it’s hard to determine exactly what reflexivity is supposed to do for us besides, grossly, some kind of ‘improvement’. As with many calls to increased reflexivity, the implicit assumption is that a reflexive version is somehow advancing the work of ethnography, although it appears difficult to articulate exactly what it is actually working towards improving. Instead, as Lynch suggests, we can see reflexivity as a method for ethnographers to construct this appearance of a somehow more privileged, accountable position, which is a particularly pressing and relevant concern in the fight for academic survival (as hinted at by Rode’s gripes in the paper about these forms of ethnography encountering trouble during peer review). Lynch argues that notions of reflexivity are typically deployed as “polemical efforts to promote theoretical and methodological advantage in a noisy field” (although to some extent Rode steps back from this argument with the notion of complementarity). Ironically we end up circling back to the earnest desire for increased ‘objectivity’ that reflexive ethnographers began reacting against in the first place—just that it’s ‘objectivity’ by another name, specifically in mobilising the veracity cache associated with doing ‘being reflexive’ in a study.

The bottom line here is that, in the end, there are no guarantees that increased reflexivity does anything, and is not a method for producing a ‘good ethnography’:

its cogency will depend upon what it says about its topic and whether it persuades relevant audiences. Depending on the case, [reflexivity] may come across as insightful, witty, convincing, unconvincing, boring or silly.

[…] attempting to be reflexive takes one no closer to a central source of illumination than attempting to be objective [10]

As Lynch points out, ethnomethodology’s approach to reflexivity provides a radical recalibration of understanding in contrast with the hang-wringing forms of reflexivity routinely found in anthropology, STS, and other social sciences. Instead of viewing reflexivity as something that can be somehow added to an ethnography, we should instead view reflexivity itself as the very core phenomena under investigation within our ethnographies.

Social interaction is intrinsically reflexive, in that the procedures and methods we use in order to make our actions accountable (observably and reportably so) to one another are the selfsame actions that “produce and manage settings of organized everyday affairs” [6, p. 1]. For ethnomethodology, “reflexivity is ubiquitous and unremarkable” [10]. Thus, Livingston argues that we should “pursue reflexive phenomena, not yet knowing and waiting to discover, for any particular domain, what such an interest may refer to […] if […] such phenomena are missing from a study, we know we’re in serious trouble” [7].

There is a wider issue at work here with this argument about reflexivity, however, particularly regarding the wholesale importation of cultural theory (via media studies and so on) into the field of HCI. Indeed, the signature of these new ‘reflexive’ ethnographies increasingly gaining exposure within CHI and other venues involves “gendered, racial and cultural ‘standpoints’ that provide existential conditions for reflexive critiques of dominant discourses” [10].

The application of such theory to ethnographies is problematic, as highlighted by Crabtree et al. [2]. By using theory as a lense, we lose the particularities of the settings we study. In the end, we end up knowing less about them, and ethnographies like these end up being about the analyst, about their academic discussions, and not about routine, everyday, observable and reportable lifeworld of those settings. This is detectable in what Rode states in the paper: “[f]or instance, in my recent work on gender I went to the field having thoroughly read up on feminist theory”.

Such practices run into danger in that the ethnography becomes a discussion around theory, rather the materialities of the setting, resulting in, say, Rode’s feminist theory being merely applied to the latest situation, and ending up with the same old story but with a scenery change. The commitment to this attitude becomes clearer as Rode further states, “impressionist text drawing from months of fieldwork can strengthen theories by grounding it in detailed examples”—i.e., fieldwork becomes a merely servant to elaborating a particular theory. As Hughes et al. point out, “concepts that sociologists routinely deploy […] are abstractions which serve sociological purposes, and only sociological purposes” [13].

So, far from “black boxing” social order, ethnomethodologically-informed so-called ‘realist’ ethnography seeks to practically engage with exploring taken-for-granted aspects of how that social order is constituted. Instead, it is the application of standard sociological theoretical preoccupations and narratives—such as gender, class, power relations or culture—that actually “black box” the material, actual ways in which the sites of anthropological studies are organised and made orderly.

Rant over.

References

1. P. Dourish. Implications for design. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI), pages 541-550, New York, NY, USA, 2006. ACM.

2. A. Crabtree, T. Rodden, P. Tolmie, and G. Button. Ethnography considered harmful. In Proc. CHI ‘09, pages 879-888, New York, NY, USA, 2009. ACM.

3. C. Heath and P. K. Luff. Collaboration and control: Crisis management and multimedia technology in London Underground line control rooms. Journal of Computer Supported Cooperative Work, 1(1-2):69-94, 1992.

4. R. Bentley, J. A. Hughes, D. Randall, T. Rodden, P. Sawyer, D. Shapiro, and I. Sommerville. 1992. Ethnographically-informed systems design for air traffic control. In Proc. CSCW ‘92. ACM, New York, NY, USA, 123-129.

5. A. Crabtree, S. Benford, T. Rodden, C. Greenhalgh, M. Flintham, R. Anastasi, A. Drozd, M. Adams, J. Row-Farr, N. Tandavanitj, and A. Steed. Orchestrating a mixed reality game `on the ground’. In Proc. CHI, pages 391-398, New York, NY, USA, 2004. ACM Press.

6. H. Garfinkel. Studies in Ethnomethodology. Prentice-Hall Press, 1967.

7. E. Livingston. Ethnographies of Reason. Directions in Ethnomethodology and Conversation Analysis. Ashgate, 2008.

8. J. A. Rode. Reflexivity in digital anthropology. In Proc. CHI ‘11, pages 123-132, New York, NY, USA, 2011. ACM.

9. P. Dourish and G. Bell. Divining a Digital Future: Mess and Mythology in Ubiquitous Computing. MIT Press, May 2011.

10. M. Lynch. Against reflexivity as an academic virtue and source of privileged knowledge. Theory, Culture & Society, 17(3):26-54, June 2000.

11. A. Taylor. 2011. Out there. In Proc. CHI ‘11. ACM, New York, NY, USA, 685-694.

12. J. Van Maanen. Tales of the field; on writing ethnography. Chicago UP, 1998.

13. J. Hughes, D. Randall, M. Rouncefield and P. Tolmie. Meetings and the accomplishment of organization. In Ethnomethodology at work, M. Rouncefield and P. Tolmie (eds.), Ashgate, 2011.