On replication in HCI
At CHI 2013 I attended a 2 day workshop on ‘replication in HCI’ (also known as ‘RepliCHI’, a programmatic concern reflected in various workshops and panel discussions at prior CHI conferences). To summarise the purpose of the workshop I will turn to the workshop proposal abstract, which does a fine job:
The replication of, or perhaps the replicability of, research is often considered to be a cornerstone of scientific progress. Yet unlike many other disciplines, like medicine, physics, or mathematics, we have almost no drive and barely any reason to consider replicating the work of other HCI researchers. Our community is driven to publish novel results in novel spaces using novel designs, and to keep up with evolving technology. The aim of this workshop is to trial a new venue that embodies the plans made in previous SIGs and panels, such that we can begin to give people an outlet to publish experiences of attempting to replicate HCI research, and challenge or confirm its findings.
Broadly speaking the position I presented (to be found elaborated in the paper Christian Greiffenhagen and I submitted) was intended to problematise the assumptions built into this notion of ‘replication’ in HCI. One of the key problems we saw was not necessarily that there is anything at issue in asking HCI to tolerate / involve more replication. Rather, that framing this in terms of making HCI more ‘scientific’ is possibly based on a mythical view of ‘good science’ of which “replicability of published research findings […] is often considered a cornerstone of progress” (see above). Our general position was that sociology of science warns us against holding a mythological view of replication in the natural sciences and then applying this to HCI. Replication of results in, say, physics, serves a highly motivated and particular purpose in working through contested parts of the discipline, rather than being a practice engaged in as a matter of course in order to be seen ‘doing normal science’. Further, the existence of any ‘decisive replication’ becomes problematic in that there is not necessarily any standard for what is considered a valid replication before said replication is attempted.
The workshop’s submissions discussed a range of instances attendees at the workshop considered examples of ‘replication’. But there were a number of nagging issues that continually arose (note that those documented below are not necessarily things voiced by me during the workshop).
There were considerable problems arising from the idea of classifying different ‘types’ of replication (the original workshop proposal identified four: “direct replication”, “replicate and extend”, “conceptual replication” and “applied case studies”). The workshop organisers themselves admitted to a significant amount of argument in even coming up with them.
This replication type became more confusing when considering if the intent of the replication was to replicate original research findings, the original research method, or both. Thus it turned out that the intent of the replicating authors was important for articulating what any given replication is ‘doing’, i.e., in terms of whether it is testing and validating ‘the instrument’ used to generate particular findings (e.g., a methodical procedure), or testing the findings themselves.
Another matter about classification was ‘self-replication’. A few presenters described what were essentially replications of their own prior work, typically with some extension. I found these quite problematic: surely for replication to have any meaning, it must be understood as a fundamentally social, negotiated process? Self-replicating an experiment or trial is not really getting to the heart of what ‘attempting to replicate findings’ is doing for negotiating contested results between members of a research community.
Thus, the important underlying issue for classification schema here is that it was unclear whether a particular set of HCI researcher activities constituted, could be classed as, or could be recognised by others as, a replication at all.
In one view, it seemed that the more details unearthed by a replicating author the better: increased fidelity of the replication could be achieved through re-creating the environment in which the original study was conducted. Involving existing authors seemed key, as well as access to their data.
Yet, in another view, it was not clear ‘how much’ of the original circumstances were necessary to be ‘really doing a replication’. In the end a ‘perfect’ replication is obviously intractable, and as such the adequacy of material actions leaving to being seen as ‘doing a replication’, aside from the practical circumstances of its production, are determined through social agreement / disagreement amongst a community of researchers.
3. Incremental versus novel
A common problem seemed to be the argument that paper reviewers were found to be frequently shooting down papers perceived to be replicating prior work with statements similar to “it’s just a replication” or “too incremental” or “not novel enough”. However, within the workshop while attendees did report similar experiences, there were also a number of successfully accepted papers (e.g., actually were appearing in the main conference track).
There are two things to say about this. Firstly, I would argue that the sorts of criticisms described are not just used against the more experimentally-oriented HCI work, but instead is a pervasive phenomenon across (CHI) submission types. So any cultural change to move away from the matters raised by authors attempting to get what are seen as ‘replications’ of prior studies published must also acknowledge that this is part of a wider problem. Secondly, and perhaps more cruelly, it is possible that the reason why authors are having trouble turning papers which exhibit some measure of replication into publications, is simply that, based on the fact that some indeed are being published, the papers are just not very good. However, rather than being a matter of the content not being good enough, I suspect that most of the time it is actually the articulation of the contribution which is causing the problems for acceptance.
Leaving this second point alone for the moment, the first point underlines the importance of good reviewing, but also good articulation of the contribution by authors. By ‘good reviewing’ in this context I mean reviewers that seriously consider the contribution of the work, irrespective of whether it is seen as ‘just a replication’. Considering the contribution is about appreciating that there may be value in ‘going over old ground’. On balance, however, I felt that while it was clear reviewers needed a cultural change, authors need to do more to articulate the contribution of their work too, instead of hiding behind a picture of ‘normal science’ in which ‘doing replication’ is seen as an adequate contribution because ‘that’s what science practice is’.
4. Normal HCI practice
The RepliCHI programme has the potential to actually work against its purpose. This is because in offering a venue for work considered to be ‘replication’, it may well silo such work and through this de-normalise it as part of HCI research practice. Ironic indeed.
A final note, which I have to attribute to a conversation with Bob Anderson around this matter of replication. The sorts of ambitions that the RepliCHI programme has (which include political ambitions) can potentially also have epistemological consequences. For instance a political manoeuvre by psychology (say) to adopt the methods and standards of the natural sciences as part of a ‘package’ (i.e., to ‘look like physics’) in order to establish itself politically as a science has had epistemological consequences in terms of debates about replication, the kinds of experimentation which is valid, how results are reported, and so on.
In HCI we should be aware of these consequences and argue that political ambitions as a field can be changed rather than being subject to them.
- Paul Resnick, one of the workshop co-organisers, has written up his own reflections on the event.
- I wrote a related blog post a while back called “HCI peer review in crisis: what is science in HCI?”, which discusses the various positions I saw emerging from a set of discussions around review process particularly within the CHI conference.
- There is a discussion related to this blog posting to be found (publicly) on Facebook.
ECSCW publication: Understanding Mobile Notification Management in Collocated Groups
We have a paper soon to be published at ECSCW 2013. This is an empirical study of the ways in which collocated groups of people, engaged in a simple photo-based task using mobile phones, coordinated their activities around the receiving of notifications. These notifications were informing them both about the activities of other (distant) groups engaged in the same photo-based task, as well as any located media they encountered (e.g., a pre-existing photograph which were ‘close by’).
Abstract and citation below. A PDF is available.
We present an observational study of how notifications are handled by collocated groups, in the context of a collaborative mobile photo-taking exercise. Interaction analysis of video recordings is used to uncover the methodical ways in which participants manage notifications, establishing and sustaining co-oriented interaction to coordinate action, such as sharing notification contents and deciding on courses of action. Findings highlight how embodied and technological resources are collectively drawn upon in situationally nuanced ways to achieve the management of notifications delivered to cohorts. The insights can be used to develop an understanding of how interruptions are dealt with in other settings, and to reflect on how to support notification management within collocated groups by design.
Joel Fischer, Stuart Reeves, Stuart Moran, Chris Greenhalgh, Steve Benford, and Stefan Rennick Egglestone. Understanding mobile notification management in collocated groups. In European Conference on Computer-Supported Cooperative Work. Springer, September 2013.
EPSRC Fellowship award
I have recently been awarded an EPSRC Early Career Fellowship. The purpose of this five year fellowship (starting later in 2013) is to:
- Understand the nature of ‘theory’ and ‘practice’ in HCI research and HCI-related industry work, and unpack the connections between the two;
- Conduct empirical studies of the actual work practices found across user experience design, interaction design, and HCI;
- Create, iterate and evaluate any supporting tools and techniques which can scaffold two-way relations between theory and practice, research and industry.
RepliCHI workshop: Is replication important for HCI?
Christian Greiffenhagen and I have a contribution to the RepliCHI workshop on ‘replication in HCI’ which is running at CHI 2013. I have made the (short) position paper available below (abstract and reference). You can download the PDF here.
Replication is emerging as a key concern within subsections of the HCI community. In this paper, we explore the relevance of science and technology studies (STS), which has addressed replication in various ways. Informed by this literature, we examine HCI’s current relationship to replication and provide a set of recommendations and points of clarification that a replication agenda in HCI should concern itself with.
Christian Greiffenhagen and Stuart Reeves. Is replication important for HCI? In Workshop on replication in HCI (RepliCHI), SIGCHI Conference on Human Factors in Computing Systems (CHI), April 2013.
CHI 2013 publication ‘See Me, Feel Me, Touch Me, Hear Me’: Trajectories and Interpretation in a Sculpture Garden
We apply the HCI concept of trajectories to the design of a sculpture trail. We crafted a trajectory through each sculpture, combining textual and audio instructions to drive directed viewing, movement and touching while listening to accompanying music. We designed key transitions along the way to oscillate between moments of social interaction and isolated personal engagement, and to deliver official interpretation only after visitors had been given the opportunity to make their own. We describe how visitors generally followed our trajectory, engaging with sculptures and making interpretations that sometimes challenged the received interpretation. We relate our findings to discussions of sense-making and design for multiple interpretations, concluding that curators and designers may benefit from considering “trajectories of interpretation”.
Lesley Fosh, Steve Benford, Stuart Reeves, Boriana Koleva, and Patrick Brundell. ‘See Me, Feel Me, Touch Me, Hear Me’: Trajectories and interpretation in a sculpture garden. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, April 2013
ACM ‘interactions’ magazine article: Building the future with envisioning
I have an article titled “Building the future with envisioning” published in the Jan/Feb edition of ACM interactions magazine. This is “the flagship magazine for the ACM’s Special Interest Group on Computer-Human Interaction (SIGCHI), with a global circulation that includes all SIGCHI members”, according to their website. This also means that the article is subscription-only and only a portion is accessible online to those without institutional access or some other subscription arrangement. Thus, if you wish to read it, please email me and I can furnish you with a copy.
Personal and Ubiquitous Computing Special Issue
I’m co-guest editing a special issue of Personal and Ubiquitous Computing in the coming year (2013).
The title is Understanding Performative Interactions in Public Settings, and the special issue has been organised by Julie R. Williamson, Lone Koefoed Hansen, Giulio Jacucci, myself and Ann Light.
Some details below:
Interactive technologies are employed in ever more diverse ways: notably, for this special issue, in a wide variety of settings where interactions are transformed into implicit or deliberate ‘performative’ spectacles. Performative spectacles range from everyday actions such as technology use while in urban spaces like city streets to ‘staged’ or deliberately designed cultural and artistic experiences and they can be experienced both by physically co-located or remote spectators. Viewed as a spectacle, human-computer interaction and particularly the individual users become subjects of public interpretation with everyone watching and making sense of each other’s interactions and intentions. These public interactions can (be purposely designed to) range from ‘invisible’ and hidden to highly visible and expressive, often challenging social norms or even evoking discomfort—physically, emotionally or socially.
While social environments have always offered space for interaction and spectacle, ‘performative’ settings are now being defined, supported and enhanced technologically by ubiquitous and mobile computing, gesture-based computing and sophisticated sensing and sensor networks. The study of human-computer interaction has already seen a growing use of perspectives from performance and theatre studies, which are increasingly brought to bear on the challenges of designing interactive systems that leverage the ‘performative’ aspects of these complex environments. In this view, as designers and researchers in HCI and ubiquitous technologies, we design performative artefacts and experiences whenever we open up new interactive possibilities for settings where others may see, feel or hear our actions.
This special issue aims to highlight the performative aspects of designing for interactions – mobile and immobile – in public spaces. It acknowledges the increasing importance of understanding how we exploit characteristics of spectacle and manage the boundaries and transitions evoked in public ‘performance’. In doing so, it offers the chance to explore how human-computer interaction can be described, designed and evaluated by understanding everyday interactions as potentially ‘performative’, as well as exploring those public settings in which technology is more explicitly used for performance purposes.
For more information on topics and important dates, see the special issue website.
Misconstruing reflexivity (again)
It is often interesting to examine the ways in which prior work has been taken, interpreted and applied in others’ subsequent research and compare it with the original intentions. While I’m sure some might argue, Barthes-like, that once our research writings are out there, then the role for us as authors has ‘died’ at their moment of publication, I tend to disagree with this view.
At CHI 2011 I co-authored a paper (“Into the wild: Challenges and opportunities for field trial methods”) that detailed a study of a field trial of technology (in our case a simple photo-sharing application that was part of the Designing the Augmented Stadium project). In the paper we explore how this popular form of HCI research—i.e., design it, build it, stick it out into the world, study it, report findings—works out in practice.
In part riffing on this paper is a recent (CHI 2012) piece from authors Johnson, Rogers, van der Linden and Bianchi-Berthouze, “Being in the Thick of In-the-wild Studies: The Challenges and Insights of Researcher Participation”. Like us, Johnson et al. have performed a field trial (a very nicely designed interactive haptic / visualisation device for assisting in violin bowing), and in addition to reporting on the field trial findings, they have provided extensive discussion on the nature of the field trial process.
What are field trials, though? We described what field trials were like this:
“Field trials have become a common method for studying the use of novel technologies, widely used in fields (such as HCI, Ubicomp and CSCW) where the interests of investigators goes beyond technical feasibility to exploring user understandings, practices and the eventual uses that systems might be put. What we refer to as trials in this paper go under a range of different names—field experiments, deployments, evaluations, field studies, technical probes—but they share a set of common features. A new system, usually developed by the researchers, is given to a set of users who are asked (often implicitly) to use the system ‘naturally’ outside the laboratory the system was designed in. They then use the system for anything from a few hours to a year, frequently as part of their day to day life, or with the system deployed in their home or workplace. In more experimental trials users’ behaviour may be constrained with tasks set for them to carry out, while other trials eschew such controls and attempt to encourage ‘natural’ use. In some cases trials are run as an extension of experiments, whereas for others a more ethnographic analytic mode is adopted.”
Johnson et al. talk about the role of the researcher in the field trial, unpacking what work was required by the investigator (in this case Johnson herself) during the trial—for instance, as facilitator, as champion, as tech support (see Tolmie & Crabtree’s paper “Deploying Research Technology in the Home”), explainer, etc. This is all quite interesting and potentially useful. However the critical part of such a paper comes with the recommendations it makes in light of this. While most are useful, there is a a point that I feel Johnson et al.’s paper starts to diverge strongly with the ways in which we conceived of a route forwards for field trials. In particular Johnson et al. argue for a ‘reflexive mode’ of approach to doing field trials that extends beyond its existing argued-for application within ethnography (see Rode’s paper “Reflexivity in Digital Anthropology” and my previous blog):
“Although this call for more reflexivity has mainly been addressed towards ethnographers studying existing contexts, it is just as relevant to those doing studies of deployments of new technologies in-the-wild. Perhaps, more so, because deploying a new technology requires hands-on intervention into the daily routines of a setting.”
In other words, Johnson et al. are promoting the notion that ‘doing being reflexive’ when investigating a technology prototype deployed in an ‘in-the-wild’ setting has benefits. Specifically, they argue that it “challeng[es] the investigator to question their own role in the research” and that “[c]urrent methods employed in in-the-wild studies do not normally show this level of reflexivity or consideration for the social and personal context beyond the boundary of the interaction being studied”. But—so what? Why should investigators be more ‘reflexive’ (if they can be)?
To flip back for a moment to our “Into the wild” paper, at its core was an explication of just what the work of the field trial is, and what it does (and can do) for delivering research findings. In particular we explored a bunch of issues relevant in that work: 1) demand characteristics, a concept from psychology that describes, broadly, issues of interactions between experimenters / investigators and participants in field trials impacting the running of the trial (although we were not studying an experiment in the traditional psychological mode per se); 2) the role of lead participants, that is, the ways in which a subset of participants may consistently produce the most interesting results for research reportage; and 3) the interdependence of methods and results, i.e., reconfiguring how field trials are seen with factors like demand characteristics and lead participants as crucial for the research work. Thus, we argue that, instead of seeing these things as ‘problems’ for trials, they are instead some key normative orientations to reconfigure (e.g., that systems trials aren’t just ‘about’ success or failure, that participants can also be seen as co-investigators), and some benefits that should be accounted for (e.g., in terms of research communication, better ‘methods’ sections for papers) which may well also be usefully exploited (e.g., purposefully locating good ‘lead participants’). None of these is about entering a reflexive mode in order to do better field trials—instead it is about simple strategies for delivering greater practical value when doing field trials.
Whereas we were interested in understanding the work of the field trial as a methodical, practical engagement with participants, Johnson et al. seem interested in pursing the idea of ‘being reflexive’ itself as a method for ‘doing better trials’. In part I think the problem stems from a profound confusion between reflexivity and reflection, which are terms used almost interchangeably in Johnson et al.’s paper. While I would expect researchers to be reflective (surely this is their primary activity; which researcher would want to be accused of being ‘unreflective’?), reflexivity on the other hand, as a term, indexes a bunch of quite unrelated potential possibilities. This is where things get sticky. The most likely candidate form of reflexivity is something along the lines of ‘standpoint reflexivity’ that is found in many postmodern accounts of cultural theory. In this way of thinking is referenced by Johnson et al. via Dourish & Bell’s book “Divining a Digital Future” in which they argue for the importance of accounting for “the author’s stance”. In this line of thought, ‘deconstructing’ the author’s stance is seen as part of a commitment to ‘radical’ theory (which normally preoccupies itself with a set of standard sociological interests).
Interestingly, not once in our paper do we mention ‘reflexivity’, most probably because we do not see reflexivity as a specialised activity that is engaged in by researchers (instead, we’d probably point to the reflexivity of accounts; as Mike Lynch puts it “[t]o imagine an unreflexive action would be like imagining a sound without amplitude”). While there is nothing wrong with this ‘reflexive’ approach per se, there is typically little to recommend it in terms of its potential contribution to design in HCI (although I remain open to being convinced).
There is a wider game being played here too, specifically the ingress of ever more theory borrowed from other disciplines and brought into HCI. Lynch argues that “[w]hen we recognize that there is no single coherent division between reflexive and unreflexive discourse, then reflexivity loses its metaphysical aura and (apparent) ideological potency for empowering theories and rallying movements”. Although Johnson et al. have meshed our particular paper within this nexus of various rallying ‘calls to reflexivity’ that have been heard recently in HCI, our approach is anything but.
Placebooks community media toolkit launched at Royal Geographic Society
Placebooks is a community media toolkit, and is based around a web service and a mobile application design to support a range of activities in and around rural communities. Placebooks developed as part of the 18 month Bridging the Rural Divide project, an RCUK-funded project base primarily at the Mixed Reality Lab, University of Nottingham and the FITLab, University of Swansea. The project explored notions of ‘rural computing’ and ways to support rural activities, rural economies and rural life.
The Placebooks platform is currently undergoing handover between ourselves as researchers and the People’s Collection Wales, a Welsh Government initiative that has built a bilingual web service to collect, interpret and display the story of Wales. Placebooks was officially launched in July 2012 at the Royal Geographic Society, presented by members of the project team to an audience that ranged from the BBC and local community groups to academics and research councils. The presentation (pictured below, photo by Giles Lane) was followed by a panel.
Placebooks itself enables users to piece together all sorts of media types from disparate sources into an interactive digital booklet. These media types can be maps, images, text, video, audio, etc. drawn from local storage (e.g., the user uploading a photo from their hard disk), or hoovered up from external web services like Everytrail, DropBox, or the People’s Collection Wales. These digital booklets can be published on the website, shared via a range of means (e.g., social media), and downloaded to a mobile application (Android and iPad versions) as a complete offline ‘package’ for use in-the-field.
An important aspect of a Placebook is that it is described by geometries, meaning that as a whole the Placebook itself may be tagged to a particular location or region. Equally, its constituent parts may also be paired with locations, areas or regions that they relate to. In this way Placebooks provides a rich (and easy) way of creating interactive booklets that are infused with a range of geographic information. The system is generic enough to serve a range of purposes and communities.
The project was PI-ed and CI-ed by Matt Jones, Andy Crabtree and Alan Chamberlain, and was run in coordination with various partners such as Ordnance Survey, the Countryside Council for Wales and towards its end, the National Museum Wales and the People’s Collection Wales.
Placebooks itself was built largely by Kevin Glover (who developed the web interface), myself (server backend stuff) and Mark Davies (mobile application) with additional work on integrating external data sources, such as Everytrail, by Mark Paxton.
The development of the software was driven in a number of ways. In the first instance we used some ethnographic work from Peter Tolmie as a way of shaping ideation. This was coupled with having Alan as researcher situated / embedded in the environment and communities in which we were siting the design. This ‘action research’ approach supported the generation of requirements (specifically, we focussed on the Borth and Ynyslas area). Through a continual process of local participant engagement via Alan we were able to direct and shape the software development process, and perform elements of participatory design. This process was also strongly collaborative within the team in terms of co-designing the whole system to quite a fine level of detail—so although only a few of the team were actually building the software, its design was fundamentally ‘worked through’ as a collaborative process. This approach, coupled with significant user engagement work, meant that a wide range of prospective users were represented in the system design and at the same time had a proxy (Alan) with whom it was possible to critically influence how work progressed.
As a rule, our world is largely dominated by digital computation. We have probably mostly forgotten what analogue computation looked like and its fundamentally different character to digital computation. Recently I read a CHI paper (with a Best Paper Award) by authors Fernaeus, Jonsson and Tholander, called “Revisiting the Jacquard Loom: Threads of History and Current Patterns in HCI”, published at CHI 2012 which seemed to make just this error of exoticising analogue computations, and then confusing them with digital computations in a problematic manner.
The central conceit of the Fernaeus et al. paper is pretty interesting: an assessment of past manufacturing machinery through the lense of modern HCI in order to ‘report back’ insights for its modern development. There is much to recommend in this paper in terms of its novel approach. It is possible that the Jacquard Loom may well have things to teach us, perhaps about the role of expert operation of machinery. However, while it is a novel way to view modern computing in a different light, if we wish to perform this kind of trick, it is strictly important that extra care is taken to consider the effect of that conceit on the topics that are being drawn into the historical comparison, which here are issues of the digital and the physical, tangible user interfaces, and the actual nature of computation.
The paper abuses (perhaps too strong a word, but the meaning is right) the nature of computation, and its specific qualities. It does this by playing quite loose and fast with the history of computing, and through this, the very nature of the relationship between the loom and digital computing.
We are quickly introduced to the notion that the loom and digital computation have some shared attributes through the common input format of punched cards. The looms were fed punched cards which dictated which threads would get pulled where, and thus resulting in a particular pattern described by the punched cards. Early computers also used punched cards as input devices—however I would argue that this is the point where any similarity or possibility of useful analogy stops since the loom used the cards as analogue instruction, whereas early digital computers used the cards as digital information. While the authors admit that “no actual computation [was] performed using the cards”, Babbage’s difference engine and analytical engine are brought in as relevant. The fact that the analytical engine was Turing complete and digital programmable (albeit mechanical) is not drawn out as a key issue. To have done so would be recognising the substantive categorical difference between analogue and digital computers. Instead we get an implied link between the loom’s design and “designers of new computational artefacts”. This is getting into non sequitur territory. Digital and analogue computation are fundamentally different.
Developing this blurring, we get further examples, including the abacus, side rule and player pianos, each of which is extolled as being significant for the argument because “they are based on ingenious compositions of material and mathematical coding schemes, alongside simple methods of manipulation”. Firstly there is the loose play here of the term ‘coding’ (do they really mean to compare this with programming? The reader is left uncertain). Secondly (and reiterating an earlier point), artefacts such as these are fundamentally incomparable in computational terms with a general digital computer since they are, like all analogue computers, intended to perform one form of ‘computation’ only, and cannot be reconfigured in any general purpose manner.
Evidence of this confusion is also found in a characterisation of how we might view the role / relevancy of punched cards: “The mechanics of the punched cards could be regarded as the birth of the binary representation, making it possible to ‘digitize’ material objects, creating a form of ‘code’ only possible to interpret by running it through a mechanical device.” Further slippage is found with the statement that the loom “clearly executes a form of code” and that we must search for a way of looking at the loom in which the “whole process [may be seen as] programming”. At the same time the authors admit the non-digital nature of the loom, but don’t pause to consider its significance. Could it be that the analogy can only provide so much purchase?
In many ways this confusing approach—the punched cards are seen as ‘digitising’ but at the same time the loom is not ‘digital’—is strangely reminiscent for me of an increasingly popular approach in HCI called ‘defamiliarisation’. The theory here is that so-called ‘defamiliarisation’ provides a technique by which we can attempt to gain novel insight through ‘making strange’. Here the loom offers insight into computation by making it strange through problematising its very foundations. Our sense of the digital, the general purpose computer, the Turing complete artefact, etc. is thrown into disarray. In this logic, ‘making strange’ enables new views of that which is familiar, here mundane digital computation, and therefore grist for the design mill. And the mill in question here is tangible computing.
Continuing to stretch the very concept of ‘code’, the paper goes on to a key discussion point on approaches to tangible interactions: “[t]he [loom’s] code is made visible through a completely transparent physical construction [on punched card], while in modern day computers, the machine code is hidden from user in layers of abstraction”. Instead of being seen as intrinsic to the very nature of digital computation, in the confusions of analogue mechanical devices versus digital computation-based, abstraction somehow, somewhere, has been transformed into a ‘problem’. Even though it is arguably fundamental if we are to use digital computers as a medium for anything.
The paper reaches a closing point that “physicality does not necessarily imply understandability. In a sense, the preoccupation with making machinery understandable is somewhat peculiar to HCI, and can be contrasted to other values such as efficiency and productivity”. But understandability is not in any sense an oppositional value to efficiency and productivity, as it appears to be portrayed here. And, furthermore, HCI has since time immemorial been hugely concerned with efficiency and productivity (ergonomics background, canonical usability studies, huge swathes of CSCW, etc.), and less about ‘understandability’.
In the end, the authors seem beguiled by the complexity of the loom and the skills required to operate it and that it can demonstrate for tangible computing an interface that is complex yet “graspable”. This is ironic, of course, since the Jacquard Loom, through automation, made loom work simpler.
HCI peer review in crisis: what is science in HCI?
HCI is in continual crisis mode with regard to peer reviewing. Every year around the time of the CHI conference - or to give it its full, unwieldy and faintly outmoded title The ACM SIGCHI Conference on Human Factors in Computing Systems - there is considerable debate about the reviewing process (see the aside at the bottom of this posting for more detail on the reviewing process itself). No doubt similar conversations take place at all other major HCI and related venues (e.g., CSCW), but it seems that the ones from CHI leak onto the web most readily. Other conferences have comparable review processes (CSCW, DIS, Ubicomp), so the conversation here is relevant beyond CHI.
It seems to me that a lot of these conversations revolve around a number of topics related to the review process itself, such as the relative weighting of reviewer and author power, the accountability of activities of programme (sub)committees, the way reviewers get selected, the rising numbers of submissions to the conferences, and (relatedly) the acceptance rates and submission types. As an author it can be incredibly frustrating if the account of why your paper was rejected is scant on detail (such as reports that your paper was discussed at the committee meeting, but the actual detail of the discussion was left out), and correspondingly as a reviewer it can be annoying to have to review work which does not seem to even hit the lower level of what could be considered worthwhile.
In response to participating in the process of peer review, Jeffery Bardzell recently wrote an interesting blog posting outlining the various ways in which not only practical changes could be made to this reviewing process, but also presented some ways the HCI community could reconceptualise what process it is actually engaging in during peer review. Bardzell presents this in his “Position on Peer Reviewing in HCI” (in three parts), and formed a lengthy response to Antti Oulasvirta’s original blog posting on “Why your paper was rejected” with specific reference to CHI.
Oulasvirta’s blog here is key in understanding the meaning of the exchange and in assessing some of the underlying concerns that are perhaps helping animate the discussion. Oulasvirta, as a participant in a subcommittee of CHI 2012, summarises some of the key problems that result in low scores being awarded to some kinds of paper submitted to CHI. He focussed particularly on what he terms “flaws in empirical work”, including such things as: errors in research strategy (i.e., applying the wrong research tool for the job); basic statistical flaws (interestingly Grounded Theory is included here related to errors in inter-coder reliability metrics); causation issues (e.g., problems with the `implies’ relationship); generalisability and replicability of findings.
Reading the blog post reveals Oulasvirta’s epistemic position on the nature of HCI and its relationship to the natural sciences. While it is not explicitly nailed down anywhere, this position seems to be that HCI (or at least parts of HCI) is a scientific discipline, and as such many submissions to conferences may be shot down on a range of scientific grounds, as outlined above. The notion of HCI as a science is grounded in and guided by popular conceptualisations of ‘what science is / what scientists do’. Specifically these are articulations of what is thought to be the essence of scientific practice, such as: the nature of larger problems is such that they may be broken down into subproblems and reassembled upstream (i.e., scientific reductionism); the mutually reinforcing relationship between theory and experiment, with each elaborating the other - experimentation provides ‘canaries in the mine’ for theory (e.g., see the controversy over string/M theory), and theory directs the ‘what next’ of experimentation; the role of replication and experimentation as separate or separable from context and situatedness. These essences from the natural sciences provide yardsticks with which to measure HCI activity against if it is to be (or remain) a normal scientific activity.
This view of HCI is not new, and often emerges in a range of different guises from HCI research. For instance, Bartneck reports how President of the Association for Computing Machinery (ACM) Stuart Feldman’s CHI 2007 opening speech included the assertion that HCI “is absolutely adherent to the classic scientific method”, which, he said, was “[n]ot a description […] of all the fields in computing”. Ben Shneiderman’s comment on Bardzell’s blog raises this issue again with the comment that “HCI is a scientific discipline”. More recently, the CHI 2012 conference is hosting an ongoing workshop series - RepliCHI - that is concerned explicitly with a cornerstone of the natural sciences, namely, replication. The workshop, expressing very common concerns within the HCI community, states that “it is not uncommon for research to be rejected in CHI for being ‘incremental’”, whereas “[i]n science, it is common for people to try and replicate discoveries, so that the community can confirm new discoveries”. There are other variations on HCI’s scientific status, such as James Landay’s blog posting on “giving up CHI/UIST” (comparable maybe to ‘giving up Christmas’ or ‘giving up on a relationship’?). This is a call for a wider notion of science and where it happens: “I think we have been blinded by the perception that ‘true scientific’ research is only found in controlled experiments and nice statistics”. A now-defunct blog posting by Joshua Kaufman (archive.org link) on a talk by Paul Cairns on ‘HCI as Science’ brings Popper’s conceptualisation of what science is to bear on whether HCI is science (e.g., via Popper’s notion of science as falsifiable). Interestingly one of the commenters argues that HCI is more like engineering: “The Wright brothers constructed a working airplane without knowledge of aerodynamics, fluid mechanics, etc. They just applied tried and true engineering best practices which they learned from working on bicycles (plus a ton of trial and error). In fact, several generations of planes were created before scientists started to grasp any of these principles, and the debate continues today. Yet we fly around in planes. They work, even if we don’t have all the ‘laws’ exactly nailed down.” The unstated assumption being that engineering something that interacts with the physical environment is somehow isomorphic with engineering of things that interact with and around humans (I would argue that they are not).
Bardzell’s blog appears to be a reaction in some ways to this implicit / explicit epistemological position on the status of HCI as a science. The perceived problems surrounding the peer review process mentioned at the start of this discussion are transposed into or driven by such epistemic issues. While it appears that Bardzell does not deny that there are scientific aspects to the reviewing process, rather than arguing for the scope of what is considered to be normal science to be enlarged (as Landay does), he suggests that we must attend to the social features of reviewing as they play an important (and overlooked) role in the development of scientific results. In sum he appears to be arguing that reviewing is by its nature “critique” more than (or perhaps instead of) being part of a rigorous scientific process that upholds the popular notions of ‘what science is / what scientists do’. (In some ways perhaps it is not much different from film criticism?) This argument obviously borrows strongly from constructionist accounts in Science and Technology Studies (STS) and Sociology of Scientific Knowledge (SSK) (e.g., Latour and Woolgar’s work), where the outputs of the work of the natural sciences is understood in terms of local practical circumstances (e.g., physical lab) typically filtered through the lense of external features such as politics, power structures, funding systems, etc. In my view Bardzell is linking reviewing work with a consideration of these sorts of aspects.
There are a great many interesting aspects of this discussion. However, it turns out that a lot of these concerns are not new, and not particularly unique to HCI. For instance, we can compare issues around peer review in HCI with some of the gripes emergent from SIGCHI’s parent community, i.e., SIGGRAPH. Ashikhmin expresses similar concerns about the review process, fashionable topics, results being presented in a normatively scientific manner in order acquire the cache of validity, reviewer power, acceptance rates, and assertions about computer graphics being a ‘scientific endeavour’. As Ashikhmin states “computer graphics is simply not a hard science and current attempts to present it as such are misguided at best […] the success or failure of any specific technique is, quite literally, in the eye of beholder”.
Another key aspect of the perceived problem is probably the way in which HCI mixes communities (which is a good, but challenging thing). A great deal of this discussion probably culture clash, perhaps between those coming at HCI from a cognitive science and / or psychology background and those from arts and humanities backgrounds (who are increasingly being involved in HCI). While those from such a non-‘science’ background may seek to conceptualise HCI in the mould of the way that the natural sciences are conceptualised in STS, so correspondingly those from the ‘labelled sciences’ (computer science, cognitive science, behavioural science, etc.) are perhaps guilty of mythologising of what the actual work practice involved in the natural sciences is like, or perhaps forgetting that HCI’s related fields like psychology have plenty of their own scientific demons (e.g., problems with WEIRD behavioural and brain science). While I would not necessarily want to align with STS, SSK, etc., particularly as they have a tendency to eschew studies of mundane practice and revert to something like structural functionalism (as noted by Mike Lynch in Scientific Practice and Ordinary Action), there is something to be said for the importance of the social features of scientific practice and its impact on the presentation of science. However, there is probably an equal tendency by those from a non-‘science’ background to presume that such social aspects are somehow hidden from scientific practitioners, and that they are unknowing ‘cultural dopes’ going about their work.
A nice example of the work of a discovering science is provided by Eric Livingston in his book Ethnographies of Reason. Through a simple procedure he unpacks the embodied and commonsense ways in which a phenomenal field may be investigated through various instruments employed to uncover it. In this account, natural phenomena are explored through practical action, and subject to everyday mundane practicalities. This leads to another issue: that of the relationship between the phenomena that are being investigated by HCI and the role of those self-same phenomena within HCI. As pointed out here in relation to Kaufman’s blog posting “HCI phenomena are constantly changing, HCI is constantly moving into new domains, redefining itself and absorbing new types of technology. Basically, there are no static phenomena so there can’t be an HCI paradigm. Furthermore, since there is no HCI paradigm, HCI is not a science.” This is perhaps not deep enough, though. In this situation we can consider the standard ethnomethodological critique. HCI’s phenomena of interest are actually the phenomena that enable us to study the phenomena in HCI in the first place. In other words, the commonsense methods of local, situated reasoning that are brought to bear in the work of HCI research often remain largely unexamined in spite of the fact that really this is the phenomena we are examining when we do a study. While not attending to these issues may be okay for the natural sciences, which may suspend interest in such matters in favour of transcendental principles of the natural world, for the human sciences (which HCI regularly brushes up against, being concerned with ‘interaction’), we ignore these issues at our peril.
Finally there is the legitimacy argument. Kaufman’s blog posting suggested that it was a good thing for “HCI [to] be considered a science because science was practically the only measurable form of progress in the 20th century”. Scientism faces us all, especially when it comes to gathering funding and establishing the legitimacy of our research work in HCI. In one view this could be seen as a cynical perspective on science (and perhaps devaluing the label), but on the other hand, we can instead see it as ‘strategic’ and necessary.
[An aside: for those who are not familiar with the CHI reviewing process, a simplified version is as follows. The conference is organised into various subcommittees which cater for different aspects of the field of HCI (this subdivison is of course a massive fudge, but practical). Authors target their paper to a given subcommittee; the subcommittee’s chairs then divvy up the papers between themselves and the associate chairs (ACs), who in turn recruit a number of reviewers (3+) for each of the papers they are AC-ing for. Reviewers do the reviewing, ACs write meta-reviews summing up the points made and scores provided by the reviewers, all of which are then sent to authors. Anonymity is top down and bottom up: authors are anonymised for reviewers, and the reviewers are anonymised for the authors. Reviewers are also anonymous between each other. ACs on the other hand know who the reviewers are. Authors, having received their paper’s reviews have the option of writing a rebuttal if they so wish. After the rebuttal period is over, reviewers can alter their review and score based on the rebuttal. The final set of reviews (sometimes augmented with further reviews for particularly divided reviewers) are then used to work out what papers get discussed at the subcommittee meeting. The ACs and chairs meet up, discuss all the papers they feel need discussing (a very opaque process but generally speaking papers below a certain average score are typically dismissed), and then collectively determine accept / reject decisions for all the papers for their particular subcommittee. Finalised reviews and final paper decisions, often including some account of `what went on’ at the subcommittee meeting, are then sent to authors, who are either left whooping with joy or crying in despair.]
A while back (12th April 2011) I set up and ran a workshop at the Mixed Reality Lab on `display ecologies’. The workshop was an attempt to (1) survey existing work in the field and the state of the art as found in the literature; (2) to determine what research questions - whether they are design, conceptual or infrastructural - need to be answered; and (3) to work out what scope there is for innovation. After the workshop I wrote up a report that documented what was discussed, and what each speaker presented.
An excerpt from the report is below, but you can download the whole thing here.
The main purpose of the workshop was to explore public interactive display ecologies – often called ‘situated displays’. Notions of ‘situated displays’, however, are not wide enough a concept to capture the diversity of configurations and forms of physical display (as well as social aspects of ‘display’ and ‘displaying’) that we both encounter in the literature as well as located within the content of the workshop. Instead, it seems that an ‘ecology’ metaphor affords us greater purchase in understanding the design space.
Ecology evokes a range of useful concepts, such as how ecologies are composed of hybrid elements, their heterogeneity in that they fit together or are made to fit together and yet retain boundaries and seams. Overall, ecology as a metaphor is invoked to suggest the ‘messy’ reality of real-world configurations of display. ‘Ecology’ also implies display systems that exist in an ever-changing environment. It implies that these messy interrelated components might self-organise in some way. Ecology implies things might be brought into play that we didn’t expect - new objects and so on which might interact with the displays somehow but we didn’t expect them to. Ecology implies display elements that mesh more with non-digital resources and a wider understanding of what a ‘display’ is or might be.
CHI 2012 publication: “Envisioning ubiquitous computing”
Reeves, S. Envisioning ubiquitous computing. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, May 2012.
Visions of the future are a common feature of discourse within ubiquitous computing and, more broadly, HCI. ‘Envisioning’, a characteristic future-oriented technique for design thinking, often features as significant part of our research processes in the field. This paper compares, contrasts and critiques the varied ways in which envisionings have been used within ubiquitous computing and traces their relationships to other, different envisionings, such as those of virtual reality. In unpacking envisioning, it argues primarily that envisioning should be foregrounded as a significant concern and interest within HCI. Foregrounding envisioning’s frequent mix of fiction, forecasting and extrapolation, the paper recommends changes in the way we read, interpret and use envisionings through taking into account issues such as context and intended audience.
CHI 2012 publication: “A hybrid mass participation approach to mobile software trials”
Morrison, A., McMillan, D., Reeves, S., Sherwood, S., and Chalmers, M. A hybrid mass participation approach to mobile software trials. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, May 2012.
User trials of mobile applications have followed a steady march out of the lab, and progressively further ‘into the wild’, recently involving ‘app store’-style releases of software to the general public. We examine the literature on these mass participation systems and identify a number of reported difficulties, which we aim to address with a hybrid methodology combining a global software release with a concurrent local trial. A phone–based game, World Cup Predictor, was created to explore the uptake and use of ad hoc peer-to-peer networking, and evaluated using our hybrid trial method, combining a small-scale local trial (11 users) with a ‘mass participation’ trial (over 10,000 users). Our hybrid method allows for locally observed findings to be verified, for patterns in globally collected data to be explained and addresses ethical issues raised by the mass participation approach. We note trends in the local trial that did not appear in the larger scale deployment, and which would therefore have led to misleading results were the application trialled using ‘traditional’ methods alone. Based on this study and previous experience, we provide a set of guidelines to researchers working in this area.