Abstract

This research proposal describes a Ph.D. research project in interaction design with information intense heavy vehicles. The research aims to explore the use of mixed reality interaction and visual presentation using see-through interfaces and symbolic metaphors, to enhance the interaction for operators working with these types of Machines.

1. Introduction and motivation for research

As systems, for example, vehicle systems, get increasingly autonomous and information intense, the information exchanged with the user, i.e. the operator, are increasingly becoming a designed interaction. This in relation to the interaction earlier being a result of the machine behavior and mechanical operation. Thus, there is a need for a proficient interaction design to establish an efficient, environmentally conscious and successful operation.

Also, as machines get connected, more information will be communicated from and between machines. And as higher levels of autonomy will be introduced, the activities and purpose of the operator will likely also transform into more managerial than operational, affecting the interaction with the machine and the need for information.

Furthermore, there is even a risk that new functionality is added that increases mental load and draws operators attention, thus potentially increasing the risks of failure as well as compromises human safety [1]. One example of such is the Llanbadarn Automatic Barrier incident report where a train passed a crossing with the bars raised. One reason why this happened was because the operator was occupied with the driver machine interface and therefore missed the crossing indicator [2].

Using mixed reality interaction, thus blending information with the surrounding area, have the potential to enhance situational awareness.

The aim of this research is to explore interaction with information intense heavy vehicles and find means for user experiences and efficient interaction between the operators and the industrial mobile machines; for example, agricultural machines and construction machines. The research combines software engineering and interaction design together with an industrial perspective (the researcher is an industrial Ph.D. student).

Increasing efficiency of machines and utilization of information can also positively impact a sustainable society. For example through less fuel consumption or a more precise and limited soil fertilizing. Also, reports claim that excessive movements are made, for example, to look out of the machine in certain angles with covered sight [3], resulting in injuries. A good design of the interaction, including placement of the interaction devices can increase operator wellbeing and sustained operation. It can also improve information detection and intake, for example, when the information is within the visual attention area and less time is required to refocus on a display placed at the side in the cabin and then back into the surroundings [4].

This type of research can impact the coming future of even more information intense and autonomous machines, as major OEMs are, possibly, more than ever seeking to information to provide higher levels of productivity and efficiency [5], [6].

2. A summary of background and related work

Different types of transparent interaction systems and augmented interfaces, that keep the user visual attention close operation, are currently an area of state-of-the-art interest [7]. The field itself is not new, head-up displays have, for example, been used for a long time in specific areas, aerospace being one early adopter. But new commercial products have made the technology increasingly available and widespread.

In automotive applications, in particular cars, the use of head-up displays have been extensively researched and used. The information needed in industrial applications is however different than to cars, as it is not only about transportation but also about the production process performed by the vehicle.

Also in the heavy machinery industry, head-up display technology have been evaluated, indicating possible benefits in ergonomics, information intake and productivity [8], [9]. These test has however mainly focused on taking production information currently visualized on displays, and replicating this in the field of vision of the operator. Focusing less on renewed ways to present production information, as well as mediation of additional information.

Another alternative would be head-worn displays, that can be used to present information within the field of view of a user, regardless of the head rotation [10]. But these technologies require extra steps before being able to operate or exit the vehicle, as well as the risk of dropping of losing the visualization devices. Smart devices such as mobile phones and tablets can also be used to display information about machine status, production, or settings. Within or at a distance from the machine. Mobile devices can also be used in augmented solutions, for example, in mobile see-through interfaces where information is overlaid on the display of the device [11], [12].

Additionally, the way to present information in information intense and increasingly autonomous vehicles are an area of research. In farming, Sørensen et.al mention that acquisition and analysis of information still proves a demanding task” [13], and the availability of data does not warrant the understanding or usefulness of the data to the user [14]. This is also indicated by the Designing Interactive Systems conference (DIS) workshop on metaphors for interaction with autonomous systems [15].

3. A description of the proposed approach/solution

The proposed solution is to use mixed reality to present information in the line of sight of the operator, when he/she is looking out through the cabin windows. This would let the operator’s attention to be closer to the outside world, compared to when the operator has to look away at screens for information intake. It would also provide a larger screen area for information presentation.

The approach will be to practically explore interaction design and technology by producing visualization concepts and prototypes.

Furthermore, the research seeks to find a common language of interaction that can be used by diverse vendors or practitioner in several industrial machinery contexts. Such as a uniform symbol language that can be used in many types of applications and handle different levels of criticality.

Audio and tactile feedback are also to be considered, but the visual channel will be in focus.

5. A brief discussion on the applied research methodology, including how the solution is going to be evaluated

According to Fällman [16], Zimmerman et. al. [17], Höök et al. [18], Schön [19] and others, interaction design research can be performed through design practice. Sennet [20] even argues that understanding is impaired when we separate practice from theory. The result of the practice can be expressed in many forms, for example, sketches [21], through artefacts and systems [22]. A strong concept is the exploration of possibilities as well as the reflection on design, its artefacts and possible future outcomes [19], [23].

The practice will result in designs and prototypes. As access to real working vehicles is a limited source, evaluation of the concepts and prototypes will mainly be made through simulators and qualitative interviews. For mature concepts, the target is to do quantitative user evaluation with a bigger sample set.

5. Expected contributions and results so far

The work until this stage has centered on gaining understanding and create a foundation for the coming work. It has focused on selected stakeholders (operators [24], designers and developers [25]) and methods for understanding, process, and practice. Among others it has:

• Provided means to evaluate operator’s daily interaction with machines, with minimal interruption of work, using eye tracking. This work included a qualitative study of operator’s attention at different vehicle types [4].
• Studied of the role of user understanding in the creation of autonomous vehicles [24].
• Ideated on different ways for a stakeholder in the lifetime of an industrial vehicle [26].
• Realized a mixed reality simulator made with audio and visuals [27]. To be extended with see-through (HUD) style visualization.

So far the effort has resulted in a licentiate degree [28] and a number of publications. The plan ahead is to use this experience in the second phase of the research, from where it is expected to result in a number of possible interaction design concepts. These, in turn, will contribute to the understanding of information exchange between software intensive heavy vehicles and its operators, its scenarios and technologies, its use, its challenges, and possibilities.

6. References

[1]         F. Lauber, A. Follmann, and A. Butz, “What You See Is What You Touch : Visualizing Touch Screen Interaction in the Head-Up Display,” pp. 171–180, 2014.

[2]         Department for Transport, “Incident at Llanbadarn Automatic Barrier Crossing (Locally Monitored), near Aberystwyth, 19 June 2011,” 2012.

[3]         T. Eger, A. Godwin, D. J. Henry, S. G. Grenier, J. Callaghan, and A. Demerchant, “Why vehicle design matters: Exploring the link between line-of-sight, driving posture and risk factors for injury.,” Work, vol. 35, no. 1, pp. 27–37, 2010.

[4]         M. Wallmyr, “Seeing through the eyes of off-highway vehicle operators,” in (accepted at) The 16th IFIP TC.13 International Conference on Human-Computer Interaction, INTERACT ’17, 2017, p. 21.

[5]         “Caterpillar Introduces ‘the Age of Smart Iron’ – Digital Technology designed to Transform Productivity, Efficiency and Safety on Job Sites.” [Online]. Available: http://www.cat.com/en_US/news/machine-press-releases/caterpillar-introduces-the-age-of-smart-iron-digital-technology-to-transform-productivity-efficiency-and-safety-on-job-sites.html. [Accessed: 31-Oct-2016].

[6]         “Trimble corporate news relases 2016,” 2016. [Online]. Available: http://www.trimble.com/corporate/news_release.aspx. [Accessed: 20-Jul-2016].

[7]         J. A. Betancur, J. Villa-Espinal, G. Osorio-Gómez, S. Cuéllar, and D. Suárez, “Research topics and implementation trends on automotive head-up display systems,” Int. J. Interact. Des. Manuf., pp. 1–16, Sep. 2016.

[8]         A. Nordlie and S. Till, “Head-Mounted Displays for Harvester Operators – A Pilot Study,” p. 113, 2015.

[9]         O. Lagnel and J. Engstr, “Bättre arbetsmiljö med Head Up Display,” 2015.

[10]       B. Kress and T. Starner, “A review of head-mounted displays (HMD) technologies and applications for consumer electronics,” in Proc. SPIE 8720, Photonic Applications for Aerospace, Commercial, and Harsh Environments IV, 87200A (May 31, 2013), 2013, vol. 8720.

[11]       R. Van Krevelen and R. Poelman, “A survey of augmented reality technologies, applications and limitations,” … J. Virtual Real., vol. 9, no. 2, 2010.

[12]       H. Seichter, J. Grubert, and T. Langlotz, “Designing mobile augmented reality,” in Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services - MobileHCI ’13, 2013, p. 616.

[13]       C. G. Sørensen, S. Fountas, E. Nash, L. Pesonen, D. Bochtis, S. M. Pedersen, B. Basso, and S. B. Blackmore, “Conceptual model of a future farm management information system,” Comput. Electron. Agric., vol. 72, no. 1, pp. 37–47, Jun. 2010.

[14]       W. Chinthammit, H. B.-L. Duh, and J. Rekimoto, “HCI in food product innovation,” in Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems - CHI EA ’14, 2014, pp. 1111–1114.

[15]       H. Strömberg, “Setting the Stage with Metaphors for Interaction – Researching Methodological Approaches for Interaction Design of Autonomous Vehicles.”

[16]       D. Fallman, “The Interaction Design Research Triangle of Design Practice, Design Studies, and Design Exploration,” Des. Issues, vol. 24, no. 3, pp. 4–18, Jul. 2008.

[17]       J. Zimmerman, J. Forlizzi, and S. Evenson, “Research Through Design as a Method for Interaction Design Research in HCI design research in HCI,” Proc. SIGCHI Conf. Hum. factors Comput. Syst., pp. 493–502, 2007.

[18]       K. Höök and J. Löwgren, “Strong concepts,” ACM Trans. Comput. Interact., vol. 19, no. 3, pp. 1–18, Oct. 2012.

[19]       D. Schon, The Reflective Practitioner: How Professionals Think In Action. Basic Books, 2008.

[20]       R. Sennett, The Craftsman. Penguin Books Limited, 2009.

[21]       B. Buxton, Sketching user Experiences. Morgan Kaufman, 2007.

[22]       N. Cross, “Design Research: A Disciplined Conversation,” Des. Issues, vol. 15, no. 2, 1999.

[23]       J. Bardzell, “Interaction criticism: An introduction to the practice,” Interact. Comput., vol. 23, no. 6, pp. 604–621, 2011.

[24]       M. Wallmyr, “Understanding the user in self-managing systems,” in Proceedings of the 2015 European Conference on Software Architecture Workshops - ECSAW ’15, 2015, pp. 1–4.

[25]       T. Holstein, M. Wallmyr, J. Wietzke, and R. Land, “Current challenges in compositing heterogeneous user interfaces for automotive purposes,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2015, vol. 9170, pp. 531–542.

[26]       M. Wallmyr, “Interactions and Applications for See-Through Technologies - Industrial application examples,” in NordiCHI ’14 Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, 2014, p. Pages 793-796.

[27]       D. Kade, D. Wallmyr, T. Holstein, R. Lindell, H. Ürey, and O. Ozcan, “Low-Cost Mixed Reality Simulator for Industrial Vehicle Environments,” 8th Int. Conf. VAMR 2016, Held as Part HCI Int. 2016, vol. 9740, pp. 597–608, 2016.

[28]       M. Wallmyr, “Exploring interaction design perspectives on heavy vehicles,” 2017.