Cover Page

Digital Tools and Uses Set

coordinated by
Imad Saleh

Volume 3

The Image-Interface

Graphical Supports for Visual Information

Everardo Reyes-Garcia

image

Preface

This book studies interfaces as images and images as interfaces. By interfaces, I mean specifically graphical user interfaces; the systems which we use to interact with information displayed on an electronic support.

My interest on this topic has grown over the years and it reflects both personal motivations and professional work as a teacher and researcher. As someone who has always been fascinated with technological objects (I remember as a child programming a VCR, discovering how to change the time in an LCD watch and climbing to the rooftop to redirect the TV antenna), I think one of the best ways to approach a domain is to “touch it”, to do things with it, either under special supervision or under our own risks. I hope this book will provide inspiration to the reader in producing his/her own examples and prototypes, either by hand or with software applications or with computer code.

I have had the opportunity to discuss many of the topics included in this book with colleagues and students. I learned a lot about computer graphics and 3D software while serving as a program director of the BA in Animation and Digital Art at Tecnológico de Monterrey in Toluca, Mexico. Later, I could appreciate the intricacies of web design and human–computer interaction for the web when I was appointed as a program director of the MA Interface Design at Université Paris 13. More recently, data culture, data visualization, digital humanities, semiotics, cultural analytics and software studies appear on a daily basis at the heart of my current professional family at Université Paris 8.

I want to extend my thankfulness to colleagues and students around the world who have kindly invited me or deposited their trust in me to collaborate somehow. Friends in Mexico at Tecnológico de Monterrey, UAEMex, Centro Multimedia, CENART, UAM Xochimilco; in USA at Cultural Analytics Lab and UCLA; in Belgium at Université de Liège; in Italy at Università di Torino; in UK at King’s College and Winchester School of Arts; in Sweden at Lund University; in France at médialab Sciences Po Paris, ENS Paris, ENSAD, MSH Paris Nord, IEA Paris, MESHS Lille, Université Paris 3 Sorbonne Nouvelle; in Brazil at UFRJ; in Spain at Universidad de Castilla La Mancha; in Argentina at Universidad Nacional de La Plata and Universidad de Buenos Aires. I hope we all meet together at the same space and time one of these days.

Everardo REYES-GARCIA
August 2017

Introduction

This book is situated within the period of time that has been influenced by the information age. The time span has developed from the invention of digital computers to the contemporary scene of massive generation and storage of data. From the standpoint of information and communication studies, this period of time is deeply related to electronic media. In this book, we understand data as digital information and electronic media as the means to access and make sense of it.

In our present study, we are interested in a particular modality of electronic media: the graphical mode. Although most of the physical world perceptible to human senses has been simulated and described with electronic media (sounds, smells and tastes can all be quantified and stored as bits and bytes for a latter representation), the vast majority of developments in electronic media have taken place in visual form. We look at the computer screen as the main material support where digital information is represented in order to convey meaning. Nowadays, screens have become pervasive. We see and interact with them in laptop and smartphone displays, in desktop monitors, as beamer projections, in LED displays in elevators, city signage, museum exhibitions, park attractions, airports and public transportation. Screens can be public or private and shared or intimate (such as head-mounted displays).

The kinds of images depicted in computer screens are closely related to the display technologies that underlie them. As computing power increases, for instance, supporting more complex and multiple tasks, larger files or higher resolutions, the look and style of images reflect on those advances. Consider the differences between a 1980s videogame graphics and today’s consoles. Or let us evoke the first computer animations that were used in music video clips to compare them with today’s amateur and blockbuster films. The World Wide Web has also witnessed this evolution. The first images supported by web browsers are not the same in quality and resolution if we consider recent developments in technologies such as CSS3 and WebGL. What we can observe in all these cases is foremost a quest for realism and hyperrealism. Technicians and artists have devoted great efforts to make images look “more real”, i.e. to represent the physical world with more detail, with the highest possible similarity.

But at the same time, there is another kind of image in electronic media. In order to produce those realistic images, creators use the same computer screen in a different manner. They use graphical interfaces. Whether the authoring software is based on user interface elements such as windows, icons, menus, pointers, or uses a command-line console, it is the same rectangular space of the screen that is redrawn to allow the interaction between data and meaning.

We are interested in studying the computer graphical interface as an image. For that matter, we have to take a step back from the representation of “realistic” images. Graphical interfaces are visual supports of information and we look at the way in which the interfaces are constructed to convey meaning. Throughout the history of computing, the community of researchers in human–computer interaction (HCI) has developed models based on cognitive and linguistic capacities in order to describe efficient ways to create graphical interfaces. Our study intends to contribute to the domain by relating interfaces to images. In other words, it is about regarding the visual constituents of interfaces before and at the moment they are identified as interfaces.

To illustrate our purpose, it is possible to list some visual constituents that exist not only in interfaces but also in any other visual form of image. Following the theoretical tradition of visual semiotics, we can talk about plastic categories: colors, forms and textures. For each category, there are smaller units of analysis (colors can be described according to numerical values of saturation, brightness, “redness”, “greenness” and “blueness”). It is interesting to note that, when we describe a picture or an interface, we can establish the relationships between units at the same granularity level or we can reassemble them in other semantic units. Imagine, for this case, that the position of forms in an image motivates the perception of vectors of direction, by means of graphical abstraction and Gestalt principles. Hence, we could now examine “vector” patterns in different images, for example.

Studying interfaces as images means that there is a visual meaning behind the construction of any authoring and exploring environment. Whereas one of the main influences of HCI in everyday interfaces can be perceived through the WIMP paradigm (Windows Icons Menus Pointers), it is necessary to broaden this perspective. Authoring software has a long tradition that builds on reading–writing schemes. The pioneering research on hypertext systems during the 1960s – long before the invention of the World Wide Web – introduced models for nonlinear writing and reading. Among such models, we may cite the relational, the taxonomical and the spatial.

While the first authoring environments were mainly conceived for textual content, the introduction of different types of media came with an explosion of paradigms and visions. With the support for images, video and audio, creators could now explore storytelling in different ways. However, it was necessary to think about new metaphors to signify actions on abstract objects: to link, to import, to animate, to copy, to delete and to paste. Closely related to graphical interfaces, there is also the development of programming languages and programming paradigms. Starting from declarative and functional languages, we then saw the emergence of generative and object-oriented languages. The syntax is, of course, a visual matter. The kind of programming marks, such as dots, commas, brackets and curly brackets, is a way to represent dependencies, inheritances and encapsulations of data. It is not surprising that more recent graphical programming languages were introduced: boxes connected to other objects by means of links and edges.

As it occurs with photographs and paintings, an interface as image can be approached as a diagram, symbol, index or other category of sign. The image of an interface conveys in itself the traces of its production. It is possible to relate the operative system, the authoring software and the complexity or simplicity of functionalities. It is also a symbol of its time: the visions foreseen to be accomplished with electronic media; the ideologies associated with such productions. Interfaces, through language and affordances, are meta-images of enunciation that speak to their users, in a similar instance as mirrors, reflections, windows and depictions of the painter inside the painting.

Today, any user of electronic devices can take photographs, add visual effects, edit them, share them and comment on them. The new user has become a sort of media producer. Graphical interfaces have become easier to use, at least at their basic level. While we can study the evolution and the effect of portable devices in the esthetics of social images, we are more interested in: how do designers produce and distribute data and images? Which functionalities are integrated and which ones are left behind? How are our new images and interfaces shaping the emerging post-digital culture?

Designers in the information age are expected to invent new ways to explore information and to make sense of data in environments that change constantly. As it happened to pioneers of authoring systems, designers need to investigate new metaphors, new platforms, new actions that are possible with digital and networked technologies. In this context, designers and developers have embraced the World Wide Web as a privileged environment to create and distribute their creations. The technological possibilities of the contemporary web, along with the development of computing devices, allow experimenting with and dreaming about such new models. The tradition of multimedia authoring systems converges today with massive quantities of social and historical data. The kind of interfaces that designers produce is related to reading–writing software in the sense that they are interactive, they support explorations, discoveries and insights that might have passed unattended otherwise. Designers are like image craftsmen because they take decisions about visual composition, arrangements and metaphors. Moreover, they must reflect on the potential outcomes of the interface: the kind of data, format and media, the parameters that can be modified, the series of actions that a user can perform, the pertinent vocabulary, language and syntax, etc.

While common users have increasing access to cameras and image editing software, designers have a wide variety of production tools at their hands. Some research is necessary to investigate the kind of software available (desktop, mobile apps and web apps) as well as programming languages, APIs and libraries that handle data and communicate with the ever-changing environments. Although many software and production tools are developed with a particular objective in mind, it is interesting to see that tools can be mixed and remixed. While the material support remains constant (the screen), it is when we pay attention to its configurations that innovations take place.

Thus, where we position our study design will be neither a specific discipline nor a bunch of recipes and ready-made toolkits to add visual effects or improve the visual style of a project. Design will be understood as a way of life. It is more like a reflective thinking posture about our practices in order to produce new kinds of explorations and actions. It is about questioning defined models to propose unexpected schemes. It is about producing meaningful tools that communicate aptly in the context in which they function. It is about going beyond the pre-defined figure of a “user” and to consider her/him/them as “visitors”, “wanderers”, “researchers”, “learners” and “meaning-makers”. It is about taking seriously and responsively the work of tool-making; to remember that through our work, we are also shaping ways of being (as it was suggested by Winograd and Flores [WIN 86, p. ix]).

As we said, screens have become pervasive; they are material supports of representations. But at the same time, the screen is alive; it reconfigures itself as formal support. The computer screen is today’s canvas of painters, designers, researchers and students in information and communication. Interfaces are the pervasive kind of images of the information age.

Contents of the book

Chapter 1 compiles relevant insights forged within the tradition of sciences, humanities and media studies about the study of images. Although this book is not about interpreting cultural images, it does take into account research that pays attention to material and technological descriptions of images. This is important because we will identify levels of analysis and methodologies that in other chapters will be used as strategies to design and produce visual interfaces. In this respect, a brief discussion regarding the notion of interface is also contemplated.

Chapter 2 is devoted to the foundations of graphical information. In this part, we consider images as data and digital information. We look at the technical aspects that define a digital image: from data types and data structures required to render an image on the computer screen to seminal algorithms and image processing techniques.

Chapter 3 adopts a pragmatic perspective towards image-interfaces; it studies how image-interfaces are practiced, observing the different manners in which those elements are put together, notably through software applications. This chapter is organized into different categories according to practices of imaging, for example, image editing, parametric design, and also writing and web environments. As long as we talk about image-interfaces, we are not only concerned with the results of some software manipulation, but also with the graphical elements that accompany the user to achieve such transformations.

Chapter 4 pays special attention to those cases where the image being manipulated acts as the graphical interface. It deals with multidisciplinary fields at the crossroad of several disciplines, such as graphic design, data science, computer science and humanities. The kinds of results that are produced from these perspectives include data visualization, networks, graphs, infographics, cultural analytics and data art.

Chapter 5 presents our own productions and experiments created within the context of image-interface and data visualization. It covers a range of prototypes, screen shots and sample codes for scripting software applications, integrating data and media visualization, and extending the generated images beyond the screen, to “data physicalization” via 3D printing.

The conclusion points to perspectives that still have little or recent exposition to information sciences. The term “visual hacking” is used to make reference to speculative advances in image-interfaces inspired by the non-visible electromagnetic spectrum, non-Euclidean geometry, obfuscated code and exotic data structures, and critical and radical design. Finally, the appendix section summarizes the software applications in the form of tables, which are discussed in Chapter 3. It also includes the tables of JavaScript libraries, web links to data visualizations and tools.