OTOinsights

Evaluating User Experiences in Games: Workshop at CHI 2008

On April 5th, members of OTOinsights t=zero partnership with the Indiana University School of Informatics participated in a workshop on “Evaluating User Experience in Games” hosted by the International Conference on Human Factors in Computing Systems (CHI 2008) conference in Florence, Italy. CHI is the largest and most prestigious conference in human-computer interaction (HCI), with over 2,000 participants from dozens of countries.

At the workshop, we presented and discussed our plans for executing a multi-modal evaluation of game play experience. Following presentations from the other workshop participants, we participated in a substantive discussion about the current state of game play experience research and practice. The remainder of this post is dedicated to some of the insights from that discussion and a reflection on how the t=zero team’s work is reflective of the latest changes to player experience evaluation methods.

A major theme from the workshop centered on the difficulty or need to distinguish immersion from engagement and determining the effects of each on player experience. Some workshop participants interchanged the terms immersion and engagement in reference to a singular concept; however, other participants saw immersion and engagement as distinct forces. To some, immersion is simply the extent to which a person prefaces the current media over other information (playing a game and not hearing someone yell for you) while engagement is something beyond immersion that fully brings a player into the game world. Participants concluded that we should establish a shared understanding of where immersion and engagement begin and end and to what extent each of those concepts (or a unified concept) affects the game play experience. Without such an understanding, valid and useful measures for evaluating player experience with games will be difficult to develop.

Numerous methods used for evaluating experience with digital media are in use. For example, with the assistance of Quantemo, we simultaneously measure 5 or more biophysical modalities and combine that information with additional behavioral research methods. One identified weakness in all of those methods is a reliance on an expert’s interpretation of the results. Researchers are crucial to the process of evaluating player experience, but in some situations players themselves are best equipped to interpret results generated from our work. Who, then, is the “expert” that interprets the data? A great example from the workshop came from the use of eye tracking equipment in the usability test of a video game. Eye tracking equipment readily shows a researcher “what” a player is looking at, but cannot as easily address “why” a player is looking at an object. Showing eye tracking footage to players and asking why their visual patterns change at any given moment may reveal meaningful information that even expert researchers cannot see.

The final issue brought up during the workshop concerns the difficulty of interpreting biophysical signals as a measure of player experience. Researchers at the workshop commented on the difficulties of relying on any individual biophysical measure to measure changes in experience. Individual measures, especially galvanic skin response, can be susceptible to even subtle changes in a research environment and may not accurately or entirely reflect a players experience with a game due to the noise added to the data by other stimuli (air temperature, background noise, etc). The multi-modal method that our Quantemo lab uses for player experience evaluation was judged by participants to be a promising example of how to compensate for the shortcomings of individual biophysical measures. Monitoring multiple biophysical signals and supplementing those measurements with traditional, vetted behavioral research methods yields results that can be viewed with increased confidence by stakeholders and researchers alike.

The t=zero team was extremely happy to have the opportunity to participate in the workshop and share our experiences with peers from around the world. Methods for evaluating experience with digital media are growing and changing at a rapid pace and events like this workshop help all of us to understand the mutual challenges that we face as a field. While we are currently focused on creating a shared understanding of our terminology, leveraging research participants as knowledge co-creators and developing robust and reliable methods for evaluating experience, you never know what the next challenge will be. Participating in this workshop gives us confidence that t=zero is well positioned to tackle the current and upcoming challenges created by the ever-changing digital landscape.

Tyler Pace, Shaowen Bardzell, Ph.D., Jeffrey Bardzell, Ph.D.

OTOinsight's Serious Games for Marketing Presentation: Corporate vs. Amature Builds

OTOinsight's t=zero research division bowed its initial set of findings focused on Serious Game Marketing by comparing corporate vs. amature virtual world builds at the MITX Digital Marketing Conference (hosted by UK Trade & Investment at the British Consulate in Cambridge, MA.).  This is an early insight to what will be the inaugural report that t=zero will publish in Q1 2008.   

---

[NOTE: If your company is interested in participating in one of the limited sponsorship positions available of the above scheduled research, please contact [email protected].]

---

"Serious games" refers to the use of games and game technologies for non-entertainment purposes. Traditionally, the education, health, and military sectors were the primary actors in this domain. But in the past few years, marketing has arisen as a major subdomain of this area.  This presentation compares player engagement in some of Second Life's most successful user-generated areas compared with some of the more ambitious corporate-sponsored efforts in Second Life.

Download a PDF of the Presentation by click on the image below:

OTOinsight's t=Zero group to lead Game Expereince Research Workshop at CHI 2008 in Florence, Italy

Yesterday, OTOinsight's t=Zero was informed that it's  paper titled "Making Player Engagement Visible: A Multimodal Strategy for Game Experience Research" was accepted for presentation at the 2008 CHI Conference in Florence, Italy (April 5th-10th).

The workshop that t=Zero will lead will focus on  the relationships among neurological, physiological and cognitive assessments of engagement in ongoing and short duration gaming experiences. It is a centerpiece of an iterative strategy toward understanding and modeling relationships among different engagement measures. The research will lead to design proposals for model-based assessments of engagement calibrated to individuals’ responses. 

You may download and read our entire research proposal here: Making Player Engagement Visible: A Multimodal Strategy for Game Experience Research

Ethnography in Industry: Notes from EPIC2007

EPIC (Ethnographic Praxis in Industry) is, as its name suggests, a conference about the implications and uses of ethnography in industry. I was a little surprised (and pleased) that about 80% of the presenters and participants were from industry, and only about 20% were academics. Intel, Microsoft, and IBM are very well represented here, as are less-than-household names, such as consultants, who are nonetheless doing some fascinating work.

The upshot of this is that the problems were quite grounded in contemporary business practices and problems. A sampling of some of the issues around which ethnography was used to improve understanding include the following:

• Sales pipeline management (IBM)
• Mailing out 38,000 individual retirement packages in 7 days (XEROX)
• Researching and modeling medical care ecosystems (Intel)
• Implementing computer automation in wastewater plant management (University of Southern Denmark)
• Teaching ethnographic practices to system engineers to improve their understandings of user needs (Fujitsu, PARC)
• Improving a real estate firm's direct marketing strategy (Ricoh)

There was an interesting tension that many of the researchers seemed to be facing. On the one hand, their work was being used to help develop models for complex business practices. On the other hand, as ethnographers, they wanted to focus on concrete situations and contexts and the real, flesh-and-blood people within them. From my perspective, one way that this tension got addressed was to work proactively to improve communication between managers (who want the models) and employees, on whom the models are ideally grounded and in any case who will have to live with them once they are developed. Stated more abstractly, the ethnographers seemed to want to make a distinction between managing complex processes (which is seen as good) and implementing rationalist control schemes (which are seen as inhuman and bad).

Another major issue is one of legitimation. How can ethnographers convince managers and marketing leaders to take them seriously? How do they justify their work both intellectually (methods, data, etc.) and also from a business perspective (actually leads to better business processes or products)? Complicating this argument is the perceived conflict between the reductionist, abstract models that managers and marketing professionals want and the rich, individual "thick" and nuanced descriptions that ethnographers value and provide. Another way of saying this is that there is a lot of thinking about how ethnographic research can, should, does, or fails to connect to business cycles, that is, there is a lot of thinking about ways that ethnography can have real business impact.

It may appear from this post that there is an ethnographer versus managers and marketing professionals, good guys versus bad guys rhetoric at the conference; that is not the mood here and is instead a misleading artifact of the way I have tried to boil down the complex dynamics that I am seeing. The managers and marketing professionals are hiring and/or collaborating with the ethnographers, whether they are in-house researchers or consultants. So the managers and marketers, too, seem to want to distinguish between (a) managing complex processes and (b) implementing inhuman rationalist control schemes. In that regard, they and the ethnographers share a common value: the two groups just engage with it at different levels.

--Jeffrey Bardzell, Ph.D., Indiana University

ANA 2007 B-to-B Conference: Harnessing the Power of New Media Platforms

Last Thursday I had the honor of sitting on a panel titled ""HARNESSING THE POWER OF NEW MEDIA PLATFORMS" at the 2007 Association of National Advertiser's (ANA) B-to-B Conference in Chicago, IL.

Joining me on the panel were: Barbara Basney (Director, Global Advertising Xerox Corporation) and Ellis Booker (Editor B-to-B Magazine).  Frank Dudley (Vice President of Marketing Guideline Research) moderated.   Our talk centered around the results of a joint ANA/Guidlines/B-to-B Magazine study that explored the use of search, social networks, user-generated content, podcasts, blogging, etc. within B-to-B and B-to-C marketing efforts.

You may download the results of the study here:

DOWNLOAD PDF  | DOWNLOAD POWERPOINT

Overall findings include:

• For B-to-B marketers, a company’s website is the oldest and most often used new media platform that B-to-B companies are using.  Email marketing, online ads, search engine optimization, paid search engine marketing and webinars are also widely used.  Podcasts, blogs and video on demand look to grow over the next year, while little interest is shown in mobile and second life/social networking media platforms.
• B-to-C companies are also using their own website, as well as email marketing, and online ads. SEO-organic and SEM-paid have been in use for the past 3 years.  Blogs and video, both on-demand and viral, look to grow over the next year.  There are no plans to use webinars, Wiki or Second Life within the next year.
• Comparing B-to-B to B-to-C: significantly more B-to-B marketers have been using email,,SEO-organic and webinars for more than 3 years, and have begun to use blogs.  They are also significantly less likely to use mobile, social networks or viral video in the next year than B-to-C marketers. B-to-C marketers have begun to use these platforms in the past year and the trend will continue.
• Of the B-to-B marketers that use these new media platforms, Brand Building is most frequently accomplished through blogs, second life, social networks, viral videos, wiki and their own websites.  Demand Generation is accomplished through email, online, SEO, SEM, and video on demand– while companies use blogs, podcasts and webinars to generate Customer Loyalty.
• Of the B-to-C marketers that use these media, blogs, podcasts, second life, viral video and wiki are use to accomplish Brand Building.   SEO and SEM are used for Demand Generation. And Customer Loyalty is mostly accomplished through emails, RSS feeds, and blogs.  Cross Selling is an objective for emails and webinars.
• Comparing B-to-B to B-to-C, B-to-B marketers use email and webinars for demand generation significantly more than B-to-C.  Of the all marketers who use mobile and podcasts, significantly more B-to-C marketers use them for brand building.
• Effectiveness varies greatly when looking at B-to-B vs. B-to-C companies.  While both have seen success from their own website, B-to-C companies have seen significantly more success from paid key word searches, online ads, and social networks, B-to-B companies have seen success in webinars.
• B-to-C companies typically look at downloads and registrations to their website and specific events.  While B-to-B companies are more likely to measure effectiveness of new media platforms through purchase behavior, site visits and time spent.
• B-to-B Companies spend significantly less on media than B-to-C companies.  As a percent of revenue, they spend an average of .58%, while B-to-C companies spend an average of 1.18%.
• Currently, B-to-B companies are significantly more likely to allocate a higher percentage of their media budget (more than 20%) to new media platforms as compared to B-to-C companies.
• All marketers expect to spend more on new media platforms in 2007 compared to 2006, with significantly more B-to-C marketers trending towards an increase.
• Of the marketers that expect to spend more on 2007,  B-to-B marketers will spend it on their own websites, email and online ads.
• Typically the responsibility for managing new media platforms is added to an internal group.  New internal groups are formed about 15% of the time.
• Internal Marketing Departments and Interactive/Digital Media Agencies drive the use of new media platforms for B-to-B companies.  B-to-C companies new media strategies are also driven by Internal Media Departments, but also by Interactive/Digital Creative Agencies.
• All marketers are concerned about their lack of experience with new media platforms, followed by their inability to prove effectiveness/ROI.

Jeremi Karnell-President, One to One Interactive

Towards a Sociology of Massive Cultural Production

I've spent the last two days devoted to the work of French sociologist Pierre Bourdieu (1930-2002), who wrote extensively on what he calls the field of cultural production. I think "cultural production" is an important category for study, because I think a lot of recent work on Web 2.0 (such as Tapscott & Williams' Wikinomics book and Benkler's Wealth of Networks) focuses too narrowly on professional collaborations in the discovery, production, and dissemination of "knowledge" and "information." These are important contributions, but many of the big sites aren't about knowledge or information. They are fundamentally about experience: of culture (YouTube, Newgrounds), of friendship (Facebook, MySpace), of shared fantasy (Second Life, World of Warcraft), etc. The Web isn't an encyclopedia: along with the home and the workplace, it's a primary environment in which we live, play, love, learn, and express ourselves. Thus, the sociology of massive cultural production is as necessary to study as the economics of networked collaboration. Which brings us back to Bourdieu.

Among Bourdieu's goals are disentangling and even modeling the incredibly complex inputs that lead to epochs of cultural production, such as late 19-century French literature. These inputs include things like the following:

• The intentions and dispositions of the artist or author
• Economic and political contexts, including material production, income, and influence of dominant class ideology
• The artist's particular use of a generic "language," such as the visual and thematic conventions of a contemporary science fiction movie
• The role of the critic in justifying, discovering, or downright creating the value of a work (think of Oprah's effect on a novel's sales and prestige)
• The education, dispositions, and tastes of the audience and how they combine to create demand

The obvious strength of Bourdieu's approach is that it avoids reducing cultural production to an oversimplified account, such as "the author's intention," or "serving the needs of the dominant economic class," or "whatever happens in the cognition of the reader/viewer," etc. Reductiveness is particularly a problem in scientific sampling, which seeks through its "operational definitions" to place discrete boundaries around phenomena whose very essence is the struggle to create boundaries; in other words, for Bourdieu, sampling of phenomena relating to cultural production predetermines the data, rather than enabling its representation.

The obvious weakness of Bourdieu's theory is that this is not an easy model to go out and apply. Fortunately, Bourdieu does apply his model in analyses of French literature, which we can, in turn, at least try to emulate in a domain of cultural production that we all care about, say, Newgrounds animations, SecondLife builds, or MySpace mini-apps.

So, simplifying for clarity and brevity, Bourdieu characterizes the "field" of cultural production as a "space" in which actors (artists, critics, etc.) struggle. They struggle not only to promote their own ideas over others, but also to draw boundaries of inclusion and exclusion as to who has a voice, who belongs in the struggle. This field he represents as a two-dimensional graph. Different agents, through an interaction between their own predispositions and the objective world of options they have in front of them, position themselves in this space.

The X-axis maps the range of popularity, from no audience to a large audience. Related to that are economic matters: large audiences tend to mean lots of money but also lots of market interference on the artist's vision. Small or no audiences mean lots of autonomy for the artist--who can do whatever she or he wishes--but at the expense of economic profit.

The Y-axis maps the degree of consecration. High consecration refers to academic and institutionalized consecration: the work is recognized as "high art," "worthy," or "important"; it also correlates to the category of the "old." Low consecration is associated with youth, the merely popular, throwaway culture.

Now, here is my central question: Can Bordieu's model be used to represent massive cultural production in the era of Web 2.0? Here are some objections that have occurred to me:

• Both French literature and contemporary mainstream film, books, and comics have high barriers to entry, that is, restricted access; that clearly is not the case (at least, not in the same way) for Web 2.0 creativity.
• Institutionally, we know how to handle all aspects of 20th century mass media: production, distribution, and consumption. Thus, we have established protocols for "consecration." With massive cultural production, the relationships between the relevant institutions (such as the hierarchies of blogs and wikis on the one hand versus universities and journalism on the other) are anything but clear, and the protocols for consecration are likewise confused.

That's all pretty abstract. Let me make it more concrete. In April, Jeremi Karnell, Carl Marci, and I did a presentation for MITX on the Numa Numa dance viral video. It doesn't get any more establishment than the president of a digital marketing services firm, a corporate researcher, and a university researcher coming together to present under the auspices of something like MITX. In Bourdieu's model, that would be a consecration of high order. Did we consecrate Numa Numa in that presentation? Surely we did on some level, but what does "consecration" even mean in this context?

Another example. Earlier this year, YouTube held awards for its "best" videos of 2006, as divided into seven categories. Each category had 10 finalists. That's 70 finalists taken from around 24 million videos uploaded that year. What possible protocol could justifiably identify the best 0.0003% of YouTube videos, from a cultural standpoint? And once this task is complete, for better or worse, again we are faced with the question of what kind of consecration is it to be a finalist of a YouTube video award.

Returning to the original question: Can Bourdieu's model be used to expose and represent the field of massive cultural production? The answer is yes, but with a caveat. Many of the underlying characteristics and concepts of Bourdieu's model will apply (e.g., field, habitus, position-taking, cultural capital), but identifying the specific categories of the space of massive cultural production, that is, finding alternatives to, or at least redefinining "degree of consecration" and large versus small audience, will require deriving or defining the new categories empirically. And that's gonna be a bit of a job. And I've got six weeks to do it. Hrm.

One to One Interactive Acquires Quantemo

On Monday, April 16th, One to One Interactive formally completed its acquisition of the Quantemo Usability Lab.  The Quantemo lab, its brand and technology platform will be incorporated as the foundation for a new neuromarketing research offering that is currently under development at One to One's professional services division OTOi. 

The Quantemo acquisition is One to One's latest step in marrying science and marketing.  This past year One to One has engaged in a number of bio-measures studies (see: Bio-Measures Able to Predict with 77% Accuracy an Online Communities Rating of Viral Video Content) that have sought to analyze the physical responses test subjects experience when exposed to certain media stimuli.  This data has in turn been used to predict, with high levels of accuracy, consumer behaviors across a variety of media channels and platforms. 

"We're excited to evolve the Quantemo technology to measure engagement across a number of emerging digital platforms, including social networks, blogs, podcasts, widgets, online video, rich media advertising, and virtual worlds such as Second Life.  Furthermore, we intend to extend the offering to measuring traditional offline marketing channels, our aim is to harneess the accuracy afforded to us by neuro-research to optimize our clients' marketing programs so that they are as relevant and engaging as possible in this new consumer controlled media landscape." - Jeremi Karnell, President

To date, the field of neuromarketing research has been predominately defined by researchers who use functional Magnetic Resonance Imaging (fMRI) to scan the brains of test subjects as they look at various products and advertisements.  The information gleaned from neuromarketing research is intended to provide deeper insight into the human brain for purposes such as more effective advertising and brand loyalty campaigns.  A significant barrier to the evolution of this research method has been both the high costs involved in a typical fMRI study as well as the method itself which limits research studies to only take place at the university or medical research facilities that house the large and expensive fMRI hardware.  One to One, through its acquisition of Quantemo, seeks to help expand the field of neuromarketing research by incorporating newer and less costly methods that measure skin conductivity, respiratory rates, heart rates, and EEGs as both an augmentation to and proxy for fMRI studies.

“I am very pleased to see One to One Interactive  taking Quantemo ™ to its full potential , with a state-of-the-art facility  in Boston and increased resources for very scalable study options. The future of digital marketing relies heavily on understanding and improving the user experience, and Quantemo™ has a unique position in the field of online research as the first method to objectively assess user emotions and mental effort. One to One Interactive and Quantemo will provide enormous mutual benefits that will ultimately help clients in terms of availability, scale and methodology.”  Theodore Agranat, Quantemo Founder

OTOi's neuromarketing research facility will be located at its corporate headquarters in Charlestown, MA.  The facility will include state-of-the art eye tracking, neurological and biological feedback platforms  as well as video and click-stream capture and analysis capabilities.  The facility will be used both for client-based studies as well as independent research conducted by OTOi.  OTOi's neuromarketing offering will also boast a mobile lab capability that will allow for research to be conducted at any location.

Alternative Media Growth Outpaces Traditional Media

Below is an article that was published today by the Center for Media Research:

"According to exclusive research by PQ Media, spending on alternative media strategies surged 16.4% in the first half of 2006 to an estimated $53.37 billion compared with the same period of 2005. PQ Media estimates that spending on alternative media will accelerate in the second half of 2006, to a full year forecast of growth at 18.5% to $115.77 billion.

Alternative marketing is expected to increase 17.6% for the year, fueled mainly by mobile and interactive marketing, according to the PQ report Alternative Advertising & Marketing Outlook 2006.

All four broad segments of Alternative Media - entertainment and out-of-home advertising, online advertising, branded entertainment marketing, and mobile and interactive marketing - posted double-digit growth for the year. Branded entertainment, including product placement, event marketing, event sponsorships, webisodes and advergaming, is the largest segment of alternative media, and is expected to grow 15.5% to $51.62 billion in 2006. The value of product placements will reach $5.71 billion at year's end, up 27.6% from the 2005 level.

Alternative Media Spending & Growth (Spending $ Millions)
	2000	2005	2006
Alternative Advertising	$23,391	$40,200	$48,189
% Change	-	21.4%	19.9%
Entertainment & Out-of-Home Advertising
	$14,065	$22,647	$25,916
% Change	-	12.8%	14.4%
Online Advertising	$9,326	$17,553	$22,273
% Change	-	34.6%	26.9%
Alternative Marketing	$29,518	$57,455	$67,577
% Change	-	17.0%	17.6%
Branded Entertainment Marketing	$25,066	$44,698	$51,618
% Change	-	16.3%	15.5%
Mobile & Interactive Marketing	$4,452	$12,757	$15,959
% Change	-	19.7%	25.1%
Total	$52,909	$97,655	$115,766
% Change	-	18.8%	18.5%
Source: PQ Media, June 2006

Patrick Quinn, president of PQ Media, said "The data gleaned from our Alternative Media Network indicates that brand marketers are accelerating the shift of media dollars away from conventional to newer media using digital technology to reach youth and influential demographics."

The report notes that, while alternative media spending surged in the first half of the year, expenditures on traditional media increased only 4.5% in the first half of 2006, reflecting the shift in advertising and marketing spending. Traditional media expenditures on various print and broadcast vehicles, and are expected to remain in the single digits for the full year as well.

An executive summary of Alternative Advertising & Marketing Outlook 2006 is available online here."

Fifth Study of the Internet by the Digital Future Project

The Fifth Study of the Internet by the Digital Future Project was recently published by the USC Annenberg Center for the Digital Future. Major findings include overall Internet use continues to increase; e-mail reigns as most popular online activity; broadband reaches highest level for online access.

You may download a free 19 page overview of the study in PDF format here.  Those interested in getting a copy of the full report providing detailed comparisons and trends as well as analysis of current developments can purchase the complete report by following this link. 

Advertising 2.0

   Karl Long, at Futurelab's Blog, has found a nicely presented white whitepaper called Advertising 2.0.  Authored by Paul Beelen, a Spanish language blogger,  the paper covers the democratization of media, mobile technologies, RSS, Word of Mouth, contextual advertising, and blogs.

Jeremi Karnell-President, One to One Interactive