Judgement devices and the evaluation of singularities: the use of performance ratings and narrative information to guide film viewer choice

As noted earlier, Jeacle and Carter (2011) find that with the advent of systems like TripAdvisor, the opinion of the layperson has been privileged over that of the expert. Our findings differ in a subtle, yet important way. Specifically, we find that the similarity in taste between the film viewer and the reviewer (be they expert or layperson) appears to be the key consideration in determining which reviewer’s viewpoints are privileged over others. As a result, individuals frequently track the viewpoints of reviewers whose tastes are similar to their own, or rely on the opinions of personal contacts that also have similar views. IMDb allows users to see all reviews by a particular user at the click of a mouse. Some websites have the capability to ‘follow’ users with demonstrably similar tastes, a capability described as ‘social cataloguing’ (see for example, Giustini et al., 2009; Spiteri, 2009). Spiteri (2009: 52) describes social cataloguing as being a tool to “…allow members to not only share publicly their cataloged inventories, but to post reviews and commentaries on the items posted, create and participate in discussion groups, and tag or classify the items cataloged. In other words, these sites serve as a user-designed, interactive, and shared catalog.” Social cataloguing is used on a number of websites, perhaps most notably, Goodreads, and allows the user to electronically ‘follow’ users whose interests and tastes are similar to their own. Clearly sites with a strong social cataloguing capability offer the potential for users to more easily identify individuals with tastes similar to their own, and to make decisions about consuming singularities accordingly. Ultimately, therefore, these sites enable users to reduce the uncertainty associated with the choice of singularities (c.f. Karpik, 2010).

We see our findings in this regard as being analogous to the literature on the imperfection of performance measures. In the face of seemingly ‘imperfect’ performance measures, prior research has shown that users of the information can respond in a number of different ways. One approach is that users ‘make do’ with the information despite its known imperfections (Andon et al., 2007). We see this as being similar to our fourth category of response, whereby users recognise the limitations of the judgement device/s, but continue to use them anyway. Another approach for users to deal with imperfect performance measurement information is to rely on other information (Bürkland et al., 2010) which we see as being similar to our first response. A third approach is to refine the problematic indicator/s (see for example, Jordan and Messner, 2012). While in our context, the film viewers did not have the ability to refine the formula used to calculate the ranking (this is controlled by IMDB), they did have the ability to potentially influence the film’s rating by providing a lower score, as well as to change the balance of existing narrative performance information available by providing a negative review (that is, our second category of response).

Where our results in this area perhaps deviate most sharply from the prior literature on the imperfection of performance measurement information, is in relation to our third category of response, where individuals went back to IMDb to review the information, and in one case, re-watch the film a number of times in an effort to better understand the viewpoints of others about a particular film. We see this approach as arguably similar to a post-completion audit (c.f., Haka, 2006), whereby the judgement device is evaluated ex-post.
6.4. Implications for management accounting, limitations and future research
Our study focuses on how filmgoers use a combination of judgement devices (particularly numeric ratings and narrative information) to make decisions about which film they should see. We believe that our findings have some important implications for management accounting research.

At a general level, our findings shed light on how individuals use information about a singularity’s quality to help them make decisions about whether or not to consume that particular singularity. Given the unique properties of singularities, which do not make them amenable to being easily compared or evaluated using standardized evaluation tools, we believe in this way, our study makes an important contribution to the management accounting literature.

More specifically, our findings provide insights regarding how users of information respond to conflicting information about a product’s quality. Our research supports prior research (e.g. Porter, 1995; Robson, 1992) regarding the primacy of numbers in such a decision making context, but also explains specifically the manner in which numeric data are used when the numeric data is inconsistent with the narrative data. Specifically, we find that when users of information were faced with conflicting qualitative and quantitative information about a product’s quality, the numeric data was used either as an initial screening tool, or as a ‘tie-breaker’ to resolve the conflict between the different signals provided by the conflicting information. This issue of how users of information combine narrative and numeric data (particularly when the information provided by these forms of data is conflicting) in their decision making is underexplored in the management accounting research literature. This would appear to be a fruitful area for future research.

Our study also provides insight in terms of the sources of information that decision makers rely on in the presence of conflicting information. Our findings indicate that in such settings, decision users place greater reliance on information supplied by individuals whose judgments in the past have mostly closely matched that of the user. While this is not problematic or harmful in the context of making choices of which film to see, it would be useful to see whether this finding holds in organizational contexts, whereby placing greater reliance on information from those whose opinions are most closely related to your own is perhaps not surprising, but potentially not in the best interests of the organization. Again, future research could explore this issue.

Our study is subject to some limitations. Firstly, we have focused on a single singularity, namely films. It is possible that findings may vary slightly, depending on the nature of the singularity examined. This is particularly the case in relation to our finding in relation to uncertainty, namely that some of our interviewees indicated that they preferred not to consult too many judgement devices prior to watching a film in order to avoid learning too much about the film. We can certainly understand this being the case for watching films – presumably some of the utility one derives from watching a film relates to the element of surprise associated with not knowing too much about it in advance; that is, the removal of too much uncertainty about the film may diminish the cinematic experience. However, in the case of other singularities, such as in the case of, for example, purchasing a fine artwork, one could argue that it is prudent to know as much about the artwork as possible prior to the purchase. To put it another way, the level of quality uncertainty may vary between different singularities. Future research could consider whether our findings in relation to uncertainty actually differ depending on the type and nature of the singularity.

Finally, in the context of making decisions about which film to see, the cost of ‘getting it wrong’, or making a decision to see a film that that the individual did not like, is relatively low. Future research could consider whether our results would hold in circumstances in which the cost of making an ill-informed purchase decision about a singularity were higher, such as in the artwork example discussed above.