Shubham Singh

Efficiency and fairness are conflicting objectives for resource allocation problems. Although well-studied, its ubiquity in numerous applications keeps presenting researchers with new challenges. In this work, we study the relationship of resources with efficiency and fairness and how it affects their tradeoff. We motivate the effect of resources on group utilities and the evaluation measures through the framework of homogeneous functions. Then, we explore the tradeoff together with three decision choices — allocating resources based on needs vs. outcomes, measuring social welfare as maximizing vs. minimizing objectives, and expressing fairness in terms of absolute vs. relative measures. Each choice presents us with a rich, complex setting and different desirable solutions. We illustrate our findings through stylized examples and then a more realistic example based on restaurant inspections.

Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of the our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches.

Most digital service providers offer free accounts to users and then generate revenue by monetizing the data provided by them. However, the way users derive value from their accounts and what factors affect their valuation needs further scrutiny. In this study, we used a novel auction-based methodology to understand users’ valuation of their digital accounts and how this relates to their security decision-making. We conducted a behavioral economics study with n = 66 participants at two university campuses to assess their reasons for being willing to provide researchers with their digital accounts’ credentials in exchange for money – hence, compromising security and privacy protection – and what contextual factors governed their decision-making. We found that the main factors influencing participants’ valuation of their accounts were their beliefs about data and privacy, envisioned threats, and the account properties and utility. Finally, we discuss the context-dependent nature of these factors and their implication on future research.

Generative artificial intelligence (AI) systems across modalities, ranging from text, code, image, audio, and video, have broad social impacts, but there is little agreement on which impacts to evaluate or how to evaluate them. In this chapter, we present a guide for evaluating base generative AI systems (i.e. systems without predetermined applications or deployment contexts). We propose a framework of two overarching categories: what can be evaluated in a system independent of context and what requires societal context. For the former, we define seven areas of interest: stereotypes and representational harms; cultural values and sensitive content; disparate performance; privacy and data protection; financial costs; environmental costs; and data and content moderation labor costs. For the latter, we present five areas: trustworthiness and autonomy; inequality, marginalization, and violence; concentration of authority; labor and creativity; and ecosystem and environment. For each, we present methods for evaluations and the limitations presented by such methods.