Multi-Evaluator Preference Alignment Under Scarce Human Feedback Budgets: Theoretical Frameworks and Aggregation Strategies

Abstract

The alignment of generative artificial intelligence systems with human values and intentions relies heavily on iterative feedback mechanisms. As models scale in complexity, the demand for high-quality human annotation escalates, presenting a critical bottleneck when financial and temporal budgets are constrained. This paper investigates the complex dynamics of multi-evaluator preference alignment under conditions of severe human feedback scarcity. Traditional approaches to preference learning typically assume a homogenous evaluator pool or infinite querying capabilities, both of which are unrealistic in practical deployment scenarios. By introducing a comprehensive analytical framework, this study addresses the dual challenges of annotator subjectivity and strict budget limitations. We propose a resource-aware active selection mechanism coupled with a dynamic evaluator profiling strategy to optimize the information gain from every queried preference pair. The methodology transitions away from mathematical formulas in favor of descriptive algorithmic logic, conceptualizing how uncertainty metrics can guide the routing of specific queries to the most appropriate human evaluators based on historical reliability and cognitive bias profiles. Through extensive simulated experiments, the findings demonstrate that leveraging nuanced disagreement resolution protocols significantly enhances alignment efficacy compared to uniform or random sampling strategies. The results highlight that strategic budget allocation across a diverse multi-evaluator pool accelerates convergence to an optimal reward distribution, providing a viable pathway for democratizing the alignment of advanced computational models without requiring prohibitive annotation expenditures.