Method Overview
BETA operates with two models: a frozen black-box target $f_B$ (e.g., a commercial API) and a lightweight local steering model $f_S$ with full gradient access. We learn an additive visual prompt $\delta$ and optimize it through the steering model's local gradients.
Why naive gradient transfer fails
Gradients from a local surrogate do not transfer directly to a different black-box architecture. The per-example gradient cosine similarity between a ViT-B/16 target and a local ViT-S/16 or ResNet-18 is near zero.
Prediction harmonization
Rather than transferring gradients, BETA fuses output probabilities from the two models into a single harmonized distribution $p_\alpha(x') = \alpha\,p_S(x') + (1-\alpha)\,p_B(x')$ and minimizes its entropy. This exposes a tractable asymmetric gradient pathway: the prompt $\delta$ is updated only through $f_S$'s gradients, while the black-box predictions $p_B$ enter as a data-dependent mixing target.
Why harmonization helps (analytical view).
The mixture $p_m = \alpha p_S + (1-\alpha) p_B$ satisfies the entropy decomposition
$H(p_m) = \alpha H(p_S) + (1-\alpha)H(p_B) + \mathrm{JS}_\alpha(p_S\,\|\,p_B)$.
With the black-box prediction held fixed during the local update, the prompt receives gradients from steering-model entropy and the Jensen–Shannon agreement term. The target therefore guides the update through its predictions while gradients flow through the local steering model.
Stability: consistency and prompt-oriented filtering
Learning prompts from random initialization is brittle. BETA adds (i) a KL consistency term between clean and prompted predictions in the local view, and (ii) a prompt-learning-oriented reliable-and-diverse filter so the prompt is updated only from samples that give a stable learning signal.