[RFC] vulkan: compensate alpha to better match perceptual blend intent

The Vulkan renderer performs alpha blending in linear RGB space, which
preserves hue better than blending sRGB-encoded values directly (as the
gles and pixman renderers do), but unfortunately tends to give a
too-bright result when blending dark and light colors.

(In desktop usage, this especially affects dark, semi-transparent
tooltips, which appear significantly more transparent than expected,
affecting readability if light text underneath shows through.)

This is a novel (I think) approach to compensating for this effect by
adjusting the alpha value of the source texture - basically the result
is that dark semi-transparent pixels are made a little more opaque,
while light semi-transparent pixels are made a little more transparent.
Alpha values of 0 and 1 are unchanged.

I am somewhat new to science of color blending (Björn Ottosson's page,
"How software gets color wrong" is very enlightening) but I think this
approach makes at least a little bit of sense theoretically, and the
result seems to me subjectively to be an improvement.

Analysis from an expert on the subject would be greatly appreciated.

v2: compensate alpha in solid color conversions also
v3: un-premultiply average value for solid color conversion

render/vulkan/README-alpha-blend

@@ -0,0 +1,55 @@
+NOTE ON ALPHA BLENDING IN THE VULKAN RENDERER
+
+Vulkan internally performs alpha blending in linear RGB color space.
+While this preserves hue better than blending in sRGB space, it tends to
+produce a too-bright result when blending dark and light colors. (See
+illustrations in "How software gets color wrong" by Björn Ottosson,
+https://bottosson.github.io/posts/colorwrong/.)
+
+(In desktop usage, this especially affects dark, semi-transparent
+tooltips, which appear significantly more transparent than expected,
+affecting readability if light text underneath shows through.)
+
+This effect can be compensated somewhat by adjusting the alpha channel
+to make dark overlay colors a bit more opaque. The Vulkan renderer
+currently performs this compensation only for sRGB source textures.
+
+To keep the math manageable, the compensation equation is derived using
+a lot of assumptions and approximations, namely:
+
+  1. the perceptual lightness of a given source pixel is approximately
+     the average of the RGB values in sRGB encoding (0-1 range)
+
+  2. alternately, the perceptual lightness is approximately the square
+     root of the average of the RGB values in linear encoding (the power
+     should really be 1/2.4, but 1/2 is close enough)
+
+  3. the lightness of the pixel underneath (in the surface being blended
+     over) is unknown and thus assumed to be 0.5. (This does make the
+     compensation less accurate when the underlying surface is very dark
+     or light, but it's still much better than nothing.)
+
+If we could blend a pixel with lightness value v2 over a lightness value
+v1 in a theoretical perceptual color space, the resulting lightness
+should be simply:
+
+  (1 - alpha)*v1 + alpha*v2
+
+However, alpha blending in linear space instead results in a perceptual
+lightness of approximately:
+
+  sqrt((1 - alpha)*(v1^2) + alpha*(v2^2))
+
+To compensate, we would like to solve for a new alpha value a' where:
+
+  sqrt((1 - a')*(v1^2) + a'*(v2^2)) = (1 - a)*v1 + a*v2
+
+Solving gives:
+
+  a' = ((v2 - v1)*a^2 + 2*v1*a) / (v2 + v1)
+
+Assuming v1 = 0.5 simplifies this to:
+
+  a' = ((v2 - 0.5)*a^2 + a) / (v2 + 0.5)
+
+which is the equation used in the shader (texture.frag).

render/vulkan/pass.c

@@ -63,10 +63,6 @@ static float color_to_linear(float non_linear) {
 		non_linear / 12.92;
 }
 
-static float color_to_linear_premult(float non_linear, float alpha) {
-	return (alpha == 0) ? 0 : color_to_linear(non_linear / alpha) * alpha;
-}
-
 static void mat3_to_mat4(const float mat3[9], float mat4[4][4]) {
 	memset(mat4, 0, sizeof(float) * 16);
 	mat4[0][0] = mat3[0];
@@ -586,11 +582,27 @@ static void render_pass_add_rect(struct wlr_render_pass *wlr_pass,
 	// space and expects in inputs in linear space since it outputs
 	// colors in linear space as well (and vulkan then automatically
 	// does the conversion for out sRGB render targets).
+	struct wlr_render_color srgb = options->color;
+
+	// Un-premultiply sRGB values
+	if (srgb.a != 0) {
+		srgb.r /= srgb.a;
+		srgb.g /= srgb.a;
+		srgb.b /= srgb.a;
+	}
+
+	// Estimate perceptual lightness from sRGB values
+	float v = (srgb.r + srgb.g + srgb.b) / 3;
+	// Adjust alpha to make dark semi-transparent overlays a bit more
+	// opaque, better matching perceptual intent compared to simple
+	// linear blending (see README-alpha-blend for a longer discussion)
+	float new_a = ((v - 0.5f) * srgb.a * srgb.a + srgb.a) / (v + 0.5f);
+
 	float linear_color[] = {
-		color_to_linear_premult(options->color.r, options->color.a),
+		color_to_linear(srgb.r) * new_a,
-		color_to_linear_premult(options->color.g, options->color.a),
+		color_to_linear(srgb.g) * new_a,
-		color_to_linear_premult(options->color.b, options->color.a),
+		color_to_linear(srgb.b) * new_a,
-		options->color.a, // no conversion for alpha
+		new_a,
 	};
 
 	pixman_region32_t clip;

render/vulkan/shaders/texture.frag

@@ -26,11 +26,17 @@ vec4 srgb_color_to_linear(vec4 color) {
 		return vec4(0);
 	}
 	color.rgb /= color.a;
+	// Estimate perceptual lightness from sRGB values
+	float v = (color.r + color.g + color.b) / 3;
 	color.rgb = vec3(
 		srgb_channel_to_linear(color.r),
 		srgb_channel_to_linear(color.g),
 		srgb_channel_to_linear(color.b)
 	);
+	// Adjust alpha to make dark semi-transparent overlays a bit more
+	// opaque, better matching perceptual intent compared to simple
+	// linear blending (see README-alpha-blend for a longer discussion)
+	color.a = ((v - 0.5) * color.a * color.a + color.a) / (v + 0.5);
 	color.rgb *= color.a;
 	return color;
 }