OKAY SO I THINK I GOT IT FIGURED OUT.

For the purposes of this, "a" means "armor points", "h" means "health points", and "p" means "armor protection ratio."

Let function P(h, a, p) = {0 when p <= 0, a when p >= 0, min(a, floor((h / (1 - p)) - h) when 0 < p < 1}.

Let function T(h, a, p) = h + P(h, a, p).

Examples:

Full health, fresh suit of green armor: P(100, 100, 0.3335) -> 50 effective health points from the armor, and T(100, 100, 0.3335) -> 150 effective health.

Full health, Quake red armor: P(100, 200, 0.8) -> 200 effective health points from the armor, and T(100, 200, 0.8) -> 300 effective health.

55 health, 65 blue armor: P(55, 65, 0.5) -> 55 effective health from the armor, and T(55, 65, 0.3335) -> 110 effective health.

55 health, 100 green armor: P(55, 100, 0.3335) -> 27 effective health from the armor, and T(55, 65, 0.3335) -> 82 effective health.

How I got to what I did is pretty simple. "1 - p" represents the proportion of the damage the health takes when wearing an infinite amount of the armor given. We divide the health by that to get how much armor you would need to fully cover your current health, and floor it since health is always integral. Then we subtract out the health to get how much health the armor saves you, assuming an infinite amount.

Now, you don't have an infinite amount of armor, hence the min(a, <shit>). The armor can't save more health than it has in armor points, so that handles that

Protection values at and below 0 don't protect at all, so we return 0 there. Protection values at and above 1 is basically "armor as health", so we return how much armor you have.

And for total health, we just add the health back in once we adjust for armor points.

THE UPSHOT:

Just choose the armor that gets you the highest (p * a). It's easy to calculate, and will get you the right answer regardless.