I think you're operating on gamma-modified data
when you should be operating on linear-light data.
Consider a background of 0x40 grey (sRBG color space),
with an object of 0xc0 grey, and an alpha of 50%. The
result should not be 0x80 grey; that would be too dim.
To avoid round-trip losses, 16-bit intermediate channels
may be required.
The above is based on seeing significant dark fringes
around an image that shouldn't have more than a tiny
bit, if any.
(Ref: Private email to Bill K. from Albert C., 2004-10-25)