[Algorithms] Normal map compression and fitting a circle to a set of 3d points

[Doug R. <DR...@nv...>]

I am exploring an implementation for compressing normal maps using DXT/S3TC.
 
Given
   hardware decode 3 or 4 palette entries for each 16 pixel block.  
 
Assumptions:
   pixel shader will renormalize texels
 
so, hardware reconstructs an arc on a circle during decompression (because
the texels are renormalized)
 
So I wanted an algorithm that does a fit of a circle centered on the origin
and radius = 1 to a set of 3D points on a sphere (normals).  Any ideas how
to do this?
 
After that, I could choose the palette endpoints (the most extreme normals
on the arc) and find the closest palette entry for each normal.
 
 
-Doug