The fine channel may not display any recognisable pattern, it certainly wouldn't follow the sine wave.
Forget about DMX and binary for a moment and consider this...
Imagine a parameter, increasing linearly by 0.75 per step. It would go 0.75, 1.5, 2.25, 3.00, 3.75 and so on. The whole numbers are going up while the fractions are going down and yet it would still plot an ascending straight line on a graph. The whole numbers are analogous to the coarse byte, the fractions to the fine byte.
As @Erics says above, calculate a 16bit result then output the most significant byte as coarse, the least significant byte as fine.
Couple of other tips...
I may be wrong about this but I think some (most? all?) fixtures ignore some of the least significant bits of the least significant (fine) channel so, even if the fine output value changes by a small amount the fixture will not move.
Most programming languages use signed integers with negative numbers represented in "twos complement". You need unsigned integers so, unless your language has an unsigned integer data type you may have some work to do there. Maybe you should do your calculations in floating point and write functions to convert results to 8bit, 16bit MSB and 16bit LSB.
This will be easier if you use the right language to start with. Some, at least, of a finished project will need to run in real time and it should be capable of multi threading and interrupt handling. Its a bit like a swan, serene and beautiful to look at but paddling like anything under the surface. 😀
Full disclosure... Before retiring I was an analyst programmer for 40 years or so and, while I never attempted the project I have spent time thinking about it. Feel free to DM me, maybe I can help here & there.