convert 2 8 bit dmx values to one 16 bit dmx value?

Are you using an existing personality or have you made your own?

Looking at the EOS personality for Skypanel S120 C Mode 14 (16 bit RGBW), it has two 8 bit channels for each color (Red is Red coarse, Red2 is Red fine). So using this personality, you can only enter the 2 8 bit values for each color- there is no way to enter a 16-bit value.

If you use the personality for Mode 9, then it allows a 16-bit value for each RGBW parameter.

or if I have 163/255 high and 138/255 low how do I make it x/65535?

That would be (163*256)+138 = 41866

and just to close the circle, to enter the DMX values for each parameter, you use @//

Example:
Channel 1 Red @// 41866

After you get all of the DMX values entered for each color, I would suggest recording them as a color palette so you don't have to enter them all each time you want that color.

Correct, if the personality expects a 16 bit value, you need enter the 16 bit number- there's no way to break it in to 2 8-bit values.

If you are not too concerned with the smoothness of color fades, then it might be easier to use a personality that expects 2 8-bit values, like Mode 9.

Best not to use any of the C/F profiles on a 16Bit console. (11 through 15).  Use 16Bit modes instead.  the C/F modes were meant to control separate channels on older 8Bit consoles.  Coarse 0 to 255, then fine tune with the Fine channel. 16Bit consoles use a "wrap arround" process: Coarse 1 plus Fine 0>255, followed by Coarse 2 plus Fine 0>255, etc.  Running these modes on a 16bit console can produce unexpected results.

Best not to use any of the C/F profiles on a 16Bit console. (11 through 15).  Use 16Bit modes instead.  the C/F modes were meant to control separate channels on older 8Bit consoles.  Coarse 0 to 255, then fine tune with the Fine channel. 16Bit consoles use a "wrap arround" process: Coarse 1 plus Fine 0>255, followed by Coarse 2 plus Fine 0>255, etc.  Running these modes on a 16bit console can produce unexpected results.