I really hope some SQL guru out there can assist with this one (and my apologies if this has been answered before. I did try and find a similar post but to no avail):
declare @theanswer numeric(38,16)
select @theanswer = 0.01 / 0.0074464347
select @theanswer
The above results in 1.3429245542165000
but the following (which looks the same to me)
declare @val1 numeric(38,16);
declare @val2 numeric(38,16);
set @val1 = 0.01;
set @val2 = 0.0074464347;
select @val1/@val2
results with 1.342924 and truncates it?
Any ideas?
To get the real precision and scale for the result (@val1/@val2) I would execute this T-SQL
script
DECLARE @val1 NUMERIC(38,16);
DECLARE @val2 NUMERIC(38,16);
SET @val1 = 0.01;
SET @val2 = 0.0074464347;
SELECT @val1/@val2
,SQL_VARIANT_PROPERTY(@val1/@val2, 'BaseType') [BaseType]
,SQL_VARIANT_PROPERTY(@val1/@val2, 'Precision') [Precision]
,SQL_VARIANT_PROPERTY(@val1/@val2, 'Scale') [Scale]
And the results will be:
(No column name) BaseType Precision Scale
1.342924 numeric 38 6
So, the result precision is 38 and result scale is 6.
MSDN has some details regarding precision and scale for arithmetic operations. These information can be found here: Precision, Scale, and Length (Transact-SQL):
Operation = e1 / e2
Result precision = p1 - s1 + s2 + max(6, s1 + p2 + 1)
Result scale * = max(6, s1 + p2 + 1)
Using these formulas we can write the next T-SQL
script to get theoretical precision and scale for the result (@val1/@val2) :
SELECT @p1 = 38 --or CONVERT(INT, SQL_VARIANT_PROPERTY(@val1, 'Precision'))
,@s1 = 16 --or CONVERT(INT, SQL_VARIANT_PROPERTY(@val1, 'Scale'))
,@p2 = 38 --or CONVERT(INT, SQL_VARIANT_PROPERTY(@val2, 'Precision'))
,@s2 = 16 --or CONVERT(INT, SQL_VARIANT_PROPERTY(@val2, 'Scale'));
--SELECT @p1 [@p1], @s1 [@s1], @p2 [@p2], @s2 [@s2];
SELECT @p_result = @p1 - @s1 + @s2 + CASE
WHEN 6 >= @s1 + @p2 + 1 THEN 6
WHEN 6 < @s1 + @p2 + 1 THEN @s1 + @p2 + 1
END
,@s_result = CASE
WHEN 6 >= @s1 + @p2 + 1 THEN 6
WHEN 6 < @s1 + @p2 + 1 THEN @s1 + @p2 + 1
END;
SELECT @p_result [@p_result], @s_result [@s_result];
The result is:
@p_result @s_result
93 55
So, for this arithmetic operation (@val1/@val2), in theory
the precision and scale are 93
and 55
but the real precision and scale are 38
and 6
.
The real precision is 38 because "The result precision and scale have an absolute maximum of 38".
Regarding the real result scale (6) MSDN it's not clear: "When a result precision is greater than 38, the corresponding scale is reduced to prevent the integral part of a result from being truncated".
To see how "the corresponding scale is reduced" I executed the above tests (script 1 for real precision and scale and script 2 for theoretical precision and scale) using NUMERIC
values having the same scale (16) but different scales (from 16 to 38). The results of these tests are:
/*
Result prec. Result scale (T=theoretical value, R=real value)
T-R T-R --@val1 and @val2 data type
49-38 33-22 --NUMERIC(16, 16)
51-38 34-21 --NUMERIC(17, 16)
53-38 35-20 --NUMERIC(18, 16)
55-38 36-19 --NUMERIC(19, 16)
...
61-38 39-16 --NUMERIC(22, 16) -- <-- data type for [real] result scale 16
...
77-38 47-8 --NUMERIC(30, 16)
79-38 48-7 --NUMERIC(31, 16)
81-38 49-6 --NUMERIC(32, 16)
83-38 50-6 --NUMERIC(33, 16)
85-38 51-6 --NUMERIC(34, 16)
...
93-38 55-6 --NUMERIC(38, 16)
*/
Examining these results:
1.I see an arithmetic progression for the real result scale: from 22 to 6, step -1.
2.Also, if the scale for @val1 and @val2 is constant (NUMERIC(...,16)
) then an inverse correlation exists between @val1 & @val2 precision (from 16 to 32) and the [real] result scale (from 16 to 6).
3.If @val1 and @val2 precision is 32 or higher (NUMERIC(32->38,16)
) the the [real] result scale is always 6 => this is your case.
4.If a greater [real] result scale is needed (over 6) you need to use a lower precision for @val1
and @val2: NUMERIC(22, 16)
:
SELECT
CONVERT(NUMERIC(22,16),@val1) / CONVERT(NUMERIC(22,16),@val2) [CONVERT(NUMERIC(22,16)]
,SQL_VARIANT_PROPERTY( CONVERT(NUMERIC(22,16),@val1) / CONVERT(NUMERIC(22,16),@val2) , 'BaseType') [BaseType]
,SQL_VARIANT_PROPERTY( CONVERT(NUMERIC(22,16),@val1) / CONVERT(NUMERIC(22,16),@val2) , 'Precision') [Precision]
,SQL_VARIANT_PROPERTY( CONVERT(NUMERIC(22,16),@val1) / CONVERT(NUMERIC(22,16),@val2) , 'Scale') [Scale]
CONVERT(NUMERIC(22,16) BaseType Precision Scale
---------------------- -------- --------- -----
1.3429245542165299 numeric 38 16