2VhK[ddlZddlmZddlmZddlmZddlmZddlmZddlmZddlmZdd l m Z dd l m Z dd l mZe d Gd deZy)N) activations) constraints)dtype_policies) initializers)ops) quantizers) regularizers) keras_export) InputSpec)Layerzkeras.layers.DenseceZdZdZ dfd ZdZedZddZdZ ddZ dZ d Z fd Z d Zd Zd ZdZddZddZddZdZxZS)Densea Just your regular densely-connected NN layer. `Dense` implements the operation: `output = activation(dot(input, kernel) + bias)` where `activation` is the element-wise activation function passed as the `activation` argument, `kernel` is a weights matrix created by the layer, and `bias` is a bias vector created by the layer (only applicable if `use_bias` is `True`). Note: If the input to the layer has a rank greater than 2, `Dense` computes the dot product between the `inputs` and the `kernel` along the last axis of the `inputs` and axis 0 of the `kernel` (using `tf.tensordot`). For example, if input has dimensions `(batch_size, d0, d1)`, then we create a `kernel` with shape `(d1, units)`, and the `kernel` operates along axis 2 of the `input`, on every sub-tensor of shape `(1, 1, d1)` (there are `batch_size * d0` such sub-tensors). The output in this case will have shape `(batch_size, d0, units)`. Args: units: Positive integer, dimensionality of the output space. activation: Activation function to use. If you don't specify anything, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix. bias_initializer: Initializer for the bias vector. kernel_regularizer: Regularizer function applied to the `kernel` weights matrix. bias_regularizer: Regularizer function applied to the bias vector. activity_regularizer: Regularizer function applied to the output of the layer (its "activation"). kernel_constraint: Constraint function applied to the `kernel` weights matrix. bias_constraint: Constraint function applied to the bias vector. lora_rank: Optional integer. If set, the layer's forward pass will implement LoRA (Low-Rank Adaptation) with the provided rank. LoRA sets the layer's kernel to non-trainable and replaces it with a delta over the original kernel, obtained via multiplying two lower-rank trainable matrices. This can be useful to reduce the computation cost of fine-tuning large dense layers. You can also enable LoRA on an existing `Dense` layer by calling `layer.enable_lora(rank)`. lora_alpha: Optional integer. If set, this parameter scales the low-rank adaptation delta (computed as the product of two lower-rank trainable matrices) during the forward pass. The delta is scaled by `lora_alpha / lora_rank`, allowing you to fine-tune the strength of the LoRA adjustment independently of `lora_rank`. Input shape: N-D tensor with shape: `(batch_size, ..., input_dim)`. The most common situation would be a 2D input with shape `(batch_size, input_dim)`. Output shape: N-D tensor with shape: `(batch_size, ..., units)`. For instance, for a 2D input with shape `(batch_size, input_dim)`, the output would have shape `(batch_size, units)`. c  t|dd|i| ||_tj||_||_tj||_tj||_ tj||_ tj||_ tj| |_tj| |_| |_| | n| |_d|_t'd|_d|_y)Nactivity_regularizerF)min_ndimT)super__init__unitsrget activationuse_biasrkernel_initializerbias_initializerr kernel_regularizerbias_regularizerrkernel_constraintbias_constraint lora_rank lora_alpha lora_enabledr input_specsupports_masking)selfrrrrrrrrrrr r!kwargs __class__s K/home/dcms/DCMS/lib/python3.12/site-packages/keras/src/layers/core/dense.pyrzDense.__init__Ms M.BMfM %//*5  "."2"23E"F , 0 01A B"."2"23E"F , 0 01A B!,1B!C*?"(2(>*I!#Q/ $cD|d|jf}|jr|j||j|jdk7r9|jd||j|j |j |_|jrE|jd|jf|j|j|j|_ nd|_ tdd|di|_d |_|j r|j#|j yy) N)modeint8kernel)nameshape initializer regularizer constraintbiasr)raxesT)rquantization_modequantized_build add_weightrrr_kernelrrrrr4r r#builtr enable_lora)r% input_shape kernel_shapes r(buildz Dense.buildms#B4  ! !  D4J4J K  ! !V + ??" 33 3311 +DL ==zzm 11 11// (DIDI#Qb+b/5JK >>   T^^ , r)c|js td|jrQ|j|j|j z t j|j|jzzS|jS)Nz3You must build the layer before accessing `kernel`.) r:AttributeErrorr"r9r!r rmatmul lora_kernel_a lora_kernel_b)r%s r(r.z Dense.kernelsszz E    <<$..0 4--t/A/AB#CC C||r)ctj||j}|j tj||j}|j |j |}|SN)rrAr.r4addr)r%inputstrainingxs r(callz Dense.callsQ JJvt{{ + 99 499%A ?? &"Ar)cLt|}|j|d<t|S)Nr+)listrtuple)r%r< output_shapes r(compute_output_shapezDense.compute_output_shapes%K( :: R\""r)c|jr td|js td|jr td|jj |j d|jjd|ftj||j|_ |j d||jjdftj||j|_ d |j_|jj!d |_||_|||_y||_y) Nz}Lora is incompatible with kernel constraints. In order to enable lora on this layer, remove the `kernel_constraint` argument.z3Cannot enable lora on a layer that isn't yet built.z>lora is already enabled. This can only be done once per layer.rBr)r/r0r1r2rCFT)r ValueErrorr:r"_trackerunlockr8r.r0rrrrBrCr9 trainablelockr r!)r%rankr! a_initializer b_initializers r(r;zDense.enable_loras4  ! !0  zzE    P  !__ ;;$$Q'.$((7// -  "__ **1-.$((7// -  "'   (2(>*Dr)c|jsy|j\}}|g}|jr|j|j|j |j dk(r|j|n|j dk(r|j|j |j|j|j|j|j|j|j|j|j|jn|j|j t|D]\}}||t|<yNr-float8)r:_get_kernel_with_merged_lorarappendr4r6 inputs_scaleinputs_amax_history kernel_scalekernel_amax_historyoutputs_grad_scaleoutputs_grad_amax_history_quantization_mode_error enumeratestr)r%store kernel_valueratarget_variablesivariables r(save_own_variableszDense.save_own_variabless-zz &*%F%F%H" l(> ==  # #DII .  ! ! -%%/ '' 5''83 ''(9(9: ''(@(@A ''(9(9: ''(@(@A ''(?(?@ ''(F(FG33D4J4JKK$%56 %KAx$E#a&M %r)cJ|js|j||jsy|jg}|jr|j |j |j|jdk(r|j |jn|jdk(r|j |j|j |j|j |j|j |j|j |j|j |jn|j|jt|D]"\}}|j!|t#|$|jr|j$j!t'j(|j$j*|j,j!t'j(|j,j*yyr[)r"_check_load_own_variablesr:r9rr^r4r6rar_r`rbrcrdrerfassignrgrBrzerosr0rC)r%rhrjrkrls r(load_own_variableszDense.load_own_variabless   * *5 1zz !LL> ==  # #DII .  ! ! -%%/ ''(9(9:''83 ''(9(9: ''(@(@A ''(9(9: ''(@(@A ''(?(?@ ''(F(FG33D4J4JKK$%56 +KAx OOE#a&M * +       % %cii0B0B0H0H&I J    % %cii0B0B0H0H&I J r)c Vt|}|jtj|j |j tj|jtj|jtj|jtj|jtj|jtj|jd }|j r|j |d<|j"|d<i||S)N) rrrrrrrrrr r!)r get_configrr serializerrrrrr rrrrrr r!)r% base_configconfigr's r(rtzDense.get_configsg(* ZZ%//@ "."8"8''#!- 6 6t7L7L M"."8"8''#!- 6 6t7L7L M!,!6!6t7M7M!N*44T5I5IJ  >>"&..F; #'??F< (+(((r)c  |j|jz}t|jt|k7rt|dk(rY|jsMt d|j dt|jd|j d|j d t d|j dt|dt|jd |Dcgc]}|j c}ycc}w) NrzLayer 'zY' was never built and thus it doesn't have any variables. However the weights file lists z variables for this layer. In most cases, this error indicates that either: 1. The layer is owned by a parent layer that implements a `build()` method, but calling the parent's `build()` method did NOT create the state of the child layer 'z'. A `build()` method must create ALL state for the layer, including the state of any children layers. 2. You need to implement the `def build_from_config(self, config)` method on layer 'a-', to specify how to rebuild it during loading. In this case, you might also want to implement the method that generates the build config at saving time, `def get_build_config(self)`. The method `build_from_config()` is meant to create the state of the layer (i.e. its variables) upon deserialization.z ' expected z variables, but received z% variables during loading. Expected: )_trainable_variables_non_trainable_variableslenkeysr:rRr/)r%rhall_varsvs r(rozDense._check_load_own_variabless,,t/L/LL uzz| H -8}!$** dii[)669%**,6G5HI( )- {3! "& ,NN.$))KH ? uzz|$%&.67aff78: 3 .:8s#Dc|dk(r|j|d|_y|dk(r|jd|_y|j|)Nr-r\T) _int8_build _float8_buildre _is_quantized)r%r=r,s r(r7zDense.quantized_build;sT 6>   \ * " X     "//5 5r)ctjd|_|jd|ddd|_|jd|j fd d |_y) Nr+axisr.rqr-F)r/r0r1dtyperUraones)r/r0r1rU)rAbsMaxQuantizerinputs_quantizerr8r9rra)r%r=s r(rzDense._int8_buildDsc * : : C '  !OO::- , r)cddlm}t|jd|j}ddddddd }|fd ddddd }|j dd d i||_|j dd d i||_|j dd di||_|j dd di||_ |j dd di||_ |j dd di||_ y)Nr)QuantizedFloat8DTypePolicyamax_history_lengthrrfloat32TF)r0r1rrUautocastoverwrite_with_gradientrqr/r_r`rarbrcrd) keras.src.dtype_policiesrgetattr dtype_policydefault_amax_history_lengthr8r_r`rarbrcrd)r%rr scale_kwargsamax_history_kwargss r(rzDense._float8_buildTs+G&    ! & B B !'+  *+"'+  ,DOOPP<P#24??$ &$ *=$  ,DOOPP<P#24??$ &$ *=$  #2$//# %# )5# *9* ,* 0C* &r)cJtjfd}||tjjtjj}j rptj |j}tj |j}tj|jjz |z}j tj|j}jj|}|S)Nc ddfd }j|\}}tj|}tj|j}tj |tj |}||fS)N)upstreamc||\}tjtjj}tj|tj |}|ddfS)N)r)rdividecast compute_dtyperA transpose)rargs float_kernel inputs_gradr.rar%s r(grad_fnzFDense._int8_call..matmul_with_inputs_gradient..grad_fns]#"&KX"zzHHV4+=+=>   "jj3==3NO #T400r))rrrArrrmultiply)rGr.rarr_rIr%s `` r(matmul_with_inputs_gradientz5Dense._int8_call..matmul_with_inputs_gradientsr(, 1 1$(#8#8#@ FL 66*AD../A 1cll<FGAg: r))rcustom_gradientconvert_to_tensorr9rar"rArBrCrFr!r r4r)r%rGrHrrIlora_xs` r( _int8_callzDense._int8_calls     $ (   ! !$,, /  ! !$"3"3 4    ZZ(:(:;FZZ(:(:;FDOOdnn<FGA 99 499%A ?? &"Ar)c jr tdtjfd}tjfd}tj||tj j tj j|tj jtj jtj j}||tj jtj j}jqj}jjdk(r6tj |d}tj ||j"}tj$||}j&j'|}|S)Nz.Currently, `_float8_call` doesn't support LoRAc nr}tjtj|d|tjt t jdjdtj||nddtj||dj}dddfd }||fS)Nrr float8_e4m3fnrr variablesc||\}|fSrEr)rrrnew_amax_history new_scales r(gradzEDense._float8_call..quantized_dequantize_inputs..grads#"&KX,<<.quantized_dequantize_inputss&;;GGLq1HHiooo>BBCY $.#I#IL$ ! #' #;;0B0BJ&*T = t# #r)c&dddfd }||fS)z;Quantize-dequantize the output gradient but not the output.Nrc V||\}tjtjdtjt t jdjd}tj|dj}tj|}|||fS)Nrr float8_e5m2r) rrrrrrrrrrr) rrrr qdq_upstreamrrrr%s r(rzFDense._float8_call..quantized_dequantize_outputs..grads#"&KX&;;GGLq1HHioom<@@A9  *AAe]D4F4F  $.#I#Il$ $Y0@@@r)r)outputsrrrr%s `` r(quantized_dequantize_outputsz8Dense._float8_call..quantized_dequantize_outputss"&*T A A$D= r)rbfloat16)r"NotImplementedErrorrrrArr_r`r9rarbrcrdr4rrrrrFr)r%rGrHrrrIr4 bias_bf16s` ` r( _float8_callzDense._float8_calls   %@    $  $4   !  !. JJ '%%d&7&78%%d&>&>?  (%%dll3%%d&7&78%%d&>&>?   )  ! !$"9"9 :  ! !$"@"@ A  99 99D  ..);HHT:6 xx 4::64 A ?? &"Ar)cD|r,t|tur|j|j|jj }|dk(r>t j|jdd\}}tj|d}|`|j|||dk(r6|jj|jj|jj6tj |d|jj"}||_ yy)Nr-rTrto_numpyr_from_)typer_not_implemented_errorquantizer9r0rabs_max_quantizersqueezer7rprarr6rrr/)r%r, type_checkr=rirapolicys r(rzDense.quantizes 4:U2--dmm< <||)) 6>)3)D)D 1t* &L,;;|! LL   -    $ $\ 2    . . 6#''4&t7H7H7M7M6N(OPF &D  7r)c|jj|j}|j}|jrt j ||}t j||j|jz t j|j|jz}tj|dd\}}t j|d}||fS|j dfS)NrTrr)rr6r9rar"rrrFr!r rArBrCrrrr.)r%riras r(r]z"Dense._get_kernel_with_merged_loras    . . :<propertyr.rJrOr;rmrrrtror7rrrrrr] __classcell__)r's@r(rrs:~+ !%@-:  #" &IP%2K:), H" ) V!FYv',!r)r)r keras.srcrrrrrrr keras.src.api_exportr keras.src.layers.input_specr keras.src.layers.layerr rrr)r(rsJ!!$" "-1("#U!EU!$U!r)