o =e!@sddlmZddlZddlmZmZddlmZddlmZddl m Z da Gdddej Z Gd d d ej ZGd d d ej Zd dZdS))OptionalN)nnTensor) checkpoint)MultiheadAttentionRelative) get_clonescsneZdZdZddededeffdd Zd ejd ejd ed efddZddejdejd e efddZ Z S) TransformerzS Transformer computes self (intra image) and cross (inter image) attention  hidden_dimnheadnum_attn_layerscsXtt||}t|||_t||}t|||_t||_ ||_ ||_ ||_ dSN) super__init__TransformerSelfAttnLayerrself_attn_layersTransformerCrossAttnLayercross_attn_layersr LayerNormnormr r r)selfr r rZself_attn_layerZcross_attn_layer __class__a/home/opencvuniv/Work/Pranav/stereo-transformer/repo-new/stereo-transformer/module/transformer.pyrs       zTransformer.__init__featpos_enc pos_indexeshnc Cstt|j|jD]A\}\}}|add}t|||||}||jdkr*dd} ndd} t| ||ddd|f|dd|df||\}} q da| S) az Alternate self and cross attention with gradient checkpointing to save memory :param feat: image feature concatenated from left and right, [W,2HN,C] :param pos_enc: positional encoding, [W,HN,C] :param pos_indexes: indexes to slice positional encoding, [W,HN,C] :param hn: size of HN :return: attention weight [N,H,W,W] cfdd}|S)Ncs|Srrinputsmodulerrcustom_self_attn7szXTransformer._alternating_attn..create_custom_self_attn..custom_self_attnr)r%r&rr$rcreate_custom_self_attn6 z>Transformer._alternating_attn..create_custom_self_attncr!)Ncg|dRS)NTrr"r$rrcustom_cross_attnBZTransformer._alternating_attn..create_custom_cross_attn..custom_cross_attnrr%r+rr$rcreate_custom_cross_attnAr(z?Transformer._alternating_attn..create_custom_cross_attncr!)Ncr*)NFrr"r$rrr+Ir,r-rr.rr$rr/Hr(Nr) enumerateziprr layer_idxrr) rrrrr idx self_attn cross_attnr'r/ attn_weightrrr_alternating_attn%s  . zTransformer._alternating_attnN feat_left feat_rightc Cs|j\}}}}|dddddddd}|dddddddd}|durkt3t|dd||d|j}td|d|d||j} || d } Wdn1sewYnd} tj ||gdd} | | || |} | ||||dddd} | S)a7 :param feat_left: feature descriptor of left image, [N,C,H,W] :param feat_right: feature descriptor of right image, [N,C,H,W] :param pos_enc: relative positional encoding, [N,C,H,2W-1] :return: cross attention values [N,H,W,W], dim=2 is left image, dim=3 is right image r)rNdim) shapepermuteflattentorchno_gradlinspaceviewtodevicelongcatr7) rr8r9rbscr wZ indexes_rZ indexes_crrr6rrrforwardTs   ""zTransformer.forward)r r r r) __name__ __module__ __qualname____doc__intrrBrr7rrM __classcell__rrrrrs (/rcsNeZdZdZdedeffdd Z  d dedeed eefd d ZZ S) rz Self attention layer r r cs&tt|||_t||_dSr)rrrr4rrnorm1rr r rrrrys  z!TransformerSelfAttnLayer.__init__NrposrcCs0||}|j|||||d\}}}||}|S)z :param feat: image feature [W,2HN,C] :param pos: pos encoding [2W-1,HN,C] :param pos_indexes: indexes to slice pos encoding [W,W] :return: updated image feature querykeyvaluerr)rTr4)rrrVrZfeat2r6_rrrrMs  z TransformerSelfAttnLayer.forward)NN) rNrOrPrQrRrrrrMrSrrrrrtsrc sreZdZdZdedeffdd Z   dded ed eed eed eef d dZ e defddZ Z S)rz Cross attention layer r r cs2tt|||_t||_t||_dSr)rrrr5rrrTnorm2rUrrrrs   z"TransformerCrossAttnLayer.__init__NFr8r9rVr last_layercCs||}||}|durt|dg}n|}|j|||||dd}||}|r9|d} || |j} nd} ||}|j|||| ||d\}} } ||}tj ||gdd} | | fS)ax :param feat_left: left image feature, [W,HN,C] :param feat_right: right image feature, [W,HN,C] :param pos: pos encoding, [2W-1,HN,C] :param pos_indexes: indexes to slicer pos encoding [W,W] :param last_layer: Boolean indicating if the current layer is the last layer :return: update image feature and attention weight NrrW)rXrYrZ attn_maskrrr)r=) rTrBflipr5size _generate_square_subsequent_maskrFrGr\rI)rr8r9rVrr]Z feat_left_2Z feat_right_2Z pos_flippedrLr^r6Zraw_attnrrrrrMs.      z!TransformerCrossAttnLayer.forwardszcCs*tjt||dd}td||dk<|S)z Generate a mask which is upper triangular :param sz: square matrix size :return: diagonal binary mask [sz,sz] r))diagonalz-inf)rBtriuonesfloat)rrbmaskrrrrasz:TransformerCrossAttnLayer._generate_square_subsequent_mask)NNF)rNrOrPrQrRrrrboolrMrBrCrarSrrrrrs  2rcCst|j|j|jdS)N)r r r)r channel_dimZnheadsr)argsrrrbuild_transformers rk)typingrrBrrtorch.utils.checkpointrZmodule.attentionrutilities.miscrr2Modulerrrrkrrrrs    c" K