a Tqe4@sddlZddlmZmZddlZGdddejZGdddeZ ddZ Gd d d eZ Gd ddeZ Gd d d e Z Gddde Z Gddde ZdS)N)SGD lr_schedulercs8eZdZd fdd ZddZddZd d d ZZS) _LRMomentumSchedulercsx|dkr&|jD]}|d|dqn*t|jD]\}}d|vr0td|q0ttdd|j|_t ||dS)Nrinitial_momentummomentumzXparam 'initial_momentum' is not specified in param_groups[{}] when resuming an optimizercSs|dS)Nr)grouprrI/home/opencvuniv/work/pranav/ADAS_2_LIDAR/SFA3D/sfa/utils/lr_scheduler.pyz/_LRMomentumScheduler.__init__..) param_groups setdefault enumerateKeyErrorformatlistmapbase_momentumssuper__init__)self optimizer last_epochr i __class__rr rs z_LRMomentumScheduler.__init__cCstdSNNotImplementedErrorrrrr get_lrsz_LRMomentumScheduler.get_lrcCstdSrrr rrr get_momentumsz!_LRMomentumScheduler.get_momentumNcCsP|dur|jd}||_t|jj||D]\}}}||d<||d<q0dS)Nlrr)rziprr r!r")repoch param_groupr$rrrr steps  "z_LRMomentumScheduler.step)r)N)__name__ __module__ __qualname__rr!r"r( __classcell__rrrr rs rc@s eZdZdZddZddZdS)ParameterUpdatea}A callable class used to define an arbitrary schedule defined by a list. This object is designed to be passed to the LambdaLR or LambdaScheduler scheduler to apply the given schedule. Arguments: params {list or numpy.array} -- List or numpy array defining parameter schedule. base_param {float} -- Parameter value used to initialize the optimizer. cCst|dg|_||_dS)Nrnphstackparams base_paramrr1r2rrr r,sParameterUpdate.__init__cCs|j||jSrr1r2rr&rrr __call__0sParameterUpdate.__call__Nr)r*r+__doc__rr7rrrr r-"s r-csfddt||DS)Ncsg|]\}}||qSrr).0lmbdabaserrr 5r z apply_lambda..)r%)rbasesZlambdasrr>r apply_lambda4srAcsPeZdZdZdddddffdd ZddZd d Zd d Zd dZZ S)LambdaScheduleraSets the learning rate and momentum of each parameter group to the initial lr and momentum times a given function. When last_epoch=-1, sets initial lr and momentum to the optimizer values. Args: optimizer (Optimizer): Wrapped optimizer. lr_lambda (function or list): A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups. Default: lambda x:x. momentum_lambda (function or list): As for lr_lambda but applied to momentum. Default: lambda x:x. last_epoch (int): The index of last epoch. Default: -1. Example: >>> # Assuming optimizer has two groups. >>> lr_lambda = [ ... lambda epoch: epoch // 30, ... lambda epoch: 0.95 ** epoch ... ] >>> mom_lambda = [ ... lambda epoch: max(0, (50 - epoch) // 50), ... lambda epoch: 0.99 ** epoch ... ] >>> scheduler = LambdaScheduler(optimizer, lr_lambda, mom_lambda) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() cCs|Srrxrrr r Vr zLambdaScheduler.rcs||_t|ttfs(|gt|j|_n6t|t|jkrTtdt|jt|t||_t|ttfs|gt|j|_ n6t|t|jkrtdt|jt|t||_ ||_ t ||dS)Nz"Expected {} lr_lambdas, but got {}z(Expected {} momentum_lambdas, but got {}) r isinstancertuplelenr lr_lambdas ValueErrorrmomentum_lambdasrrr)rr lr_lambdamomentum_lambdarrrr rVs"  zLambdaScheduler.__init__cCsdd|jD}dgt|j|d<dgt|j|d<tt|j|jD]H\}\}}t|tj sx|j |d|<t|tj sN|j |d|<qN|S)a>Returns the state of the scheduler as a :class:`dict`. It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate and momentum lambda functions will only be saved if they are callable objects and not if they are functions or lambdas. cSsi|]\}}|dvr||qS))rrHrJr)r;keyvaluerrr ss z.LambdaScheduler.state_dict..NrHrJ) __dict__itemsrGrHrJrr%rEtypes FunctionTypecopy)r state_dictidxZlr_fnZmom_fnrrr rUls  zLambdaScheduler.state_dictcCs||d}|d}|j|t|D]"\}}|dur(|j|j|q(t|D]"\}}|durT|j|j|qTdS)zLoads the schedulers state. Arguments: state_dict (dict): scheduler state. Should be an object returned from a call to :meth:`state_dict`. rHrJN)poprPupdaterrHrJ)rrUrHrJrVfnrrr load_state_dicts   zLambdaScheduler.load_state_dictcCst|j|j|jSr)rArbase_lrsrHr rrr r!szLambdaScheduler.get_lrcCst|j|j|jSr)rArrrJr rrr r"szLambdaScheduler.get_momentum) r)r*r+r:rrUrZr!r"r,rrrr rB8s rBc@s eZdZdZddZddZdS)r-aA callable class used to define an arbitrary schedule defined by a list. This object is designed to be passed to the LambdaLR or LambdaScheduler scheduler to apply the given schedule. If a base_param is zero, no updates are applied. Arguments: params {list or numpy.array} -- List or numpy array defining parameter schedule. base_param {float} -- Parameter value used to initialize the optimizer. cCs8t|dg|_||_|dkr4d|_|jdd|_dS)Nrg-q=r#gg?r.r3rrr rs r4cCs|j||jSrr5r6rrr r7sr8Nr9rrrr r-s cs"eZdZdZdfdd ZZS) ListSchedulera<Sets the learning rate and momentum of each parameter group to values defined by lists. When last_epoch=-1, sets initial lr and momentum to the optimizer values. One of both of lr and momentum schedules may be specified. Note that the parameters used to initialize the optimizer are overriden by those defined by this scheduler. Args: optimizer (Optimizer): Wrapped optimizer. lrs (list or numpy.ndarray): A list of learning rates, or a list of lists, one for each parameter group. One- or two-dimensional numpy arrays may also be passed. momentum (list or numpy.ndarray): A list of momentums, or a list of lists, one for each parameter group. One- or two-dimensional numpy arrays may also be passed. last_epoch (int): The index of last epoch. Default: -1. Example: >>> # Assuming optimizer has two groups. >>> lrs = [ ... np.linspace(0.01, 0.1, 100), ... np.logspace(-2, 0, 100) ... ] >>> momentums = [ ... np.linspace(0.85, 0.95, 100), ... np.linspace(0.8, 0.99, 100) ... ] >>> scheduler = ListScheduler(optimizer, lrs, momentums) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() Nrcs|j}durdd}nRtttfr0tntjdkrVfdd|D}nddt|D}dur|dd}nRtttfrtntjdkrfdd|D}nd dt|D}t |||dS) NcSs|SrrrCrrr r r z(ListScheduler.__init__..r#csg|]}t|dqSr$r-r;g)lrsrr r?r z*ListScheduler.__init__..cSsg|]\}}t||dqSr]r^r;lr`rrr r?r cSs|SrrrCrrr r r csg|]}t|dqSrr^r_) momentumsrr r?r cSsg|]\}}t||dqSrdr^rbrrr r?r ) r rErrFr/arrayrGshaper%rr)rrrarergroupsrKrLr)rarer rs  zListScheduler.__init__)NNrr)r*r+r:rr,rrrr r\sr\cs eZdZdZfddZZS) RangeFindera9Scheduler class that implements the LR range search specified in: A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay. Leslie N. Smith, 2018, arXiv:1803.09820. Logarithmically spaced learning rates from 1e-7 to 1 are searched. The number of increments in that range is determined by 'epochs'. Note that the parameters used to initialize the optimizer are overriden by those defined by this scheduler. Args: optimizer (Optimizer): Wrapped optimizer. epochs (int): Number of epochs over which to run test. Example: >>> scheduler = RangeFinder(optimizer, 100) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() cs tdd|}t||dS)Nir)r/logspacerr)rrepochsrarrr rszRangeFinder.__init__rirrrr rjsrjcs*eZdZdZddgdffdd ZZS)OneCyclePolicyaScheduler class that implements the 1cycle policy search specified in: A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay. Leslie N. Smith, 2018, arXiv:1803.09820. Args: optimizer (Optimizer): Wrapped optimizer. lr (float or list). Maximum learning rate in range. If a list of values is passed, they should correspond to parameter groups. epochs (int): The number of epochs to use during search. momentum_rng (list). Optional upper and lower momentum values (may be both equal). Set to None to run without momentum. Default: [0.85, 0.95]. If a list of lists is passed, they should correspond to parameter groups. phase_ratio (float): Fraction of epochs used for the increasing and decreasing phase of the schedule. For example, if phase_ratio=0.45 and epochs=100, the learning rate will increase from lr/10 to lr over 45 epochs, then decrease back to lr/10 over 45 epochs, then decrease to lr/100 over the remaining 10 epochs. Default: 0.45. g333333?gffffff?g?c sZt|t|ttfr0fdd|D}nFtt|d|t||dt|d|ddg}|durBt|}t|j dkrt |j D]\}}||d|d<qfdd|D} nnt |j D]\}}|d|d<qtt|d|d t|d |dt|d|ddg} nd} t ||| dS) Nc sRg|]J}tt|d|t||dt|d|ddgqS)皙?{Gz?r/r0linspace)r;rcrlZ phase_epochsrr r?s z+OneCyclePolicy.__init__..rnrorpr#rc sZg|]R}tt|d|dt|d|dt|d|ddgqS)r#rrprq)r;mrsrr r?&s r)intrErrFr/r0rrrfrGrgrr rr) rrr$rl momentum_rng phase_ratiorarr`rerrsr rs6     zOneCyclePolicy.__init__rirrrr rmsrm)torchZ torch.optimrrnumpyr/ _LRSchedulerrobjectr-rArBr\rjrmrrrr sa4