Changeset 3286


Ignore:
Timestamp:
Dec 3, 2010 10:56:26 PM (10 years ago)
Author:
wehart
Message:

Merged revisions 3187-3285 via svnmerge from
https://software.sandia.gov/svn/public/coopr/coopr.pysp/trunk

........

r3201 | jwatson | 2010-10-29 13:18:17 -0600 (Fri, 29 Oct 2010) | 3 lines


Inverting order of .dat files in PySP when loading from a node representation - now root-to-leaf, instead of leaf-to-root. This allows for deeper-in-the-tree nodes to over-write parameter values defined higher in the tree, which is a more "expected" behavior than the converse. The real answer is to throw an exception if a parameter is re-defined, but we're not there yet.

........

r3217 | jwatson | 2010-11-05 11:29:42 -0600 (Fri, 05 Nov 2010) | 3 lines


Various updates to support heteogeneous index sets in PH for different nodes in the scenario tree - more work / testing remains.

........

r3218 | jwatson | 2010-11-05 12:35:51 -0600 (Fri, 05 Nov 2010) | 3 lines


More changes associated with generalizing the PySP index structures from per-stage to per-node.

........

r3220 | jwatson | 2010-11-05 20:28:29 -0600 (Fri, 05 Nov 2010) | 3 lines


Various fixes to the WW PH extension, bringing it in compliance to previous commit changes.

........

r3221 | jwatson | 2010-11-05 20:43:40 -0600 (Fri, 05 Nov 2010) | 1 line


Removing some older PySP test problems from the repository

........

r3222 | jwatson | 2010-11-05 20:55:15 -0600 (Fri, 05 Nov 2010) | 1 line


Moving PySP forestry example to local sandbox, to streamline distribution.

........

r3261 | jwatson | 2010-11-29 15:26:46 -0700 (Mon, 29 Nov 2010) | 9 lines


Adding two options to the runef and runph pysp scripts, to facilitate scenario downsampling - the case where you have a big tree, but you don't want to use it all.


The options are:
--scenario-tree-downsample-fraction=X
--scenario-tree-random-seed


The options are fairly self-explanatory - the only possible nuance is that the downsample fraction is the fraction of scenarios retained.

........

r3263 | jwatson | 2010-12-01 10:47:21 -0700 (Wed, 01 Dec 2010) | 3 lines


Corrected issue with cvar generation introduced a while back.

........

r3264 | jwatson | 2010-12-01 11:16:01 -0700 (Wed, 01 Dec 2010) | 3 lines


Adding PySP extensive form tests.

........

r3265 | wehart | 2010-12-01 13:19:56 -0700 (Wed, 01 Dec 2010) | 2 lines


Auxmenting the filter

........

r3266 | wehart | 2010-12-01 13:58:59 -0700 (Wed, 01 Dec 2010) | 2 lines


Adding further diagnostics to the filter.

........

r3267 | wehart | 2010-12-01 14:12:15 -0700 (Wed, 01 Dec 2010) | 2 lines


Another attempt to fix this filter...

........

r3271 | wehart | 2010-12-01 15:51:23 -0700 (Wed, 01 Dec 2010) | 4 lines


Simplifying filter.


The error was really a Python 2.5 portability issue in EF. :(

........

r3275 | jwatson | 2010-12-02 19:35:52 -0700 (Thu, 02 Dec 2010) | 3 lines


Fixing Python 2.5-related issue with string.translate in phutils.py - using None as the translation table is not allowed in Python 2.5.

........

Location:
coopr.pysp/stable
Files:
4 deleted
11 edited
2 copied

Legend:

Unmodified
Added
Removed
  • coopr.pysp/stable

  • coopr.pysp/stable/coopr/pysp/convergence.py

    r3115 r3286  
    117117         if stage != scenario_tree._stages[-1]:
    118118
    119             for (reference_variable, index_template, reference_variable_indices) in stage._variables:
     119            for (reference_variable, index_template) in stage._variables:
    120120               
    121121               reference_variable_name = reference_variable.name
    122122               
    123123               for tree_node in stage._tree_nodes:
     124
     125                  variable_indices = tree_node._variable_indices[reference_variable_name]
    124126
    125127                  node_variable_average = tree_node._averages[reference_variable_name]
    126128                 
    127                   for index in reference_variable_indices:
     129                  for index in variable_indices:
    128130                     
    129131                     is_used = True # until proven otherwise
     
    174176         if stage != scenario_tree._stages[-1]:
    175177
    176             for (reference_variable, index_template, reference_variable_indices) in stage._variables:
     178            for (reference_variable, index_template) in stage._variables:
    177179               
    178180               reference_variable_name = reference_variable.name
     
    181183
    182184                  node_variable_average = tree_node._averages[reference_variable_name]
     185
     186                  variable_indices = tree_node._variable_indices[reference_variable_name]                 
    183187                 
    184                   for index in reference_variable_indices:
     188                  for index in variable_indices:
    185189
    186190                     # should think about nixing the magic constant below (not sure how to best pararamterize it).
  • coopr.pysp/stable/coopr/pysp/ef.py

    r3185 r3286  
    9595         print ""
    9696
    97       # update the scenario tree with cvar-specific variable names, so
    98       # they will get cloned accordingly in the master instance.
    99       first_stage = scenario_tree._stages[0]
    100       second_stage = scenario_tree._stages[1]
    101       root_node = first_stage._tree_nodes[0]
    102 
    103       # NOTE: because we currently don't have access to the reference
    104       #       instance in this method, temporarily (and largely orphaned)
    105       #       variables are constructed to supply to the scenario tree.
    106       #       this decision should be ultimately revisited.
    107       cvar_eta_variable_name = "CVAR_ETA"
    108       cvar_eta_variable = Var(name=cvar_eta_variable_name)
    109       cvar_eta_variable.construct()               
    110 
    111       first_stage.add_variable(cvar_eta_variable, "*", [None])
    112 
    113       cvar_excess_variable_name = "CVAR_EXCESS"
    114       cvar_excess_variable = Var(name=cvar_excess_variable_name)
    115       cvar_excess_variable.construct()
    116 
    117       second_stage.add_variable(cvar_excess_variable, "*", [None])
    118 
    11997      # create the eta and excess variable on a per-scenario basis,
    12098      # in addition to the constraint relating to the two.
     
    147125         scenario_instance.preprocess()
    148126
     127      # update the scenario tree with cvar-specific variable names, so
     128      # they will get cloned accordingly in the master instance.
     129      first_stage = scenario_tree._stages[0]
     130      second_stage = scenario_tree._stages[1]
     131      root_node = first_stage._tree_nodes[0]
     132
     133      # NOTE: because we currently don't have access to the reference
     134      #       instance in this method, temporarily (and largely orphaned)
     135      #       variables are constructed to supply to the scenario tree.
     136      #       this decision should be ultimately revisited.
     137      cvar_eta_variable_name = "CVAR_ETA"
     138      cvar_eta_variable = Var(name=cvar_eta_variable_name)
     139      cvar_eta_variable.construct()               
     140
     141      first_stage.add_variable(cvar_eta_variable, "*", scenario_instances)
     142
     143      cvar_excess_variable_name = "CVAR_EXCESS"
     144      cvar_excess_variable = Var(name=cvar_excess_variable_name)
     145      cvar_excess_variable.construct()
     146
     147      second_stage.add_variable(cvar_excess_variable, "*", scenario_instances)
     148
    149149   #
    150150   # create the new and empty binding instance.
     
    166166
    167167      # first loop is to create master (blended) variables across all stages but the last.
    168       for (stage_variable, index_template, stage_variable_indices) in stage._variables:
     168      for (stage_reference_variable, index_template) in stage._variables:
    169169
    170170         if verbose_output is True:
    171             print "Creating master variable and blending constraints for decision variable="+stage_variable.name+", indices="+str(index_template)
     171            print "Creating master variable and blending constraints for decision variable="+stage_reference_variable.name+", indices="+str(index_template)
    172172
    173173         for tree_node in stage._tree_nodes:
     
    175175            if stage != scenario_tree._stages[-1]:     
    176176
    177                master_variable_name = stage_variable.name               
     177               stage_variable_name = stage_reference_variable.name
     178
     179               stage_variable_indices = tree_node._variable_indices[stage_variable_name]
    178180
    179181               # because there may be a single stage variable and multiple indices, check
     
    182184               master_variable = None
    183185               try:
    184                   master_variable = getattr(binding_instance, master_variable_name)
     186                  master_variable = getattr(binding_instance, stage_variable_name)
    185187               except:
    186                   new_master_variable_index = stage_variable._index
     188                  new_master_variable_index = getattr(scenario_instances[tree_node._scenarios[0]._name], stage_variable_name)._index
    187189                  new_master_variable = None
    188190                  if (len(new_master_variable_index) is 1) and (None in new_master_variable_index):
    189                      new_master_variable = Var(name=stage_variable.name)
     191                     new_master_variable = Var(name=stage_variable_name)
    190192                  else:
    191                      new_master_variable = Var(new_master_variable_index, name=stage_variable.name)
     193                     new_master_variable = Var(new_master_variable_index, name=stage_variable_name)
    192194                  new_master_variable.construct()
    193195                  new_master_variable._model = binding_instance
    194                   setattr(binding_instance, master_variable_name, new_master_variable)
     196                  setattr(binding_instance, stage_variable_name, new_master_variable)
    195197
    196198                  master_variable = new_master_variable
     
    202204                  for scenario in tree_node._scenarios:
    203205                     instance = scenario_instances[scenario._name]
    204                      if getattr(instance,stage_variable.name)[index].status == VarStatus.unused:
     206                     if getattr(instance,stage_variable_name)[index].status == VarStatus.unused:
    205207                        is_used = False
    206208
     
    208210                  for scenario in tree_node._scenarios:
    209211                     instance = scenario_instances[scenario._name]
    210                      if getattr(instance,stage_variable.name)[index].fixed is True:
     212                     if getattr(instance,stage_variable_name)[index].fixed is True:
    211213                        is_fixed = True
    212214
     
    215217                     for scenario in tree_node._scenarios:
    216218                        scenario_instance = scenario_instances[scenario._name]
    217                         scenario_variable = getattr(scenario_instance, stage_variable.name)
    218                         new_constraint_name = scenario._name + "_" + master_variable_name + "_" + str(index)
     219                        scenario_variable = getattr(scenario_instance, stage_variable_name)
     220                        new_constraint_name = scenario._name + "_" + stage_variable_name + "_" + str(index)
    219221                        new_constraint = Constraint(name=new_constraint_name)
    220222                        new_expr = master_variable[index] - scenario_variable[index]
     
    506508
    507509def write_ef_from_scratch(model_directory, instance_directory, output_filename,
    508                           verbose_output, linearize,
     510                          verbose_output, linearize_expressions, tree_downsample_fraction, tree_random_seed,
    509511                          generate_weighted_cvar, cvar_weight, risk_alpha):
    510512
     
    575577
    576578   #
     579   # compress/down-sample the scenario tree, if operation is required.
     580   #
     581   if tree_downsample_fraction < 1.0:
     582       
     583      scenario_tree.downsample(tree_downsample_fraction, tree_random_seed, verbose_output)   
     584
     585   #
    577586   # print the input tree for validation/information purposes.
    578587   #
     
    624633                                                      verbose=verbose_output,
    625634                                                      preprocess=True,
    626                                                       linearize=linearize)
     635                                                      linearize=linearize_expressions)
    627636
    628637      scenario_instances[scenario._name] = scenario_instance
     
    632641   if re_enable_gc is True:
    633642      gc.enable()
     643
     644   # with the scenario instances now available, have the scenario tree compute the
     645   # variable match indices at each node.
     646   scenario_tree.defineVariableIndexSets(scenario_instances)     
    634647
    635648   print "Creating extensive form binding instance"
  • coopr.pysp/stable/coopr/pysp/ef_writer_script.py

    r3185 r3286  
    4141   parser.usage=usage_string
    4242
    43    inputOpts  = OptionGroup( parser, 'Input Options' )
    44    efOpts     = OptionGroup( parser, 'EF Options' )
    45    solverOpts = OptionGroup( parser, 'Solver Options' )
    46    outputOpts = OptionGroup( parser, 'Output Options' )
    47    otherOpts  = OptionGroup( parser, 'Other Options' )
     43   inputOpts        = OptionGroup( parser, 'Input Options' )
     44   scenarioTreeOpts = OptionGroup( parser, 'Scenario Tree Options' )   
     45   efOpts           = OptionGroup( parser, 'EF Options' )
     46   solverOpts       = OptionGroup( parser, 'Solver Options' )
     47   outputOpts       = OptionGroup( parser, 'Output Options' )
     48   otherOpts        = OptionGroup( parser, 'Other Options' )
    4849   parser.add_option_group( inputOpts )
     50   parser.add_option_group( scenarioTreeOpts )
    4951   parser.add_option_group( efOpts )
    5052   parser.add_option_group( solverOpts )
     
    6466     type='string',
    6567     default='.')
     68
     69   scenarioTreeOpts.add_option('--scenario-tree-seed',
     70     help="The random seed associated with manipulation operations on the scenario tree (e.g., down-sampling). Default is 0, indicating unassigned.",
     71     action="store",
     72     dest="scenario_tree_random_seed",
     73     type="int",
     74     default=None)
     75   scenarioTreeOpts.add_option('--scenario-tree-downsample-fraction',
     76     help="The proportion of the scenarios in the scenario tree that are actually used. Specific scenarios are selected at random. Default is 1.0, indicating no down-sampling.",
     77     action="store",
     78     dest="scenario_tree_downsample_fraction",
     79     type="float",
     80     default=1.0)   
    6681
    6782   efOpts.add_option('--cvar-weight',
     
    192207                                                                               options.verbose,
    193208                                                                               options.linearize_expressions,
     209                                                                               options.scenario_tree_downsample_fraction,
     210                                                                               options.scenario_tree_random_seed,
    194211                                                                               generate_weighted_cvar, cvar_weight, risk_alpha)
    195212
  • coopr.pysp/stable/coopr/pysp/ph.py

    r3185 r3286  
    154154         for tree_node in stage._tree_nodes:
    155155
    156             for (variable, index_template, variable_indices) in stage._variables:
     156            for (variable, index_template) in stage._variables:
     157
     158               variable_indices = tree_node._variable_indices[variable.name]
    157159
    158160               for index in variable_indices:
     
    190192         for tree_node in stage._tree_nodes:
    191193
    192             for (variable, index_template, variable_indices) in stage._variables:
     194            for (variable, index_template) in stage._variables:
     195
     196               variable_indices = tree_node._variable_indices[variable.name]
    193197
    194198               for index in variable_indices:
     
    282286         for stage in self._scenario_tree._stages[:-1]: # no blending over the final stage, so no weights to worry about.
    283287
    284             for (variable, index_template, variable_indices) in stage._variables:
     288            for (variable, index_template) in stage._variables:
    285289
    286290               variable_name = variable.name
     
    306310         for stage in self._scenario_tree._stages[:-1]: # no blending over the final stage, so no rhos to worry about.
    307311
    308             for (variable, index_template, variable_indices) in stage._variables:
     312            for (variable, index_template) in stage._variables:
    309313
    310314               variable_name = variable.name
     
    668672         self._instance_augmented_attributes[scenario._name] = []
    669673
    670 
    671674      # perform a single pass of garbage collection and re-enable automatic collection.
    672675      if re_enable_gc is True:
    673676         gc.collect()
    674677         gc.enable()
     678
     679      # with the scenario instances now available, have the scenario tree compute the
     680      # variable match indices at each node.
     681      self._scenario_tree.defineVariableIndexSets(self._instances)
    675682
    676683      # let plugins know if they care - this callback point allows
     
    712719
    713720         stage_variables = {}
    714          for (reference_variable, index_template, reference_index) in stage._variables:
     721         for (reference_variable, index_template) in stage._variables:
    715722            if reference_variable.name not in stage_variables.keys():
    716723               stage_variables[reference_variable.name] = reference_variable
     
    720727         #       explicitly.
    721728         for (variable_name, reference_variable) in stage_variables.items():
     729           
    722730            for tree_node in stage._tree_nodes:
    723731
    724                new_min_index = reference_variable._index
     732               new_variable_index = getattr(self._instances[tree_node._scenarios[0]._name], variable_name)._index
     733
     734               new_min_index = new_variable_index
    725735               new_min_parameter_name = "NODEMIN_"+reference_variable.name
    726736               # this bit of ugliness is due to Pyomo not correctly handling the Param construction
     
    736746               tree_node._minimums[reference_variable.name] = new_min_parameter
    737747
    738                new_avg_index = reference_variable._index
     748               new_avg_index = new_variable_index
    739749               new_avg_parameter_name = "NODEAVG_"+reference_variable.name
    740750               new_avg_parameter = None
     
    747757               tree_node._averages[reference_variable.name] = new_avg_parameter
    748758
    749                new_max_index = reference_variable._index
     759               new_max_index = new_variable_index
    750760               new_max_parameter_name = "NODEMAX_"+reference_variable.name
    751761               new_max_parameter = None
     
    924934         for tree_node in stage._tree_nodes:
    925935
    926             for (variable, index_template, variable_indices) in stage._variables:
     936            for (variable, index_template) in stage._variables:
    927937
    928938               variable_name = variable.name
     939
     940               variable_indices = tree_node._variable_indices[variable_name]
    929941
    930942               avg_parameter_name = "PHAVG_"+variable_name
     
    9961008         for tree_node in stage._tree_nodes:
    9971009
    998             for (variable, index_template, variable_indices) in stage._variables:
     1010            for (variable, index_template) in stage._variables:
    9991011
    10001012               variable_name = variable.name
     
    10021014               weight_parameter_name = "PHWEIGHT_"+variable_name
    10031015               rho_parameter_name = "PHRHO_"+variable_name
     1016
     1017               variable_indices = tree_node._variable_indices[variable_name]
    10041018
    10051019               for index in variable_indices:
     
    13691383
    13701384      if self._output_scenario_tree_solution is True:
    1371          self._scenario_tree.snapshotSolutionFromAverages()
     1385         self._scenario_tree.snapshotSolutionFromAverages(self._instances)
    13721386         print "Final solution (scenario tree format):"
    13731387         self._scenario_tree.pprintSolution()
     
    14001414   # just the variable/stage.
    14011415   #
    1402    def should_print(self, stage, variable, variable_indices):
     1416   def should_print(self, stage, variable):
    14031417
    14041418      if self._output_continuous_variable_stats is False:
     
    14291443         num_outputs_this_stage = 0 # tracks the number of outputs on a per-index basis.
    14301444
    1431          for (variable, index_template, variable_indices) in stage._variables:
     1445         for (variable, index_template) in stage._variables:
    14321446
    14331447            variable_name = variable.name
    14341448
    1435             if self.should_print(stage, variable, variable_indices) is True:
     1449            if self.should_print(stage, variable) is True:
    14361450
    14371451               num_outputs_this_variable = 0 # track, so we don't output the variable names unless there is an entry to report.
    14381452
    1439                for index in variable_indices:
    1440 
    1441                   weight_parameter_name = "PHWEIGHT_"+variable_name
    1442 
    1443                   num_outputs_this_index = 0 # track, so we don't output the variable index more than once.
    1444 
    1445                   for tree_node in stage._tree_nodes:
     1453               for tree_node in stage._tree_nodes:
     1454
     1455                  variable_indices = tree_node._variable_indices[variable_name]               
     1456
     1457                  for index in variable_indices:
     1458
     1459                     weight_parameter_name = "PHWEIGHT_"+variable_name
     1460
     1461                     num_outputs_this_index = 0 # track, so we don't output the variable index more than once.
    14461462
    14471463                     # determine if the variable/index pair is used across the set of scenarios (technically,
  • coopr.pysp/stable/coopr/pysp/phinit.py

    r3185 r3286  
    5151   parser.usage = usage_string
    5252
    53    inputOpts       = OptionGroup( parser, 'Input Options' )
    54    phOpts          = OptionGroup( parser, 'PH Options' )
    55    solverOpts      = OptionGroup( parser, 'Solver Options' )
    56    postprocessOpts = OptionGroup( parser, 'Postprocessing Options' )   
    57    outputOpts      = OptionGroup( parser, 'Output Options' )
    58    otherOpts       = OptionGroup( parser, 'Other Options' )
     53   # NOTE: these groups should eventually be queried from the PH, scenario tree, etc. classes (to facilitate re-use).
     54   inputOpts        = OptionGroup( parser, 'Input Options' )
     55   scenarioTreeOpts = OptionGroup( parser, 'Scenario Tree Options' )
     56   phOpts           = OptionGroup( parser, 'PH Options' )
     57   solverOpts       = OptionGroup( parser, 'Solver Options' )
     58   postprocessOpts  = OptionGroup( parser, 'Postprocessing Options' )   
     59   outputOpts       = OptionGroup( parser, 'Output Options' )
     60   otherOpts        = OptionGroup( parser, 'Other Options' )
    5961   parser.add_option_group( inputOpts )
     62   parser.add_option_group( scenarioTreeOpts )
    6063   parser.add_option_group( phOpts )
    6164   parser.add_option_group( solverOpts )
     
    8184     dest="bounds_cfgfile",
    8285     default=None)
     86
     87   scenarioTreeOpts.add_option('--scenario-tree-seed',
     88     help="The random seed associated with manipulation operations on the scenario tree (e.g., down-sampling). Default is 0, indicating unassigned.",
     89     action="store",
     90     dest="scenario_tree_random_seed",
     91     type="int",
     92     default=None)
     93   scenarioTreeOpts.add_option('--scenario-tree-downsample-fraction',
     94     help="The proportion of the scenarios in the scenario tree that are actually used. Specific scenarios are selected at random. Default is 1.0, indicating no down-sampling.",
     95     action="store",
     96     dest="scenario_tree_downsample_fraction",
     97     type="float",
     98     default=1.0)   
    8399
    84100   phOpts.add_option('-r','--default-rho',
     
    412428   scenario_tree = ScenarioTree(scenarioinstance=reference_instance,
    413429                                scenariotreeinstance=scenario_tree_instance)
     430
     431   #
     432   # compress/down-sample the scenario tree, if operation is required.
     433   #
     434   if options.scenario_tree_downsample_fraction < 1.0:
     435       
     436      scenario_tree.downsample(options.scenario_tree_downsample_fraction, options.scenario_tree_random_seed, options.verbose)
    414437
    415438   return reference_model, reference_instance, scenario_tree, scenario_tree_instance
  • coopr.pysp/stable/coopr/pysp/phobjective.py

    r3185 r3286  
    243243   quad_expression = 0.0
    244244
     245#   print "ORIGINAL OBJECTIVE EXPRESSION:"
     246#   original_objective_expression.pprint()
     247
     248#   print "CLONED OBJECTIVE EXPRESSION:"
     249#   objective_expression.pprint()
     250
     251#   foobar
     252
    245253   # for each blended variable (i.e., those not appearing in the final stage),
    246254   # add the linear and quadratic penalty terms to the objective.
     
    255263               break
    256264
    257       for (reference_variable, index_template, variable_indices) in stage._variables:
     265      for (reference_variable, index_template) in stage._variables:
    258266
    259267         variable_name = reference_variable.name
     
    281289
    282290         instance_variable = instance.active_components(Var)[variable_name]
     291
     292         variable_indices = variable_tree_node._variable_indices[variable_name]
    283293
    284294         for index in variable_indices:
  • coopr.pysp/stable/coopr/pysp/phutils.py

    r3185 r3286  
    102102   # replace all parens in the string with commas and
    103103   # proceed with the split.
    104    full_index=string.translate(full_index, None, "()")
     104   full_index=string.translate(full_index, string.maketrans("",""), "()")
    105105   indices = full_index.split(',')
    106106
     
    116116      # strings. remove these, as otherwise you have an
    117117      # illegal index.
    118       index=string.translate(index, None, "\'")
     118      index=string.translate(index, string.maketrans("",""), "\'")
    119119
    120120      # if the index is an integer, make it one!
     
    212212      else:
    213213         scenario_instance = reference_model.clone()
    214          scenario_data = ModelData()
     214
     215         data_files = []
    215216         current_node = scenario._leaf_node
    216217         while current_node is not None:
    217218            node_data_filename = scenario_data_directory_name + os.sep + current_node._name + ".dat"
     219            data_files.append(node_data_filename)
     220            current_node = current_node._parent         
     221
     222         # to make sure we read from root node to leaf node
     223         data_files.reverse()
     224
     225         scenario_data = ModelData()
     226         for data_file in data_files:
    218227            if verbose is True:
    219                print "Node data for scenario=" + scenario._name + " partially loading from file=" + node_data_filename
    220             scenario_data.add(node_data_filename)
    221             current_node = current_node._parent
     228               print "Node data for scenario=" + scenario._name + " partially loading from file=" + data_file
     229            scenario_data.add(data_file)
     230
    222231         scenario_data.read(model=scenario_instance)
    223232         scenario_instance.load(scenario_data, simplify=simplify)
     
    262271   for stage in scenario_tree._stages[:-1]:
    263272
    264       for (reference_variable, index_template, reference_indices) in stage._variables:
     273      for (reference_variable, index_template) in stage._variables:
    265274
    266275         if reference_variable.name not in instance_variables.keys():
  • coopr.pysp/stable/coopr/pysp/scenariotree.py

    r3115 r3286  
    1717from math import fabs
    1818
     19import random
     20
    1921from coopr.pyomo import *
    2022from phutils import *
     
    2628   # variable/match-template/variable-index triple.
    2729   #
    28    def _update_solution_map(self, variable, match_template, variable_indices):
     30   def _update_solution_map(self, variable, match_template, scenario_instance_map):
    2931
    3032      # don't bother copying bounds for variables, as the values stored
     
    3234      # ensuring feasibility. this also leaves room for specifying infeasible
    3335      # or partial solutions.
    34       new_variable_index = variable._index
    35       new_variable_name = variable.name
     36      new_variable_name = variable.name     
     37      new_variable_index = getattr(scenario_instance_map[self._scenarios[0]._name], variable.name)._index
    3638      new_variable = None
    3739      if (len(new_variable_index) is 1) and (None in new_variable_index):
     
    4143      new_variable.construct()
    4244
     45      variable_indices = self._variable_indices[variable.name]     
     46
    4347      # by default, deactive all variable values - we're copying the
    4448      # full index set of a variable, which is necessarily wasteful and
     
    5761   #
    5862
    59    def _initialize_solution_map(self):
     63   def _initialize_solution_map(self, scenario_instance_map):
    6064
    6165      # clear whatever was there before.
     
    6973      # this is something that we might revisit if space/performance
    7074      # is an issue (space is the most likely issue)
    71       for variable, match_template, variable_indices in self._stage._variables:
    72          self._update_solution_map(variable, match_template, variable_indices)
     75      for variable, match_template in self._stage._variables:
     76         self._update_solution_map(variable, match_template, scenario_instance_map)
    7377
    7478      self._solution_map_initialized = True
     
    7781   """ Constructor
    7882   """
    79    def __init__(self, name, conditional_probability, stage, initialize_solution=False):
     83   def __init__(self, name, conditional_probability, stage, initialize_solution=False, scenario_instance_map={}):
    8084
    8185      self._name = name
     
    8589      self._conditional_probability = conditional_probability # conditional on parent
    8690      self._scenarios = [] # a collection of all Scenarios passing through this node in the tree
     91      self._variable_indices = {} # a map from a variable name to the indices blended at this node.
    8792
    8893      # general use statistics for the variables at each node.
     
    111116
    112117      if initialize_solution is True:
    113          self._initialize_solution_map()
     118         self._initialize_solution_map(scenario_instance_map)
     119
     120   #
     121   # given a set of scenario instances, compute the set of indices being blended
     122   # for each variable at this node.
     123   #
     124
     125   def defineVariableIndexSets(self, scenario_instances):
     126
     127      # find a representative scenario instance belonging to this node. the
     128      # first scenario is as good as any.
     129      scenario_instance = scenario_instances[self._scenarios[0]._name]
     130
     131      for reference_variable, match_template in self._stage._variables:
     132
     133         # the stage variable simply references the variable object in the
     134         # reference scenario instance - we need to grab the variable in the
     135         # scenario instance, as the index set might be different.
     136         variable_name = reference_variable.name
     137
     138         instance_variable = getattr(scenario_instance, variable_name)
     139
     140         match_indices = extractVariableIndices(instance_variable, match_template)
     141
     142         self._variable_indices[variable_name] = match_indices
    114143
    115144   #
     
    118147   #
    119148
    120    def snapshotSolutionFromAverages(self):
     149   def snapshotSolutionFromAverages(self, scenario_instance_map):
    121150
    122151      if self._solution_map_initialized is False:
    123          self._initialize_solution_map()
     152         self._initialize_solution_map(scenario_instance_map)
    124153
    125154      for variable_name, variable in self._solutions.items():
     
    145174
    146175      if self._solution_map_initialized is False:
    147          self._initialize_solution_map()
     176         self._initialize_solution_map(scenario_instance_map)
    148177
    149178      for variable_name, variable in self._solutions.items():
     
    195224      self._tree_nodes = []
    196225
    197       # a collection of triples consisting of (1) a reference to a Pyomo model Var object, (2) the original match
    198       # template string (for output purposes), and (3) a *list* of the corresponding indices. the variables are
    199       # references to those objects belonging to the Pyomo reference scenario instance associated with the parent
    200       # ScenarioTree of this Stage.
    201       # NOTE: if the variable index is none, it is assumed that the entire variable is blended.
     226      # a collection of pairs consisting of (1) a reference to a Pyomo model Var object (in the reference scenario instance) and
     227      # (2) the original match template string (for output purposes). the specific indices that match belong to the tree node.
    202228      self._variables = []
    203229
     
    209235   # add a new variable to the stage, which will include updating the solution maps for each associated ScenarioTreeNode.
    210236   #
    211    def add_variable(self, variable, match_template, indices):
    212 
    213       self._variables.append((variable, match_template, indices))
     237   def add_variable(self, variable, match_template, scenario_instance_map):
     238
     239      self._variables.append((variable, match_template))
     240      match_indices = extractVariableIndices(variable, match_template)
    214241
    215242      for tree_node in self._tree_nodes:
    216          tree_node._update_solution_map(variable, match_template, indices)
     243         tree_node._variable_indices[variable.name] = match_indices
     244         tree_node._update_solution_map(variable, match_template, scenario_instance_map)
    217245
    218246class Scenario(object):
     
    263291               variable = self._reference_instance.active_components(Var)[variable_name]
    264292
    265                # extract all "real", i.e., fully specified, indices matching the index template.
    266                match_indices = extractVariableIndices(variable, index_template)
    267 
    268                # there is a possibility that no indices match the input template.
    269                # if so, let the user know about it.
    270                if len(match_indices) == 0:
    271                   raise RuntimeError, "No indices match template="+str(index_template)+" for variable="+variable_name+" ; encountered in scenario tree specification for model="+self._reference_instance.name
    272 
    273                stage._variables.append((variable, index_template, match_indices))
     293               stage._variables.append((variable, index_template))
    274294
    275295            else:
     
    293313               match_indices.append(None)
    294314
    295                stage._variables.append((variable, "", match_indices))
     315               stage._variables.append((variable, ""))
    296316
    297317      for stage_id in stage_cost_variable_ids.keys():
     
    506526
    507527   #
     528   # given a set of scenario instances, compute the set of variable indices being blended at each node.
     529   # this can't be done until the scenario instances are available, as different scenarios can have
     530   # different index sets.
     531   #
     532
     533   def defineVariableIndexSets(self, scenario_instances):
     534
     535      for tree_node in self._tree_nodes:
     536
     537         tree_node.defineVariableIndexSets(scenario_instances)
     538
     539   #
    508540   # is the indicated scenario in the tree?
    509541   #
     
    626658
    627659   #
     660   # utility for automatically selecting a proportion of scenarios from the
     661   # tree to retain, eliminating the rest.
     662   #
     663
     664   def downsample(self, fraction_to_retain, random_seed, verbose=False):
     665
     666      random.seed(random_seed)
     667
     668      random_sequence=range(0,len(self._scenarios))
     669      random.shuffle(random_sequence)
     670
     671      number_to_retain = max(int(round(float(len(random_sequence)*fraction_to_retain))), 1)
     672
     673      scenario_bundle_list = []
     674      for i in range(0,number_to_retain):
     675         scenario_bundle_list.append(self._scenarios[random_sequence[i]]._name)
     676
     677      if verbose is True:
     678         print "Downsampling scenario tree - retained scenarios: "+str(scenario_bundle_list)
     679
     680      self.compress(scenario_bundle_list)
     681
     682
     683   #
    628684   # returns the root node of the scenario tree
    629685   #
     
    689745   #
    690746
    691    def snapshotSolutionFromAverages(self):
     747   def snapshotSolutionFromAverages(self, scenario_instance_map):
    692748
    693749      for tree_node in self._tree_nodes:
    694750
    695          tree_node.snapshotSolutionFromAverages()
     751         tree_node.snapshotSolutionFromAverages(scenario_instance_map)
    696752
    697753   #
     
    717773
    718774      for stage in self._stages:
    719          # stage_var is a VarValue - the rest are strings and list of indices, respectively.
    720          for (stage_var, match_template, match_indices) in stage._variables:
     775         for (stage_var, match_template) in stage._variables:
    721776            if (variable.name == stage_var.name) and (index in match_indices):
    722777               return stage
     
    819874            print "\t\t" + tree_node._name
    820875         print "\tVariables: "
    821          for (variable, index_template, indices) in stage._variables:
    822             if (len(indices) == 1) and (indices[0] == None):
    823                print "\t\t" + variable.name
    824             else:
    825                print "\t\t",variable.name,":",index_template
     876         for (variable, index_template) in stage._variables:
     877            print "\t\t",variable.name,":",index_template
    826878         print "\tCost Variable: "
    827879         if stage._cost_variable[1] is None:
     
    867919            print "\tParent=" + "None"
    868920         print "\tVariables: "
    869          for (variable, index_template, indices) in tree_node._stage._variables:
     921         for (variable, index_template) in tree_node._stage._variables:
     922            indices = tree_node._variable_indices[variable.name]
    870923            solution_variable = tree_node._solutions[variable.name]
    871924            if (len(indices) == 1) and (indices[0] == None):
  • coopr.pysp/stable/coopr/pysp/tests/unit/test_ph.py

    r2931 r3286  
    2424import coopr.plugins.mip
    2525import coopr.pysp.phinit
     26import coopr.pysp.ef_writer_script
    2627
    2728def filter_time(line):
    28    return "seconds" in line
     29    return "seconds" in line or line.startswith("Data for")
    2930
    3031cplex = None
     
    127128        self.cleanup()       
    128129
     130    def test_farmer_ef(self):
     131        pyutilib.misc.setup_redirect(current_directory+"farmer_ef.out")
     132        farmer_examples_dir = pysp_examples_dir + "farmer"
     133        model_dir = farmer_examples_dir + os.sep + "models"
     134        instance_dir = farmer_examples_dir + os.sep + "scenariodata"       
     135        argstring = "runef --verbose --output-file=farmer_ef.out --model-directory="+model_dir+" --instance-directory="+instance_dir
     136        args = string.split(argstring)
     137        coopr.pysp.ef_writer_script.run(args=args)       
     138        pyutilib.misc.reset_redirect()
     139        self.cleanup()               
     140        self.failUnlessFileEqualsBaseline(current_directory+"farmer_ef.out",current_directory+"farmer_ef.baseline", filter=filter_time)
     141
     142    def test_farmer_ef_cvar(self):
     143        pyutilib.misc.setup_redirect(current_directory+"farmer_ef_cvar.out")
     144        farmer_examples_dir = pysp_examples_dir + "farmer"
     145        model_dir = farmer_examples_dir + os.sep + "models"
     146        instance_dir = farmer_examples_dir + os.sep + "scenariodata"       
     147        argstring = "runef --verbose --output-file=farmer_ef_cvar.out --generate-weighted-cvar --risk-alpha=0.90 --cvar-weight=0.0 --model-directory="+model_dir+" --instance-directory="+instance_dir
     148        args = string.split(argstring)
     149        coopr.pysp.ef_writer_script.run(args=args)       
     150        pyutilib.misc.reset_redirect()
     151        self.cleanup()               
     152        self.failUnlessFileEqualsBaseline(current_directory+"farmer_ef_cvar.out",current_directory+"farmer_ef_cvar.baseline", filter=filter_time)
     153
    129154TestPH = unittest.category('nightly')(TestPH)
    130155
  • coopr.pysp/stable/coopr/pysp/wwphextension.py

    r3115 r3286  
    247247
    248248      # set up tree storage for statistics that we care about tracking.
    249       for stage in ph._scenario_tree._stages:
    250 
    251          if stage != ph._scenario_tree._stages[-1]:
    252 
    253             # first, gather all unique variables referenced in self stage
    254             # self "gather" step is currently required because we're being lazy
    255             # in terms of index management in the scenario tree
    256 ##            stage_variables = {}
    257 ##            for (reference_variable, index_template, reference_index) in stage._variables:
    258 ##               if reference_variable.name not in stage_variables.keys():
    259 ##                  stage_variables[reference_variable.name] = reference_variable
     249      for stage in ph._scenario_tree._stages[:-1]:
     250
     251         # first, gather all unique variables referenced in self stage
     252         # self "gather" step is currently required because we're being lazy
     253         # in terms of index management in the scenario tree
     254##         stage_variables = {}
     255##         for (reference_variable, index_template, reference_index) in stage._variables:
     256##            if reference_variable.name not in stage_variables.keys():
     257##               stage_variables[reference_variable.name] = reference_variable
    260258##
    261             for tree_node in stage._tree_nodes:
    262                tree_node._num_iters_converged = {}
    263                tree_node._last_converged_val = {}
    264                tree_node._w_hash = {}
    265                tree_node._fixed_var_flag = {}
    266 
    267                for (variable, index_template, variable_indices) in stage._variables:
    268 
    269                   variable_name = variable.name
    270                   variable_type = variable.domain
    271 
    272                   # create the parameters for all indices, even though this might be a bit
    273                   # wasteful. in practice, this probably won't be the case, and determining
    274                   # the reduced index set would probably take longer that it is worth.
    275 
    276                   new_stat_index = variable._index
    277                   new_stat_parameter_name = "NODESTAT_NUM_ITERS_CONVERGED_"+variable.name
    278                   new_stat_parameter = None
    279                   # this bit of ugliness is due to Pyomo not correctly handling the Param construction
    280                   # case when the supplied index set consists strictly of None, i.e., the source variable
    281                   # is a singleton. this case be cleaned up when the source issue in Pyomo is fixed.                     
    282                   if (len(new_stat_index) is 1) and (None in new_stat_index):
    283                      new_stat_parameter = Param(name=new_stat_parameter_name, mutable=True)
    284                   else:
    285                      new_stat_parameter = Param(new_stat_index, name=new_stat_parameter_name, mutable=True)
    286                   for newindex in new_stat_index:
    287                      new_stat_parameter[newindex] = 0
    288                   tree_node._num_iters_converged[variable.name] = new_stat_parameter
     259         for tree_node in stage._tree_nodes:
     260               
     261            tree_node._num_iters_converged = {}
     262            tree_node._last_converged_val = {}
     263            tree_node._w_hash = {}
     264            tree_node._fixed_var_flag = {}
     265
     266            for (reference_variable, index_template) in stage._variables:
     267
     268               variable_name = reference_variable.name
     269               variable_type = reference_variable.domain
     270
     271               variable_indices = tree_node._variable_indices[variable_name]
     272
     273               new_param_index = getattr(ph._instances[tree_node._scenarios[0]._name], variable_name)._index
     274
     275               # create the parameters for all indices, even though this might be a bit
     276               # wasteful. in practice, this probably won't be the case, and determining
     277               # the reduced index set would probably take longer that it is worth.
     278
     279               new_stat_index = new_param_index
     280               new_stat_parameter_name = "NODESTAT_NUM_ITERS_CONVERGED_"+reference_variable.name
     281               new_stat_parameter = None
     282               # this bit of ugliness is due to Pyomo not correctly handling the Param construction
     283               # case when the supplied index set consists strictly of None, i.e., the source variable
     284               # is a singleton. this case be cleaned up when the source issue in Pyomo is fixed.                     
     285               if (len(new_stat_index) is 1) and (None in new_stat_index):
     286                  new_stat_parameter = Param(name=new_stat_parameter_name, mutable=True)
     287               else:
     288                  new_stat_parameter = Param(new_stat_index, name=new_stat_parameter_name, mutable=True)
     289               for newindex in new_stat_index:
     290                  new_stat_parameter[newindex] = 0
     291               tree_node._num_iters_converged[reference_variable.name] = new_stat_parameter
    289292                     
    290                   # need to know to what we have most recently converged
    291                   new_conv_index = variable._index
    292                   new_conv_parameter_name = "NODESTAT_LAST_CONVERGED_VAL_"+variable.name
    293                   new_conv_parameter = None
    294                   if (len(new_conv_index) is 1) and (None in new_conv_index):
    295                      new_conv_parameter = Param(name=new_conv_parameter_name, mutable=True)
    296                   else:
    297                      new_conv_parameter = Param(new_conv_index, name=new_conv_parameter_name, mutable=True)
    298                   for newindex in new_conv_index:
    299                      new_conv_parameter[newindex] = 0.5 # not an int, so harmless
    300                   tree_node._last_converged_val[variable.name] = new_conv_parameter
     293               # need to know to what we have most recently converged
     294               new_conv_index = new_param_index
     295               new_conv_parameter_name = "NODESTAT_LAST_CONVERGED_VAL_"+reference_variable.name
     296               new_conv_parameter = None
     297               if (len(new_conv_index) is 1) and (None in new_conv_index):
     298                  new_conv_parameter = Param(name=new_conv_parameter_name, mutable=True)
     299               else:
     300                  new_conv_parameter = Param(new_conv_index, name=new_conv_parameter_name, mutable=True)
     301               for newindex in new_conv_index:
     302                  new_conv_parameter[newindex] = 0.5 # not an int, so harmless
     303               tree_node._last_converged_val[reference_variable.name] = new_conv_parameter
    301304                     
    302                   # need to know to what has been fixed
    303                   new_fix_index = variable._index
    304                   new_fix_parameter_name = "NODESTAT_FIXED_FLAG_VAL_"+variable.name
    305                   new_fix_parameter = None
    306                   if (len(new_fix_index) is 1) and (None in new_fix_index):
    307                      new_fix_parameter = Param(name=new_fix_parameter_name, mutable=True)
    308                   else:
    309                      new_fix_parameter = Param(new_fix_index, name=new_fix_parameter_name, mutable=True)
    310                   for newindex in new_fix_index:
    311                      new_fix_parameter[newindex] = False
    312                   tree_node._fixed_var_flag[variable.name] = new_fix_parameter
    313 
    314                   # now make the w hash value storage array
    315                   new_hash_index = variable._index
    316                   new_hash_parameter_name = "W_HASH_STORAGE_"+variable.name
    317                   new_hash_parameter = None
    318                   if (len(new_hash_index) is 1) and (None in new_hash_index):
    319                      new_hash_parameter = Param(ph._iteration_index_set, name=new_hash_parameter_name, mutable=True)
    320                   else:
    321                      new_hash_parameter = Param(new_hash_index, ph._iteration_index_set, name=new_hash_parameter_name, mutable=True)
    322                   for newindex in new_hash_index:
    323                      for i in range(0, ph._max_iterations+1):
    324                         # the following if-then block is a complete hack, due to the
    325                         # fact that we can't index by None if the Param is unary.
    326                         if new_hash_parameter.dim() == 1:
    327                            new_hash_parameter[i] = 0
    328                         else:
    329                            new_hash_parameter[newindex,i] = 0
    330                   tree_node._w_hash[variable.name] = new_hash_parameter
     305               # need to know to what has been fixed
     306               new_fix_index = new_param_index
     307               new_fix_parameter_name = "NODESTAT_FIXED_FLAG_VAL_"+reference_variable.name
     308               new_fix_parameter = None
     309               if (len(new_fix_index) is 1) and (None in new_fix_index):
     310                  new_fix_parameter = Param(name=new_fix_parameter_name, mutable=True)
     311               else:
     312                  new_fix_parameter = Param(new_fix_index, name=new_fix_parameter_name, mutable=True)
     313               for newindex in new_fix_index:
     314                  new_fix_parameter[newindex] = False
     315               tree_node._fixed_var_flag[reference_variable.name] = new_fix_parameter
     316
     317               # now make the w hash value storage array
     318               new_hash_index = new_param_index
     319               new_hash_parameter_name = "W_HASH_STORAGE_"+reference_variable.name
     320               new_hash_parameter = None
     321               if (len(new_hash_index) is 1) and (None in new_hash_index):
     322                  new_hash_parameter = Param(ph._iteration_index_set, name=new_hash_parameter_name, mutable=True)
     323               else:
     324                  new_hash_parameter = Param(new_hash_index, ph._iteration_index_set, name=new_hash_parameter_name, mutable=True)
     325               for newindex in new_hash_index:
     326                  for i in range(0, ph._max_iterations+1):
     327                     # the following if-then block is a complete hack, due to the
     328                     # fact that we can't index by None if the Param is unary.
     329                     if new_hash_parameter.dim() == 1:
     330                        new_hash_parameter[i] = 0
     331                     else:
     332                        new_hash_parameter[newindex,i] = 0
     333               tree_node._w_hash[reference_variable.name] = new_hash_parameter
    331334                 
    332                   # JPW has no idea why the following code block is here, or if it is necessary.
    333                   for index in variable_indices:
    334 
    335                      # IMPT: it has bitten us before in this plug-in, so I'll state the obvious:
    336                      #       variable values in the last stage of a stochastic program will *not*
    337                      #       have a defined _stage attribute.
    338                      variable[index]._stage = stage
     335               # JPW has no idea why the following code block is here, or if it is necessary.
     336               for index in variable_indices:
     337
     338                  # IMPT: it has bitten us before in this plug-in, so I'll state the obvious:
     339                  #       variable values in the last stage of a stochastic program will *not*
     340                  #       have a defined _stage attribute.
     341                  # NOTE: the use of the reference variable here, in contrast to a instance-specific
     342                  #       variable, is potentially dangerous.
     343                  reference_variable[index]._stage = stage
    339344
    340345
     
    362367   def post_iteration_0_solves(self, ph):
    363368
    364       for stage in ph._scenario_tree._stages:
     369      for stage in ph._scenario_tree._stages[:-1]:
    365370         
    366          if stage != ph._scenario_tree._stages[-1]: # no blending over the final stage
    367            
    368             for tree_node in stage._tree_nodes:
    369 
    370                for (variable, index_template, variable_indices) in stage._variables:
     371         for tree_node in stage._tree_nodes:
     372
     373            for (variable, index_template) in stage._variables:
    371374                 
    372                   variable_name = variable.name
    373                   variable_type = variable.domain
    374 
    375                   for index in variable_indices:
    376 
    377                      # determine if this index is used - otherwise, don't waste time
    378                      # fixing and cycle checking. for one, the code will crash :-) with
    379                      # none values during the cycle checking computation!
    380 
    381                      is_used = True # until proven otherwise                     
     375               variable_name = variable.name
     376               variable_type = variable.domain
     377
     378               variable_indices = tree_node._variable_indices[variable_name]
     379
     380               for index in variable_indices:
     381
     382                  # determine if this index is used - otherwise, don't waste time
     383                  # fixing and cycle checking. for one, the code will crash :-) with
     384                  # none values during the cycle checking computation!
     385
     386                  is_used = True # until proven otherwise                     
     387                  for scenario in tree_node._scenarios:
     388                     instance = ph._instances[scenario._name]
     389                     if getattr(instance,variable_name)[index].status == VarStatus.unused:
     390                        is_used = False
     391
     392                  if is_used is True:
     393
     394                     # unlikely, but variables might be fixed even at this stage, depending on
     395                     # what weirdness users do prior to the iteration 0 solves.
     396                     instance_fixed_count = 0
    382397                     for scenario in tree_node._scenarios:
    383398                        instance = ph._instances[scenario._name]
    384                         if getattr(instance,variable_name)[index].status == VarStatus.unused:
    385                            is_used = False
    386 
    387                      if is_used is True:
    388 
    389                         # unlikely, but variables might be fixed even at this stage, depending on
    390                         # what weirdness users do prior to the iteration 0 solves.
    391                         instance_fixed_count = 0
    392                         for scenario in tree_node._scenarios:
    393                            instance = ph._instances[scenario._name]
    394                            if getattr(instance,variable_name)[index].fixed is True:
    395                               instance_fixed_count += 1
    396                         if ((instance_fixed_count > 0) and (instance_fixed_count < len(tree_node._scenarios))):
    397                            raise RuntimeError, "Variable="+variable_name+str(index)+" is fixed in "+str(instance_fixed_count)+" scenarios, but less than the number of scenarios at tree node="+tree_node._name
    398 
    399                         if instance_fixed_count == 0:
    400 
    401                            if isinstance(variable_type, IntegerSet) or isinstance(variable_type, BooleanSet):                           
    402                               node_min = self.Int_If_Close_Enough(ph, value(tree_node._minimums[variable_name][index]))
    403                               node_max = self.Int_If_Close_Enough(ph, value(tree_node._maximums[variable_name][index]))
    404 
    405                               # update convergence prior to checking for fixing.
    406                               self._int_convergence_tracking(ph, tree_node, variable_name, index, node_min, node_max)
    407                               attrvariable = ph._model_instance.active_components(Var)[variable_name][index]
    408                               if hasattr(attrvariable, 'Iter0FixIfConvergedAtLB'):
    409                                  lb = getattr(attrvariable, 'Iter0FixIfConvergedAtLB')
    410                               else:
    411                                  lb = self.Iter0FixIfConvergedAtLB
    412                               if hasattr(attrvariable, 'Iter0FixIfConvergedatUB'):
    413                                  ub = getattr(attrvariable, 'Iter0FixIfConvergedAtUB')
    414                               else:
    415                                  ub = self.Iter0FixIfConvergedAtUB
    416                               if hasattr(attrvariable, 'Iter0FixIfConvergedAtNB'):
    417                                  nb = getattr(attrvariable, 'Iter0FixIfConvergedAtNB')
    418                               else:
    419                                  nb = self.Iter0FixIfConvergedAtNB
    420                               if self._should_fix_discrete_due_to_conv(tree_node, variable, index, lb, ub, nb):
    421                                  self._fix_var(ph, tree_node, variable, index, node_min)
    422                               elif self.W_hash_history_len > 0:   # if not fixed, then hash - no slamming at iteration 0
    423                                  self._w_hash_acct(ph, tree_node, variable_name, index) # obviously not checking for cycles at iteration 0!
    424 
     399                        if getattr(instance,variable_name)[index].fixed is True:
     400                           instance_fixed_count += 1
     401                     if ((instance_fixed_count > 0) and (instance_fixed_count < len(tree_node._scenarios))):
     402                        raise RuntimeError, "Variable="+variable_name+str(index)+" is fixed in "+str(instance_fixed_count)+" scenarios, but less than the number of scenarios at tree node="+tree_node._name
     403
     404                     if instance_fixed_count == 0:
     405
     406                        if isinstance(variable_type, IntegerSet) or isinstance(variable_type, BooleanSet):                           
     407                           node_min = self.Int_If_Close_Enough(ph, value(tree_node._minimums[variable_name][index]))
     408                           node_max = self.Int_If_Close_Enough(ph, value(tree_node._maximums[variable_name][index]))
     409
     410                           # update convergence prior to checking for fixing.
     411                           self._int_convergence_tracking(ph, tree_node, variable_name, index, node_min, node_max)
     412                           attrvariable = ph._model_instance.active_components(Var)[variable_name][index]
     413                           if hasattr(attrvariable, 'Iter0FixIfConvergedAtLB'):
     414                              lb = getattr(attrvariable, 'Iter0FixIfConvergedAtLB')
    425415                           else:
     416                              lb = self.Iter0FixIfConvergedAtLB
     417                           if hasattr(attrvariable, 'Iter0FixIfConvergedatUB'):
     418                              ub = getattr(attrvariable, 'Iter0FixIfConvergedAtUB')
     419                           else:
     420                              ub = self.Iter0FixIfConvergedAtUB
     421                           if hasattr(attrvariable, 'Iter0FixIfConvergedAtNB'):
     422                              nb = getattr(attrvariable, 'Iter0FixIfConvergedAtNB')
     423                           else:
     424                              nb = self.Iter0FixIfConvergedAtNB
     425                           if self._should_fix_discrete_due_to_conv(tree_node, variable, index, lb, ub, nb):
     426                              self._fix_var(ph, tree_node, variable, index, node_min)
     427                           elif self.W_hash_history_len > 0:   # if not fixed, then hash - no slamming at iteration 0
     428                              self._w_hash_acct(ph, tree_node, variable_name, index) # obviously not checking for cycles at iteration 0!
     429
     430                        else:
    426431                             
    427                               node_min = value(tree_node._minimums[variable_name][index])
    428                               node_max = value(tree_node._maximums[variable_name][index])
     432                           node_min = value(tree_node._minimums[variable_name][index])
     433                           node_max = value(tree_node._maximums[variable_name][index])
    429434                             
    430                               self._continuous_convergence_tracking(ph, tree_node, variable_name, index, node_min, node_max)
     435                           self._continuous_convergence_tracking(ph, tree_node, variable_name, index, node_min, node_max)
    431436                             
    432437# jpw: not sure if we care about cycle detection in continuous variables?
    433 #                              if self.W_hash_history_len > 0: 
    434 #                                 self._w_hash_acct(ph, tree_node, variable_name, index)
     438#                           if self.W_hash_history_len > 0: 
     439#                              self._w_hash_acct(ph, tree_node, variable_name, index)
    435440                             
    436441
     
    455460   def post_iteration_k_solves(self, ph):
    456461
    457       for stage in ph._scenario_tree._stages:
     462      for stage in ph._scenario_tree._stages[:-1]:
    458463         
    459          if stage != ph._scenario_tree._stages[-1]: # no blending over the final stage
    460            
    461             for tree_node in stage._tree_nodes:
    462 
    463                for (variable, index_template, variable_indices) in stage._variables:
    464 
    465                   variable_name = variable.name
    466                   variable_type = variable.domain
    467 
    468                   for index in variable_indices:
    469 
    470                      # determine if this index is used - otherwise, don't waste time
    471                      # fixing and cycle checking. for one, the code will crash :-) with
    472                      # None values during the cycle checking computation!
    473 
    474                      is_used = True # until proven otherwise                     
     464         for tree_node in stage._tree_nodes:
     465
     466            for (variable, index_template) in stage._variables:
     467
     468               variable_name = variable.name
     469               variable_type = variable.domain
     470
     471               variable_indices = tree_node._variable_indices[variable_name]
     472
     473               for index in variable_indices:
     474
     475                  # determine if this index is used - otherwise, don't waste time
     476                  # fixing and cycle checking. for one, the code will crash :-) with
     477                  # None values during the cycle checking computation!
     478
     479                  is_used = True # until proven otherwise                     
     480                  for scenario in tree_node._scenarios:
     481                     instance = ph._instances[scenario._name]
     482                     if getattr(instance,variable_name)[index].status == VarStatus.unused:
     483                        is_used = False
     484
     485                  if is_used is True:                       
     486
     487                     # determine if the variable is already fixed.
     488                     instance_fixed_count = 0
    475489                     for scenario in tree_node._scenarios:
    476490                        instance = ph._instances[scenario._name]
    477                         if getattr(instance,variable_name)[index].status == VarStatus.unused:
    478                            is_used = False
    479 
    480                      if is_used is True:                       
    481 
    482                         # determine if the variable is already fixed.
    483                         instance_fixed_count = 0
    484                         for scenario in tree_node._scenarios:
    485                            instance = ph._instances[scenario._name]
    486                            if getattr(instance,variable_name)[index].fixed is True:
    487                               instance_fixed_count += 1
    488                         if ((instance_fixed_count > 0) and (instance_fixed_count < len(tree_node._scenarios))):
    489                            raise RuntimeError, "Variable="+variable_name+str(index)+" is fixed in "+str(instance_fixed_count)+" scenarios, but less than the number of scenarios at tree node="+tree_node._name
    490 
    491                         if instance_fixed_count == 0:
    492 
    493                            if isinstance(variable_type, IntegerSet) or isinstance(variable_type, BooleanSet):                           
    494                               node_min = self.Int_If_Close_Enough(ph, value(tree_node._minimums[variable_name][index]))
    495                               node_max = self.Int_If_Close_Enough(ph, value(tree_node._maximums[variable_name][index]))
    496 
    497                               # update convergence prior to checking for fixing.
    498                               self._int_convergence_tracking(ph, tree_node, variable_name, index, node_min, node_max)
    499 
    500                               # now check on permissions to converge to various placed (e.g., lb is lb permission)
    501                               attrvariable = ph._model_instance.active_components(Var)[variable_name][index]
    502                               if hasattr(attrvariable, 'FixWhenItersConvergedAtLB'):
    503                                  lb = getattr(attrvariable, 'FixWhenItersConvergedAtLB')
    504                               else:
    505                                  lb = self.FixWhenItersConvergedAtLB
    506                               if hasattr(attrvariable, 'FixWhenItersConvergedAtUB'):
    507                                  ub = getattr(attrvariable, 'FixWhenItersConvergedAtUB')
    508                               else:
    509                                  ub = self.FixWhenItersConvergedAtUB
    510                               if hasattr(attrvariable, 'FixWhenItersConvergedAtNB'):
    511                                  nb = getattr(attrvariable, 'FixWhenItersConvergedAtNB')
    512                               else:
    513                                  nb = self.FixWhenItersConvergedAtNB
     491                        if getattr(instance,variable_name)[index].fixed is True:
     492                           instance_fixed_count += 1
     493                     if ((instance_fixed_count > 0) and (instance_fixed_count < len(tree_node._scenarios))):
     494                        raise RuntimeError, "Variable="+variable_name+str(index)+" is fixed in "+str(instance_fixed_count)+" scenarios, but less than the number of scenarios at tree node="+tree_node._name
     495
     496                     if instance_fixed_count == 0:
     497
     498                        if isinstance(variable_type, IntegerSet) or isinstance(variable_type, BooleanSet):                           
     499                           node_min = self.Int_If_Close_Enough(ph, value(tree_node._minimums[variable_name][index]))
     500                           node_max = self.Int_If_Close_Enough(ph, value(tree_node._maximums[variable_name][index]))
     501
     502                           # update convergence prior to checking for fixing.
     503                           self._int_convergence_tracking(ph, tree_node, variable_name, index, node_min, node_max)
     504
     505                           # now check on permissions to converge to various placed (e.g., lb is lb permission)
     506                           attrvariable = ph._model_instance.active_components(Var)[variable_name][index]
     507                           if hasattr(attrvariable, 'FixWhenItersConvergedAtLB'):
     508                              lb = getattr(attrvariable, 'FixWhenItersConvergedAtLB')
     509                           else:
     510                              lb = self.FixWhenItersConvergedAtLB
     511                           if hasattr(attrvariable, 'FixWhenItersConvergedAtUB'):
     512                              ub = getattr(attrvariable, 'FixWhenItersConvergedAtUB')
     513                           else:
     514                              ub = self.FixWhenItersConvergedAtUB
     515                           if hasattr(attrvariable, 'FixWhenItersConvergedAtNB'):
     516                              nb = getattr(attrvariable, 'FixWhenItersConvergedAtNB')
     517                           else:
     518                              nb = self.FixWhenItersConvergedAtNB
    514519                                 
    515                               if self._should_fix_discrete_due_to_conv(tree_node, variable, index, lb, ub, nb):
     520                           if self._should_fix_discrete_due_to_conv(tree_node, variable, index, lb, ub, nb):
    516521                                 
    517                                  self._fix_var(ph, tree_node, variable, index, node_min)
     522                              self._fix_var(ph, tree_node, variable, index, node_min)
    518523                                 
    519                               else: # if not fixed, then hash and slam as necessary.
     524                           else: # if not fixed, then hash and slam as necessary.
    520525                                 
    521                                  if self.W_hash_history_len > 0:   
    522                                     self._w_hash_acct(ph, tree_node, variable_name, index)
    523                                     computed_cycle_length,msg = self.hash_hit_len(ph, tree_node, variable_name, index, False)
    524                                     if (computed_cycle_length >= self.CycleLengthSlamThreshold) and ((ph._current_iteration - self._last_slam_iter) > self.PH_Iters_Between_Cycle_Slams):
    525                                        # TBD: we may not want to slam immediately - it may disappear on it's own after a few iterations, depending on what other variables do.
    526                                        # note: we are *not* slamming the offending variable, but a selected variable
    527                                        if index is None:
    528                                           print "Cycle issue detected with variable="+variable_name
    529                                        else:
    530                                           print "Cycle issue detected with variable="+variable_name+"["+str(index)+"]"
    531                                        print msg
    532                                        print "Cycle length exceeds iteration slamming threshold="+str(self.CycleLengthSlamThreshold)+"; choosing a variable to slam"
    533                                        self._pick_one_and_slam_it(ph)
    534                                     elif (computed_cycle_length > 1) and (computed_cycle_length < self.CycleLengthSlamThreshold):
    535                                        # there was a (potential) cycle, but the slam threshold wasn't reached.
    536                                        if self.ReportPotentialCycles is True:
    537                                           if index is None:
    538                                              print "Cycle issue detected with variable="+variable_name
    539                                           else:                                         
    540                                              print "Cycle issue detected with variable="+variable_name+"["+str(index)+"]"                                         
    541                                           print msg
    542                                           print "Taking no action to break cycle - length="+str(computed_cycle_length)+" does not exceed slam threshold="+str(self.CycleLengthSlamThreshold)
    543                                     elif (computed_cycle_length >= self.CycleLengthSlamThreshold) and ((ph._current_iteration - self._last_slam_iter) > self.PH_Iters_Between_Cycle_Slams):
    544                                        # we could have slammed, but we recently did and are taking a break to see if things settle down on their own.
     526                              if self.W_hash_history_len > 0:   
     527                                 self._w_hash_acct(ph, tree_node, variable_name, index)
     528                                 computed_cycle_length,msg = self.hash_hit_len(ph, tree_node, variable_name, index, False)
     529                                 if (computed_cycle_length >= self.CycleLengthSlamThreshold) and ((ph._current_iteration - self._last_slam_iter) > self.PH_Iters_Between_Cycle_Slams):
     530                                    # TBD: we may not want to slam immediately - it may disappear on it's own after a few iterations, depending on what other variables do.
     531                                    # note: we are *not* slamming the offending variable, but a selected variable
     532                                    if index is None:
     533                                       print "Cycle issue detected with variable="+variable_name
     534                                    else:
     535                                       print "Cycle issue detected with variable="+variable_name+"["+str(index)+"]"
     536                                    print msg
     537                                    print "Cycle length exceeds iteration slamming threshold="+str(self.CycleLengthSlamThreshold)+"; choosing a variable to slam"
     538                                    self._pick_one_and_slam_it(ph)
     539                                 elif (computed_cycle_length > 1) and (computed_cycle_length < self.CycleLengthSlamThreshold):
     540                                    # there was a (potential) cycle, but the slam threshold wasn't reached.
     541                                    if self.ReportPotentialCycles is True:
    545542                                       if index is None:
    546543                                          print "Cycle issue detected with variable="+variable_name
     
    548545                                          print "Cycle issue detected with variable="+variable_name+"["+str(index)+"]"                                         
    549546                                       print msg
    550                                        print "Taking no action to break cycle - length="+str(computed_cycle_length)+" does exceed slam threshold="+str(self.CycleLengthSlamThreshold)+ \
    551                                              ", but another variable was slammed within the past "+str(self.PH_Iters_Between_Cycle_Slams)+" iterations"
    552                            else:
    553 
    554                               # obviously don't round in the continuous case.
    555                               node_min = value(tree_node._minimums[variable_name][index])
    556                               node_max = value(tree_node._maximums[variable_name][index])
    557 
    558                               # update convergence prior to checking for fixing.
    559                               self._continuous_convergence_tracking(ph, tree_node, variable_name, index, node_min, node_max)
    560 
    561                               if self._should_fix_continuous_due_to_conv(tree_node, variable, index):
    562                                  # fixing to max value for safety (could only be an issue due to tolerances).
    563                                  self._fix_var(ph, tree_node, variable, index, node_max)
    564                                  # note: we currently don't clam continuous variables!
     547                                       print "Taking no action to break cycle - length="+str(computed_cycle_length)+" does not exceed slam threshold="+str(self.CycleLengthSlamThreshold)
     548                                 elif (computed_cycle_length >= self.CycleLengthSlamThreshold) and ((ph._current_iteration - self._last_slam_iter) > self.PH_Iters_Between_Cycle_Slams):
     549                                    # we could have slammed, but we recently did and are taking a break to see if things settle down on their own.
     550                                    if index is None:
     551                                       print "Cycle issue detected with variable="+variable_name
     552                                    else:                                         
     553                                       print "Cycle issue detected with variable="+variable_name+"["+str(index)+"]"                                         
     554                                    print msg
     555                                    print "Taking no action to break cycle - length="+str(computed_cycle_length)+" does exceed slam threshold="+str(self.CycleLengthSlamThreshold)+ \
     556                                          ", but another variable was slammed within the past "+str(self.PH_Iters_Between_Cycle_Slams)+" iterations"
     557                        else:
     558
     559                           # obviously don't round in the continuous case.
     560                           node_min = value(tree_node._minimums[variable_name][index])
     561                           node_max = value(tree_node._maximums[variable_name][index])
     562
     563                           # update convergence prior to checking for fixing.
     564                           self._continuous_convergence_tracking(ph, tree_node, variable_name, index, node_min, node_max)
     565
     566                           if self._should_fix_continuous_due_to_conv(tree_node, variable, index):
     567                              # fixing to max value for safety (could only be an issue due to tolerances).
     568                              self._fix_var(ph, tree_node, variable, index, node_max)
     569                              # note: we currently don't clam continuous variables!
    565570
    566571      # TBD: the 1 might need to be parameterized - TBD - the 1 should be the PH ITERATIONS BETWEEN CYCLE SLAMS
Note: See TracChangeset for help on using the changeset viewer.