<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 PathMovers
This notebook is an introduction to handling PathMover and MoveChange instances. It mostly covers the questions on 
1. how to check if a certain mover was part of a change. 
2. What are the possible changes a mover can generate.
3. ...
Load OPENPATHSAMPLING
Step1:  Let's open a file that contains a mover and lot's of changes
Step2:  A Simulator creates simulation steps. So we load a single step.
Step3:  Each step posesses a movechange which we will get
Step4:  And the mover that was used to generate this change
Step5:  Let's first check if the pmc was really created by the pathmover
Step6:  This should be obvious, but under the hood there is some more fancy stuff happening. The in keyword with pathmovers and changes actually works on trees of movers. These trees are used to represent a specific order or movers being called and these trees are unique for changes. This means there is a way to label each possible pmc that can be generated by a mover. This possible only refers to choices in actual movers. 
This is due to special PathMovers that can pick one or more movers among a set of possibilities. The simplest of these is the OneWayShooter which choses equal between forward and backward shooting. So a OneWayShooter has two possible outcomes. Let's look at these
Step7:  The property enum will evaluate the (potentially VERY large) number of all possible changes
Step8:  which is two in our case. Another example
Step9:  Now we have 4 choices as expected 2 for the RandomChoice * 2 for the OneWayShooter. Although effectively there are only 2 distinct ones. Why is that? The problem in ow_mover_2 is that we defined two separate instances of the OneWayShootingMover and in general different instances are considered different. In this case it might even be possible to check for equality, but we decided to leave this to the user. If you create two instances we assume there is a reason. Maybe just calling the differently (although it makes no sense for the monte carlo scheme).
The next example uses the same mover twice
Step10:  And we get only two distinct movers as we wanted to.
Let's look at the real mover
Step11:  This one has 31. Let's if we can reconstruct that number. The pathmover first choses between 4 different move types. 
Shooting, with one shooter for each of the 6 ensembles and two directions (forward and backward) are 6 * 2 = 12 in total. 
Reversal moves for each ensemble = 6
ReplicaExchange moves between all neighboring ensembles [1,2], [2,3], ..., [5,6]  = 5
And a Minus Move that consists of choices for picking the first or final inner trajectory from the minus ensemble and 2 choices for extending forward or backward. Here is important that we use a ConditionalSequentialMover that might call only some of its inner sequence. To be precise it could call only the subtrajectoryselection (2 choices), the selection and the replicaexchange (also 2 choices) or all three movers (4 choices). This will give a total of 8 possibilities for the minus move
In total we have 12 + 6 + 5 + 8 = 31 as enum predicted. 
Note that this is only the maximal possible number of moves and not the real accessible moves. It could be that in a conditional mover some moves are always run in sequence. E.g. in our case the subtrajectoryselection should always work and hence the first 2 choices of the minus mover will never happen. It will always try a swap (which can fail if by chance the trajectory in the innermost interface crosses away from the state). So the effective number of choices will be 29 and not 31.
Let's do some more testing
First let's check if the keys we get for all changes in the storage are actually contained in the mover or put more simple the mover could have generated these changes. This should be true for all changes that originate from our mover.
Step12:  What happened here? Why are only some changes in the storage made by our mover? Shouldn't all of them (except the 2 in the beginning) be the result of our pathmover? Yes and no. They are, but we are checking if the total change is made from a mover and we are also loading all subchanges subchanges.
Step13:  We will now cache all changes to see if the next steps are really fast or not.
Step14:  Now
Step15:  While this works it would be better to rely on the steps in the simulation and not on the changes itself. We might store some other changes for whatever reason, but we only want the ones that are associated with a MC step. The steps in the storage do exactly that and point to the changes that are used to generate the next sampleset that we wanted.
Step16:  We exclude the first step since it is the initialization which is not generated by our pathmover.
Let's now check which and how often one the 31 possible steps was called.
Step17:  We see that especially minus moves are underrepresented. This is due to the standard weighting of the minus move which runs minus moves much less frequent than other moves. 
Let's just do that again and pick a random chance among all the possibilities. This is done only from the possibilities without regard for acceptance etc. We only assume that all possible choices for a single mover are equally likely.
Step18:  We realize that a specific shooter is called less likely than other movers
not sure if this is helpful. We could have additional weighting factors for the randomchoice, but each mover would have to know about it. Problem is that in most movers these weighting probabilities are dependent on the context and hence make no sense here!
One solution is to let a mover be able to give a formal expression for each possible choice, since each mover will only depend on 
1. results of accepted/rejected of submoves and ultimately if certain samples are accepted which in turn depends on .valid and .proposal_probability
2. some random numbers (choices)
So the SampleGeneratingMovers will return something like [acc(samp[]), acc(samp[])]
while other movers will return something that depends on the submovers, it could just be a list of products.
Each mover has an expression for it being accepted
Each mover has an expression for picking a specific choice
Finally .random() will use the product of all probabilities and a specific value for accepting
Step19:  Get the (unique) location of the randomchoicemover. You can search for Mover classes, Mover instances or by the .name property of a mover which is a string.
Step20:  the location object is effectively a tree mover instances described by nested tuples. For convenience it is wrapped to make searching easier and format the output.
Step21:  In most cases you can use python tuples instead of TupleTree. The structure of a tree of tuples looks as follows
Step22:  Instead of checking for a pmc directly we can also check for the tuple tree representation
Step23:  Now get the mover at the loc_rc location (which is of course a RandomChoiceMover).
Step24:  These are the weights by mover
Step25:  So a shooting move is about 30x more likely than a minus move.
Finally do some more checks
Step26:  Note, that certain trees of movers can be arranged differently in a tree, but result in the same possible steps. Like Sequential(Forward, Sequential([Forward, Forward])) and Sequential([Forward, Forward, Forward]). We can regard this as being associative (placing arbitrary brackets) but we will not check for these types of equality. We assume that you arrange steps in a certain way for a reason and that your grouping of sequenced will reflect a certain logical idea. On the other hand are your locators / keys dependend on that choice!
What else can we learn or ask for?
We can check if a particular mover was run. Let's pick the very last MinusExtensionDirectionChoser. To check for it we can either just check for the instance in a pmc
Step27:  the expression locator in pmc checks for the overall appearance of a specific mover somewhere dependent of the location. While using mover_instance in pmc check for the appearance of that mover instance independent of the location. If a particular mover_instance appears only once then both methods are equal.
Step28:  In this case the MinusExtentionDirectionChoose was not called in this particular pmc.
Remember the following things about moves and changes
movers and changes represent a tree like structure of movers.
a change is a concrete realization of moves (and hence movers) while a mover itself has a certain degree of choice. 
Some nodes in a mover can have different outcomes, like a RandomChoiceMover will pick one of several choices. The change it generates will have only the chosen one, while the RandomChoiceMover itself has all possibilities.
A concrete tree like structure (without choice) of movers can be represented by a tree of tuples. Reading the tuple-tree from left to right the opening bracket '(' means go down a level (a new child) while the closing bracket means trael up a level ')' go to the parent.
The smallest unique tupletree for a pmc can be accessed using .unique
Step29:  will be interpreted as
Step30:  Some speed tests
<ASSISTANT_TASK:>
Python Code:
import openpathsampling as p
Explanation: PathMovers
This notebook is an introduction to handling PathMover and MoveChange instances. It mostly covers the questions on 
1. how to check if a certain mover was part of a change. 
2. What are the possible changes a mover can generate.
3. ...
Load OPENPATHSAMPLING
End of explanation
st = p.storage.Storage('_toy_retis.nc', mode='r')
Explanation: Let's open a file that contains a mover and lot's of changes
End of explanation
mc = st.steps[3]
print mc
Explanation: A Simulator creates simulation steps. So we load a single step.
End of explanation
pmc = mc.change
print pmc
Explanation: Each step posesses a movechange which we will get
End of explanation
pm = pmc.mover
print pm.treeprint()
Explanation: And the mover that was used to generate this change
End of explanation
pmc in pm
Explanation: Let's first check if the pmc was really created by the pathmover
End of explanation
ow_mover = p.OneWayShootingMover([], []) # we use dummy arguments since are not going to use it
Explanation: This should be obvious, but under the hood there is some more fancy stuff happening. The in keyword with pathmovers and changes actually works on trees of movers. These trees are used to represent a specific order or movers being called and these trees are unique for changes. This means there is a way to label each possible pmc that can be generated by a mover. This possible only refers to choices in actual movers. 
This is due to special PathMovers that can pick one or more movers among a set of possibilities. The simplest of these is the OneWayShooter which choses equal between forward and backward shooting. So a OneWayShooter has two possible outcomes. Let's look at these
End of explanation
list(ow_mover.enum)
Explanation: The property enum will evaluate the (potentially VERY large) number of all possible changes
End of explanation
ow_mover_2 = p.RandomChoiceMover([
        p.OneWayShootingMover([], []),
        p.OneWayShootingMover([], [])
        ])
list(ow_mover_2.enum)
Explanation: which is two in our case. Another example
End of explanation
ow_mover_3 = p.RandomChoiceMover([
        ow_mover,
        ow_mover
        ])
list(ow_mover_3.enum)
Explanation: Now we have 4 choices as expected 2 for the RandomChoice * 2 for the OneWayShooter. Although effectively there are only 2 distinct ones. Why is that? The problem in ow_mover_2 is that we defined two separate instances of the OneWayShootingMover and in general different instances are considered different. In this case it might even be possible to check for equality, but we decided to leave this to the user. If you create two instances we assume there is a reason. Maybe just calling the differently (although it makes no sense for the monte carlo scheme).
The next example uses the same mover twice
End of explanation
all_changes = list(pm.enum)
print len(all_changes)
Explanation: And we get only two distinct movers as we wanted to.
Let's look at the real mover
End of explanation
print [pc in pm for pc in st.movechanges[0:20]]
Explanation: This one has 31. Let's if we can reconstruct that number. The pathmover first choses between 4 different move types. 
Shooting, with one shooter for each of the 6 ensembles and two directions (forward and backward) are 6 * 2 = 12 in total. 
Reversal moves for each ensemble = 6
ReplicaExchange moves between all neighboring ensembles [1,2], [2,3], ..., [5,6]  = 5
And a Minus Move that consists of choices for picking the first or final inner trajectory from the minus ensemble and 2 choices for extending forward or backward. Here is important that we use a ConditionalSequentialMover that might call only some of its inner sequence. To be precise it could call only the subtrajectoryselection (2 choices), the selection and the replicaexchange (also 2 choices) or all three movers (4 choices). This will give a total of 8 possibilities for the minus move
In total we have 12 + 6 + 5 + 8 = 31 as enum predicted. 
Note that this is only the maximal possible number of moves and not the real accessible moves. It could be that in a conditional mover some moves are always run in sequence. E.g. in our case the subtrajectoryselection should always work and hence the first 2 choices of the minus mover will never happen. It will always try a swap (which can fail if by chance the trajectory in the innermost interface crosses away from the state). So the effective number of choices will be 29 and not 31.
Let's do some more testing
First let's check if the keys we get for all changes in the storage are actually contained in the mover or put more simple the mover could have generated these changes. This should be true for all changes that originate from our mover.
End of explanation
print st.movechanges[2]
print st.movechanges[2] in pm
print
print st.movechanges[5]
print st.movechanges[5] in pm
Explanation: What happened here? Why are only some changes in the storage made by our mover? Shouldn't all of them (except the 2 in the beginning) be the result of our pathmover? Yes and no. They are, but we are checking if the total change is made from a mover and we are also loading all subchanges subchanges.
End of explanation
_ = list(st.movechanges)
_ = list(st.steps)
Explanation: We will now cache all changes to see if the next steps are really fast or not.
End of explanation
real_changes = filter(lambda x : x in pm, st.movechanges)
print len(real_changes), 'of', len(st.movechanges)
Explanation: Now: How do we get all the changes that are really important? One way is to scan all changes and use only the ones from the mover
End of explanation
step_changes = [step.change for step in st.steps[1:]]
print len(step_changes)
Explanation: While this works it would be better to rely on the steps in the simulation and not on the changes itself. We might store some other changes for whatever reason, but we only want the ones that are associated with a MC step. The steps in the storage do exactly that and point to the changes that are used to generate the next sampleset that we wanted.
End of explanation
import collections
counter = collections.defaultdict(lambda: 0)
for ch in step_changes:
    counter[ch.unique] += 1
s = '%d of %d different changes run' % (len(counter), len(list(pm.enum)))
print s, '\n', '-' * len(s), '\n'
for y in sorted(counter.items(), key=lambda x : -x[1]):
    print
    print y[1], 'x'
    print y[0].treeprint()
Explanation: We exclude the first step since it is the initialization which is not generated by our pathmover.
Let's now check which and how often one the 31 possible steps was called.
End of explanation
pmc_list = [pm.random() for x in xrange(10000)]
counter2 = collections.defaultdict(lambda: 0)
for ch in pmc_list:
    counter2[ch] += 1
s = '%d of %d different changes run' % (len(counter2), len(list(pm.enum)))
print s, '\n', '-' * len(s), '\n'
for y in sorted(counter2.items(), key=lambda x : -x[1]):
    print (100.0 * y[1]) / len(pmc_list), '%', repr(y[0])
Explanation: We see that especially minus moves are underrepresented. This is due to the standard weighting of the minus move which runs minus moves much less frequent than other moves. 
Let's just do that again and pick a random chance among all the possibilities. This is done only from the possibilities without regard for acceptance etc. We only assume that all possible choices for a single mover are equally likely.
End of explanation
print pm.treeprint()
Explanation: We realize that a specific shooter is called less likely than other movers
not sure if this is helpful. We could have additional weighting factors for the randomchoice, but each mover would have to know about it. Problem is that in most movers these weighting probabilities are dependent on the context and hence make no sense here!
One solution is to let a mover be able to give a formal expression for each possible choice, since each mover will only depend on 
1. results of accepted/rejected of submoves and ultimately if certain samples are accepted which in turn depends on .valid and .proposal_probability
2. some random numbers (choices)
So the SampleGeneratingMovers will return something like [acc(samp[]), acc(samp[])]
while other movers will return something that depends on the submovers, it could just be a list of products.
Each mover has an expression for it being accepted
Each mover has an expression for picking a specific choice
Finally .random() will use the product of all probabilities and a specific value for accepting
End of explanation
loc_rc = pm.locate('RootMover')
print loc_rc
Explanation: Get the (unique) location of the randomchoicemover. You can search for Mover classes, Mover instances or by the .name property of a mover which is a string.
End of explanation
print type(loc_rc)
print isinstance(loc_rc, tuple)
print repr(loc_rc)
print str(loc_rc)
Explanation: the location object is effectively a tree mover instances described by nested tuples. For convenience it is wrapped to make searching easier and format the output.
End of explanation
print pmc
print repr(pmc.unique)
Explanation: In most cases you can use python tuples instead of TupleTree. The structure of a tree of tuples looks as follows:
Each node has the following structure
(head, (child1, ), (child2, ), ...)
where head is a the node content (almost always a Mover instance) and child is again a node. It is important to note that a tuple tree does not support choices of different children like a real pathmover can. Therefore we could generate a tree tuple of the content of a pathmover but it will not reflect the possible choices in it. To get the associated tree tuple of a change (which has no choice in it) we can use .unique.
End of explanation
print pmc
print pmc in pm # check if pmc could have been generated by pm
print pmc.unique in pm # check if the tuple tree representation could have been generated by pm
Explanation: Instead of checking for a pmc directly we can also check for the tuple tree representation
End of explanation
rc = pm[loc_rc]
Explanation: Now get the mover at the loc_rc location (which is of course a RandomChoiceMover).
End of explanation
dict(zip(rc.movers, rc.weights))
Explanation: These are the weights by mover
End of explanation
print rc in pmc # check if the RandomChoiceMover was called in pmc
print pc in pm # check if the pathmover has a RandomChoiceMover at that position in the tree
Explanation: So a shooting move is about 30x more likely than a minus move.
Finally do some more checks
End of explanation
loc_medc = pm.locate('MinusExtensionDirectionChooser')
medc = pm[loc_medc]
print medc
%%time
for pc in step_changes:
    if medc in pc:
        print repr(pc)
        print pc.unique
        print
Explanation: Note, that certain trees of movers can be arranged differently in a tree, but result in the same possible steps. Like Sequential(Forward, Sequential([Forward, Forward])) and Sequential([Forward, Forward, Forward]). We can regard this as being associative (placing arbitrary brackets) but we will not check for these types of equality. We assume that you arrange steps in a certain way for a reason and that your grouping of sequenced will reflect a certain logical idea. On the other hand are your locators / keys dependend on that choice!
What else can we learn or ask for?
We can check if a particular mover was run. Let's pick the very last MinusExtensionDirectionChoser. To check for it we can either just check for the instance in a pmc
End of explanation
print pmc
print loc_medc in pm
print medc in pm
print loc_medc in pmc
print medc in pmc
Explanation: the expression locator in pmc checks for the overall appearance of a specific mover somewhere dependent of the location. While using mover_instance in pmc check for the appearance of that mover instance independent of the location. If a particular mover_instance appears only once then both methods are equal.
End of explanation
first_minus_change = filter(lambda x : p.MinusMover in x, step_changes)[0]
print first_minus_change.unique
Explanation: In this case the MinusExtentionDirectionChoose was not called in this particular pmc.
Remember the following things about moves and changes
movers and changes represent a tree like structure of movers.
a change is a concrete realization of moves (and hence movers) while a mover itself has a certain degree of choice. 
Some nodes in a mover can have different outcomes, like a RandomChoiceMover will pick one of several choices. The change it generates will have only the chosen one, while the RandomChoiceMover itself has all possibilities.
A concrete tree like structure (without choice) of movers can be represented by a tree of tuples. Reading the tuple-tree from left to right the opening bracket '(' means go down a level (a new child) while the closing bracket means trael up a level ')' go to the parent.
The smallest unique tupletree for a pmc can be accessed using .unique
End of explanation
print first_minus_change.unique
pm.map_tree(lambda x : len(x.name))
Explanation: will be interpreted as
End of explanation
%%timeit
pm.enum
%%timeit
pmc in pm
%%timeit
[p in pmc for p in pm.enum]
%%timeit
pm in pm
%%timeit
pmc.unique in pm
Explanation: Some speed tests
End of explanation
<END_TASK>