<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 ClinVar documentation
Submission guidelines can be found here, in particular note the requirement
Step1:  Dataframe
Load measures into a dataframe where columns are the properties.
Step2:  XML
Probably not all elements from the raw xml are captured in the object, can we get find anything there?
<ASSISTANT_TASK:>
Python Code:
complex_xml = os.path.join(PROJECT_ROOT, 'complex-events.xml.gz')
# get just "complex events"
# Q: what's complex? -- complex == no full coordinates
def complex_measures(x):
    if x.measure:
        return (
            # smattering of all non SNV variants
            (x.measure.variant_type.lower() not in {'single nucleotide variant'} and np.random.random() < 0.01)
            # be sure to get the rare ones
            or (x.measure.variant_type.lower() in {'tandem duplication', 'fusion', 'complex', 'translocation', 'inversion'})
        )
    return False
filter_xml(
    input_xml=clinvar_path,
    output_xml=complex_xml,
    filter_fct=complex_measures,
)
dataset = ClinVarDataset(complex_xml)
Explanation: ClinVar documentation
Submission guidelines can be found here, in particular note the requirement:
a valid description of the variant, one of:
   * an HGVS expression
   * chromosome coordinates and change
   * cytogenetic description
Also found the xsd for ClinVar XML submissions, which includes all possible measure types:
xml
&lt;xs:simpleType name="Measuretype"&gt;
    &lt;xs:restriction base="xs:string"&gt;
        &lt;xs:enumeration value="Gene"/&gt;
        &lt;xs:enumeration value="Variation"/&gt;
        &lt;xs:enumeration value="Insertion"/&gt;
        &lt;xs:enumeration value="Mobile element insertion"/&gt;
        &lt;xs:enumeration value="Novel sequence insertion"/&gt;
        &lt;xs:enumeration value="Microsatellite"/&gt;
        &lt;xs:enumeration value="Deletion"/&gt;
        &lt;xs:enumeration value="single nucleotide variant"/&gt;
        &lt;xs:enumeration value="Multiple nucleotide variation"/&gt;
        &lt;xs:enumeration value="Indel"/&gt;
        &lt;xs:enumeration value="Duplication"/&gt;
        &lt;xs:enumeration value="Tandem duplication"/&gt;
        &lt;xs:enumeration value="copy number loss"/&gt;
        &lt;xs:enumeration value="copy number gain"/&gt;
        &lt;xs:enumeration value="protein only"/&gt;
        &lt;xs:enumeration value="Inversion"/&gt;
        &lt;xs:enumeration value="Translocation"/&gt;
        &lt;xs:enumeration value="Interchromosomal breakpoint"/&gt;
        &lt;xs:enumeration value="Intrachromosomal breakpoint"/&gt;
        &lt;xs:enumeration value="Complex"/&gt;
    &lt;/xs:restriction&gt;
&lt;/xs:simpleType&gt;
Compare measure types we've actually found in the data, here.
Filter
Filter full dataset to get just a manageable (hopefully representative) sample of records with measures representing complex events
End of explanation
def get_measures(dataset):
    for r in dataset:
        if r.measure:
            yield r.measure
for m in get_measures(dataset):
    break
    
dir(m)
# just all the properties
props = [
 'all_names',
 'clinvar_record',
#  'explicit_insertion_length',
#  'has_complete_coordinates',
 'hgnc_ids',
 'hgvs',
 'is_repeat_expansion_variant',
#  'measure_xml',
 'microsatellite_category',
 'nsv_id',
 'preferred_gene_symbols',
#  'preferred_name',
 'preferred_or_other_name',
#  'pubmed_refs',
 'rs_id',
#  'sequence_location_helper',
#  'toplevel_refseq_hgvs',
 'variant_type',
 'chr',
 'vcf_alt',
#  'vcf_full_coords',
 'vcf_pos',
 'vcf_ref'
]
# replaces empty list with None
measures = [[getattr(v, p) if getattr(v, p) != [] else None for p in props] for v in get_measures(dataset)]
df = pd.DataFrame(measures, columns=props)
df.count()
set(df['variant_type'])
df[df.variant_type == 'Translocation']
Explanation: Dataframe
Load measures into a dataframe where columns are the properties.
End of explanation
def get_measure_xml_for_rcv(dataset, rcv):
    for r in dataset:
        if r.accession == rcv:
            return r.measure.measure_xml
# pretty print xml
def pprint(x):
    print(ElementTree.tostring(x, encoding='unicode'))
def print_measure_xml_for_rcv(dataset, rcv):
    x = get_measure_xml_for_rcv(dataset, rcv)
    pprint(x)
xml = get_measure_xml_for_rcv(dataset, 'RCV001372309')
pprint(xml)
xml2 = get_measure_xml_for_rcv(dataset, 'RCV001255994')
pprint(xml2)
Explanation: XML
Probably not all elements from the raw xml are captured in the object, can we get find anything there?
End of explanation
<END_TASK>