<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Catalog
Step1:  Setup OpenCGA Variables
Once we have defined a variable with the client configuration and credentials, we can access to all the methods defined for the client. These methods implement calls to query different data models in OpenCGA. 
Over the user case addressed in this notebook we will be performing queries to the users, projects, studies, samples, individuals and cohorts<br> OpenCGA data models.
Use Cases
In this seciton we are going to show how to work with some of the most common scenarios.<br>
- The user-cases addresed here constute a high-level introduction aimed to provide a basis for the user to make their own explorations. 
- The examples can be adapted to each individual user-case.
Exploring User Account
Step2:  Using the REST response API
Step3:  User Projects
Step4:  The fqn owner@project shows the owner of the project/s; this owner has granted permission to our user to the projects above.
User Studies
Step5:  Obtain the studies for the project_id defined above with the result_iterator() method
Step6:  For the rest of the notebook, we will use a specific study to query catalog information. The study is defined in the variable study_id.
Step7:  2. Checking Groups and Permissions
[NOTE]
Step8:  User Groups
If we want to check in which groups is our user included
Step9:  Independently of the groups defined for a study, our user always belongs to the group members, which is one of the default groups in OpenCGA.
User Permissions
We might be wondering which specific permissions our user has. We can check this using the client.acl() method (acl = access control list)
Step10:  Default Groups in OpenCGA
The default groups in OpenCGA are
Step11:  We can see that the number of samples in the study is given by #Num matches by using the parameter count=True.
Execise
Step12:  Individuals
Step13:  We might be interested in knowing when the individuals were added to OpenCGA, or the individuals sex. Since pyopencga 2.0.1.1 it is possible to export the results to a pandas dataframe object with the function to_data_frame()
Step14:  2. Exploring Files
Files in a study
We can start by exploring the number of files in the study, and retrieveing information about one file as an example of which kind of data is stored in the file data model of OpenCGA.
Step15:  File Specific Info
There is plenty of useful information contained in the file data model like the file format, the stats, size of the file. If we want to look for more concrete information about one specific file
Step16:  Files with a specific sample
We can also be interested in knowing the number of files for a specific sample
Step17:  3. Exploring Cohorts
One powerful feature of OpenCGA is the possibility of define cohorts that include individuals with common traits of interest, like a phenotype, nationality etc.
The cohorts are defined at the study level. OpenCGA creates a default cohort ALL, which includes all the individuals of the study.
We can explore which cohorts are defined in the  study by
<ASSISTANT_TASK:>
Python Code:
## Step 1. Import pyopencga dependecies
from pyopencga.opencga_config import ClientConfiguration # import configuration module
from pyopencga.opencga_client import OpencgaClient # import client module
from pprint import pprint
from IPython.display import JSON
import matplotlib.pyplot as plt
import seaborn as sns
## Step 2. OpenCGA host
host = 'https://ws.opencb.org/opencga-prod'
## Step 3. User credentials
user = 'demouser'
passwd = 'demouser' ## you can skip this, see below.
## Step 4. Create the ClientConfiguration dict
config_dict = {'rest': {
                       'host': host 
                    }
               }
## Step 5. Create the ClientConfiguration and OpenCGA client
config = ClientConfiguration(config_dict)
oc = OpencgaClient(config)
## Step 6. Login to OpenCGA using the OpenCGA client 
## Option1: Pass the credentials to the client
# (here we put only the user in order to be asked for the password interactively)
# oc.login(user)
## Option2: you can pass the user and passwd
oc.login(user, passwd)
print('Logged succesfuly to {}, your token is: {} well done!'.format(host, oc.token))
Explanation: Catalog: Overview
What is Catalog?
Catalog is the OpenCGA component that takes care of clinical data, metadata and permissions.
This notebook is intended to provide guidance for querying an OpenCGA server through pyopencga to explore:
- Studies which the user has access to
- Clinical data provided in the study (Samples, Individuals Genotypes etc.) 
- Other types of metadata, like permissions.
A good first step when start working with OpenCGA is to retrieve information about our user, which projects and studies are we allowed to see.<br>
It is also recommended to get a taste of the clinical data we are encountering in the study: How many samples and individuals does the study have? Is there any defined cohorts? Can we get some statistics about the genotypes of the samples in the Sudy?
For guidance on how to loggin and get started with opencga you can refer to : pyopencga_first_steps.ipynb
[NOTE] The server methods used by pyopencga client are defined in the following swagger URL:
- https://ws.opencb.org/opencga-prod/webservices/   
Setup the Client and Login into pyopencga
Configuration and Credentials 
Let's assume we already have pyopencga installed in our python setup (all the steps described on pyopencga_first_steps.ipynb).
You need to provide at least a host server URL in the standard configuration format for OpenCGA as a python dictionary or in a json file.
End of explanation
## Getting user information
## [NOTE] User needs the quey_id string directly --> (user)
#Print using the print_results() function:
user_info = oc.users.info(user)
user_info.print_results( title='User info with print_results() function:') # metadata=False
## Uncomment next line to display an interactive JSON viewer
# JSON(user_info.get_results())
Explanation: Setup OpenCGA Variables
Once we have defined a variable with the client configuration and credentials, we can access to all the methods defined for the client. These methods implement calls to query different data models in OpenCGA. 
Over the user case addressed in this notebook we will be performing queries to the users, projects, studies, samples, individuals and cohorts<br> OpenCGA data models.
Use Cases
In this seciton we are going to show how to work with some of the most common scenarios.<br>
- The user-cases addresed here constute a high-level introduction aimed to provide a basis for the user to make their own explorations. 
- The examples can be adapted to each individual user-case.
Exploring User Account: Permissios, Projects and Studies
In this use case we cover retrieving information for our user.
In OpenCGA, all the user permissions are established at a study level. One project contains at least one study, although it may contain several.
Full Qualified Name (fqn) of Studies
It is also very important to understand that in OpenCGA, the projects and studies have a full qualified name (fqn) with the format:<br>
[[owner]@[project]]:[study]
We cannot be sure if there might be other studies with the same name contained in other projects.<br> (E.g: the study platinium might be defined in two different projects: GRch37_project and GRch38_project)
Because of that that, it is recomended to use the fqn when referencing studies.
1. Exploring Projects and Studies with our user
Users: owner and members
Depending on the permissions granted, a user can be the owner of a study or just have access to some studies owned by other users.<br>We can retrieve information about our user and its permissions by:
- Using the print_results() function
End of explanation
# Using REST response API:
print("\n User info using REST response API: \n")
user_info = oc.users.info(user).get_result(0)
user_projects = user_info['projects']  # Define projects owned by our user
print('id:{}\taccount_type: {}\t projects_owned: {}'.format(user, user_info['account']['type'], len(user_projects)))
print('\nWe can appreciate that our user: {} has {} projects from its own: {}'.format(user, len(user_projects), user_projects))
Explanation: Using the REST response API
End of explanation
## Getting user projects
## [NOTE] Client specific methods have the query_id as a key:value (i.e (user=user_id)) 
projects_info = oc.projects.search()
projects_info.print_results(fields='id,name,organism.scientificName,organism.assembly,fqn', title='Projects our user ({}) has access to:'.format(user), metadata=False)
Explanation: User Projects:
Although an user doesn't own any project, it might has been granted access to projects created by other users. Let's see how to find this out.
We can list our user's projects using project client search() function.
End of explanation
# First we define one projectId
project_info = oc.projects.search().get_result(0)
project_id = project_info['id']
print('For this user-case, we can use project:{}'.format(project_id))
Explanation: The fqn owner@project shows the owner of the project/s; this owner has granted permission to our user to the projects above.
User Studies:
Let's see which studies do we have access within the project.
- Define a random project for the use case within the ones available for demouser.
End of explanation
studies = oc.studies.search(project_id)
## Print the studies using the result_iterator() method
print('Our user [{}] has access to 2 different studies within the [{}] project\n'.format(user, project_id))
for study in studies.result_iterator():
    print("project:{}\t study_id:{}\t study_fqn:{} ".format(project_id, study['id'], study['fqn']))
    
Explanation: Obtain the studies for the project_id defined above with the result_iterator() method:
End of explanation
# Define the study we are going to work with
study_info = oc.studies.search(project_id).get_result(0)
study_id = study_info['id']
study_fqn = study_info['fqn']
print("Let's use the study: [{}] with fqn: [{}]".format(study_id, study_fqn))
Explanation: For the rest of the notebook, we will use a specific study to query catalog information. The study is defined in the variable study_id.
End of explanation
# # Query to the study web service
# groups = oc.studies.groups(study_fqn)
# study_groups = []  # Define an empty list for the groups
# ## This will give us the whole list of groups existing in the studya
# for group in groups.result_iterator():
#     study_groups.append(group['id'])
#     print("group_id: {}".format(group['id']))
    
# print('\nThere are 3 groups in the study {}: {}'.format(study_fqn, study_groups))
Explanation: 2. Checking Groups and Permissions
[NOTE]: This can ONLY be done by an admin or the owner. If your user is not any of these, skipt this section.
Now we can assume that we want to check to which groups our user belongs to and which permisions pur user has been granted for the study (remember that all the permissions are established at the study level).
Groups in the Study
OpenCGA define the permissions (for both groups and users) at the Study level. The first step might be check which groups exist within the study.
End of explanation
# user_groups = [] # Define an empty list 
# ## This will give us only the groups our user belongs to
# for group in groups.result_iterator():
#     if user_id in group['userIds']:
#         user_groups.append(group['id'])
#         print("group_id: {}".format(group['id']))
        
# print('\nOur user {} belongs to group/s: {}'.format(user_id, user_groups))
Explanation: User Groups
If we want to check in which groups is our user included
End of explanation
# Permissions granted directly to user:
acls = oc.studies.acl(study_id, member=user_id).get_result(0)
print('The user',user_id,' has the following permissions:\n\n', acls[user_id])
    
Explanation: Independently of the groups defined for a study, our user always belongs to the group members, which is one of the default groups in OpenCGA.
User Permissions
We might be wondering which specific permissions our user has. We can check this using the client.acl() method (acl = access control list):
End of explanation
## Call to the sample web endpoint
samples = oc.samples.search(study=study_fqn, includeIndividual=True, count=True, limit = 5) ## other possible params, count=False, id='NA12880,NA12881'
samples.print_results(fields='id,creationDate,somatic,phenotypes.id,phenotypes.name,individualId', title='Info from 5 samples from study {}'.format(study_fqn))
## Uncomment next line to display an interactive JSON viewer
# JSON(samples.get_results())
Explanation: Default Groups in OpenCGA
The default groups in OpenCGA are: members and admins.
Intuitively, the group members is the basic group and has any default permissions. On the other hand, users in the group admins have permission to see and edit the study information.
For more information about user and group permissions, check the official OpenCGA documentation: Catalog and Security - Users and Permissions
Exploring Catalog Clinical Metadata
A genomic data analysis platform need to keep track of different resources such as:
Clinical Data: information about individuals, samples from those individuals etc.
Files Metadata: information about files contained in the platform, such as VCFs and BAMs.
OpenCGA Catalog is the component that assumes this role by storing this kind of information
1. Exploring Samples and Individuals
Once we know the studies our user has access to, we can explore the samples within the study.<br>
To fetch samples you need to use the sample client built in pyopencga. Remember that it is recomended to use the fqn when referencing studies.<br>
Samples:
Let's imagine we want to know how many samples are in the study stored in the study_fqn variable, and list information about the first two samples:
End of explanation
sample_ids = [] # Define an empty list
# Define a new sample query without limit
samples = oc.samples.search(study=study_fqn, count=True) 
for sample in samples.result_iterator():
    sample_ids.append(sample['id'])
print('There are {} samples with ids:\n {}\n'.format(len(sample_ids), sample_ids))
Explanation: We can see that the number of samples in the study is given by #Num matches by using the parameter count=True.
Execise: How to get all the sample ids? 
Above, we have used the parameter limit to restrict the number of samples the query returns. We can get all the samples ids by:
End of explanation
## Using the individuals search web service
individuals = oc.individuals.search(study=study_fqn, count=True, limit=5) ## other possible params, count=False, id='NA12880,NA12881'
individuals.print_results( title='Information about 5 individuals in the study{}'.format(study_fqn))
## Uncomment next line to display an interactive JSON viewer
# JSON(individuals.get_results())
Explanation: Individuals:
Now, we can repite the same process for check the number of individuals in the study . The difference is that now we will be making a call to the individuals web service:
End of explanation
## Using the individuals search web service without limit param
individuals = oc.individuals.search(study=study_fqn) 
## Using the new function to_data_frame()
individuals_df = individuals.to_data_frame()
print(individuals_df[['id', 'sex', 'uuid', 'creationDate']].head())
## Retrieve metrics 
individuals_df.describe()
Explanation: We might be interested in knowing when the individuals were added to OpenCGA, or the individuals sex. Since pyopencga 2.0.1.1 it is possible to export the results to a pandas dataframe object with the function to_data_frame():
End of explanation
## Using the files web service
files = oc.files.search(study=study_fqn, count=True, type='FILE', limit=5, exclude='attributes') ## other possible params, count=False, id='NA12880,NA12881'
files.print_results(fields='id,format,size,software', title='Information about files in study {}'.format(study_fqn))
## Uncomment next line to display an interactive JSON viewer
#JSON(files.get_results())
Explanation: 2. Exploring Files
Files in a study
We can start by exploring the number of files in the study, and retrieveing information about one file as an example of which kind of data is stored in the file data model of OpenCGA.
End of explanation
my_vcf = files.get_result(1)
print('The study {} contains a {} file with id: {},\ncreated on: {}'.format(study_fqn, my_vcf['format'], 
                                                                            my_vcf['id'], my_vcf['creationDate']))
Explanation: File Specific Info
There is plenty of useful information contained in the file data model like the file format, the stats, size of the file. If we want to look for more concrete information about one specific file:
End of explanation
## Using the samples info web service
sample_of_interest = sample_ids[0]
## List the files for a concrete sample
sample = oc.samples.info(study=study_fqn, samples= sample_of_interest) ## other possible params, count=False, id='NA12880,NA12881'
sample_files = sample.get_result(0)['fileIds']
print('The sample {} has file/s: {}'.format(sample_of_interest, sample_files))
Explanation: Files with a specific sample
We can also be interested in knowing the number of files for a specific sample:
End of explanation
## Using the cohorts search web service
cohorts = oc.cohorts.search(study=study_fqn, count=True, exclude='samples') ## other possible params, count=False, id='NA12880,NA12881'
cohorts.print_results(fields='id,type,description,numSamples', title='Information about cohorts in study {}'.format(study_fqn))
## Uncomment next line to display an interactive JSON viewer
#JSON(cohorts.get_results())
Explanation: 3. Exploring Cohorts
One powerful feature of OpenCGA is the possibility of define cohorts that include individuals with common traits of interest, like a phenotype, nationality etc.
The cohorts are defined at the study level. OpenCGA creates a default cohort ALL, which includes all the individuals of the study.
We can explore which cohorts are defined in the  study by:
End of explanation
<END_TASK>