<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Python datamover module tutorial
Introduction
The datamover Python module provides high-level functionalities to transfer files in between different file resources (local, GitHub, S3 buckets and FTP-drives). To activate the functionalities, import the library functions
Step1:  The module provide for each of the file resources a class, with for each of the classes the methods download_file and list_files defined
Step2:  As the S3 bucket provides an essential part in the Enram data infrastructure, an additional class S3EnramHandler is available, providing the required functions to handle the enram bucket. As the class is inherited from the S3Connector, those functions are available as well in the S3 handler
Step3:  In order to transfer files, Transporter classes are available to define specific transfers. Currently a LocalToS3 and a BaltradToS3 are defined to manage the file transfer from respectively a local file resource and the Baltrad file server to the Enram S3 bucket
Step4:  From a enram file managament perspective, the S3EnramHandler, together with the transporter classes are most relevant, as explained in the following sections.
Access to the S3 instance
Access rights to the S3 instance are managed indirectly, using the ~/.aws/credentials file and by attributing the proper rights to the user in the AWS console. When the user rights are configured and the proper policy is attributed, connection to the S3 bucket from the datamover package is as follows
Step5:  Functions for S3 file checks are provided to support file checks and enlisting
Step6:  As the file listing provides a generator, different options are available to have an overview list
Step7:  As the datamover is just a thin layer around the boto3 package, the other boto3 S3 client options are still available to use
Step8:  File transfer
Baltrad FTP to S3 bucket
In order to have access to the Baltrad-server, a credentials file (creds.py) is required, defining the variables URL, LOGIN and PASSWORD
Step9:  The transporter class BaltradToS3 supports the file transfer
Step10:  The necessity of the profile_name depends from your AWS account setup. If you're default profile has the appropriate policy rights (as it is with the EC2 instance running the daily cron job), the AWS package will automatically use the default credentials and you do not need to specify the profile to use.  
A transfer of files is provided by the transfer method. It is possible to limit the scope of the file transfer by defining a name match string. As a user, you can decide to overwrite the S3 bucket files or not. Furthermore, for testing purposes, a limit option has been provided and the option to print the transfers to stdout
Step11:  The results of the transfer are logged in the atributes btos.transferred and btos.stalled. A combined report can be written to a file log_file_transfer. The transfertype option provides the user the ability to have a custom message in the transfer header
Step12:  The log is written to a file log_file_transfer
Step13:  The transporter classes provide direct access to the individual connectors of the transfer, analog as the usage of the connector as such
Step14:  Local files to S3 bucket
The transfer from a local file directory towards the S3 Bucket is similar in the API, with the transfer method
Step15:  S3 enram handler functions
The S3EnramHandler class provides some additional functions to support the enram infrastructure
Step16:  The data coverage for each radar can be derived for multiple temporal intervals
Step17:  Remark the usage of the _ to ignore the second output of the function, which is the information on the most recent available file for each radar.
As an example, derive the number of files available for April 7th 2017 for the tra radar
Step18:  The same aggregation function can be used for monthly and yearly counts
Step19:  For example, derive the yearly counts for the Belgian radars
Step20:  The counts can be exported to a CSV-file as well, available as general datamover utility function
Step21:  The most recent files for each radar can be extracted using the same function
Step22:  and saved to a file as well
Step23:  Zip file support
As downloading the individual .h5 files from the website would be cumbersome, two options are available for easier data access
Step24:  A typical use-case is the update of those files that were transferred during a transfer operation, e.g. btos.transferred
Step25:  Other options are possible, e.g. update those zip files for a specific radar
Step26:  An entire update can be done, by using the available coverage on a monthly or daily level (in comments, as this is a large operation)
Step27:  Further remarks
An essential element in the file and folder handling, is that the (sub)folder information is inherent to the file name itself
<ASSISTANT_TASK:>
Python Code:
import datamover as dm
Explanation: Python datamover module tutorial
Introduction
The datamover Python module provides high-level functionalities to transfer files in between different file resources (local, GitHub, S3 buckets and FTP-drives). To activate the functionalities, import the library functions:
End of explanation
print(dm.LocalConnector.list_files, dm.LocalConnector.download_file)
print(dm.S3Connector.list_files, dm.S3Connector.download_file)
print(dm.FTPConnector.list_files, dm.FTPConnector.list_files)
print(dm.GithubConnector.list_files, dm.GithubConnector.download_file)
Explanation: The module provide for each of the file resources a class, with for each of the classes the methods download_file and list_files defined:
End of explanation
# print available methods:
print([method for method in dir(dm.S3EnramHandler)  if not method.startswith("_")])
Explanation: As the S3 bucket provides an essential part in the Enram data infrastructure, an additional class S3EnramHandler is available, providing the required functions to handle the enram bucket. As the class is inherited from the S3Connector, those functions are available as well in the S3 handler:
End of explanation
print(dm.LocalToS3, dm.BaltradToS3)
Explanation: In order to transfer files, Transporter classes are available to define specific transfers. Currently a LocalToS3 and a BaltradToS3 are defined to manage the file transfer from respectively a local file resource and the Baltrad file server to the Enram S3 bucket:
End of explanation
s3 = dm.S3Connector("lw-enram") # analog for S3EnramHandler
s3.bucket_name
Explanation: From a enram file managament perspective, the S3EnramHandler, together with the transporter classes are most relevant, as explained in the following sections.
Access to the S3 instance
Access rights to the S3 instance are managed indirectly, using the ~/.aws/credentials file and by attributing the proper rights to the user in the AWS console. When the user rights are configured and the proper policy is attributed, connection to the S3 bucket from the datamover package is as follows:
End of explanation
# check if a file(path) already exists on the S3 bucket:
s3.key_exists('cz/brd/2017/04/09/23/czbrd_vp_20170409230000.h5')
Explanation: Functions for S3 file checks are provided to support file checks and enlisting:
End of explanation
set(s3.list_files(path='cz/brd/2016/09/23/00')) # using set
for filepath in s3.list_files(path='cz/brd/2016/09/23/00'):
    print(filepath)
    # do something...
Explanation: As the file listing provides a generator, different options are available to have an overview list:
End of explanation
print([method for method in dir(s3.s3client)  if not method.startswith("_")])
Explanation: As the datamover is just a thin layer around the boto3 package, the other boto3 S3 client options are still available to use:
End of explanation
from creds import URL, LOGIN, PASSWORD
Explanation: File transfer
Baltrad FTP to S3 bucket
In order to have access to the Baltrad-server, a credentials file (creds.py) is required, defining the variables URL, LOGIN and PASSWORD:
End of explanation
btos = dm.BaltradToS3(URL, LOGIN, PASSWORD, "lw-enram", profile_name="lw-enram")
Explanation: The transporter class BaltradToS3 supports the file transfer:
End of explanation
# transfer files with _vp_ in the name, overwriting existing files and limiting the transferred files to 5:
btos.transfer(name_match="_vp_", overwrite=True, 
              limit=5, verbose=True)
Explanation: The necessity of the profile_name depends from your AWS account setup. If you're default profile has the appropriate policy rights (as it is with the EC2 instance running the daily cron job), the AWS package will automatically use the default credentials and you do not need to specify the profile to use.  
A transfer of files is provided by the transfer method. It is possible to limit the scope of the file transfer by defining a name match string. As a user, you can decide to overwrite the S3 bucket files or not. Furthermore, for testing purposes, a limit option has been provided and the option to print the transfers to stdout:
End of explanation
btos.transferred
btos.report(reset_file=True, transfertype="Baltrad to S3")
Explanation: The results of the transfer are logged in the atributes btos.transferred and btos.stalled. A combined report can be written to a file log_file_transfer. The transfertype option provides the user the ability to have a custom message in the transfer header:
End of explanation
!cat log_file_transfer
Explanation: The log is written to a file log_file_transfer:
End of explanation
btos.s3.key_exists('de/boo/2018/01/13/00/deboo_vp_20180113T0015Z_0x5.h5') # S3 check for existing file
set(btos.s3.list_files(path='de/boo/2018/01/13/00')) # S3 file listing
set(btos.ftp.list_files(name_match="deboo_vp_20180113T0015Z_0x5.h5")) # ftp file listing
Explanation: The transporter classes provide direct access to the individual connectors of the transfer, analog as the usage of the connector as such:
End of explanation
ltos = dm.LocalToS3(filepath="../example_data/", bucket_name="lw-enram", 
                    profile_name="lw-enram")
ltos.transfer(name_match="_vp_", overwrite=False, 
              limit=5, verbose=True)
ltos.transferred
Explanation: Local files to S3 bucket
The transfer from a local file directory towards the S3 Bucket is similar in the API, with the transfer method:
End of explanation
s3enram = dm.S3EnramHandler("lw-enram", profile_name="lw-enram") # Connecto to S3 client
Explanation: S3 enram handler functions
The S3EnramHandler class provides some additional functions to support the enram infrastructure:
* coverage check: check how many files are available for a specific time basis and get the most recent file for each of the country/radar combination
* zip file creation for bulk data transfers
Bird profile data coverage
End of explanation
# Rerun file list overview to extract the current coverage
coverage_day, _ = s3enram.count_enram_coverage(level='day')
Explanation: The data coverage for each radar can be derived for multiple temporal intervals: day | month | year. For the heatmap on the repository, the daily counts are used:
End of explanation
coverage_day['frtra 2017-04-07']
Explanation: Remark the usage of the _ to ignore the second output of the function, which is the information on the most recent available file for each radar.
As an example, derive the number of files available for April 7th 2017 for the tra radar:
End of explanation
coverage_year, _ = s3enram.count_enram_coverage(level='year')
Explanation: The same aggregation function can be used for monthly and yearly counts:
End of explanation
{k:v for k,v in coverage_year.items() if k.startswith("be")}
Explanation: For example, derive the yearly counts for the Belgian radars:
End of explanation
with open("my_coverage_filename.csv", 'w') as outfile:
    dm.coverage_to_csv(outfile, coverage_year)
Explanation: The counts can be exported to a CSV-file as well, available as general datamover utility function:
End of explanation
_, most_recent_file = s3enram.count_enram_coverage(level='month')
most_recent_file
Explanation: The most recent files for each radar can be extracted using the same function:
End of explanation
with open("radars_latest.csv", 'w') as outfile:
    dm.most_recent_to_csv(outfile, most_recent_file)
!head radars_latest.csv
Explanation: and saved to a file as well:
End of explanation
keyset = ['bewid_vp_20161120233000.h5', 
          'bewid_vp_20161120233500.h5',
          'bewid_vp_20161120234000.h5',
          'bewid_vp_20161120234500.h5',
          'bewid_vp_20161120235000.h5',
          'bewid_vp_20161120235500.h5',
          'bejab_vp_20161120235000.h5']
s3enram.create_zip_version(keyset)
Explanation: Zip file support
As downloading the individual .h5 files from the website would be cumbersome, two options are available for easier data access:
* Inclusion of a download function in the BioRad R package, dedicated for bird profile research
* Download of aggregated monthly dataset, provided as a zip-folders
The preparation and creation of these zip-folders is supported by the S3EnramHandler module, using the create_zip_version function. The function uses a Counter with the key/counts or a list of keys from which the monthly counts will be derived as input.
As downloading the individual .h5 files from the website would be cumbersome, two options are available for easier data access:
* Inclusion of a download function in the BioRad R package, dedicated for bird profile research
* Download of aggregated monthly dataset, provided as a zip-folders
The preparation and creation of these zip-folders is supported by the S3EnramHandler module, using the create_zip_version function. The function uses a Counter with the key/counts or a list of keys from which the monthly counts will be derived as input.
Using a list of keys, the relevant month/radar combinations are updated:
End of explanation
s3enram.create_zip_version(btos.transferred)
Explanation: A typical use-case is the update of those files that were transferred during a transfer operation, e.g. btos.transferred
End of explanation
import os
country = "be"
radar = "wid"
keyset = []
for key in s3enram.list_files(path="/".join([country, radar])):
    keyset.append(os.path.split(key)[1])
s3enram.create_zip_version(keyset)
Explanation: Other options are possible, e.g. update those zip files for a specific radar:
End of explanation
# s3client.create_zip_version(s3client.count_enram_coverage(level="month"))
Explanation: An entire update can be done, by using the available coverage on a monthly or daily level (in comments, as this is a large operation):
End of explanation
dm.parse_filename("dkrom_vp_20170114231500.h5")
Explanation: Further remarks
An essential element in the file and folder handling, is that the (sub)folder information is inherent to the file name itself:
Parsing the file name for metadata, e.g. dkrom_vp_20170114231500.h5:
country: 2 characters dk
radar: 3 characters rom
ignore _vp_
year: 4 characters 2017
month: 2 characters 01
day: 2 characters 14
hour: 2 characters 23
minutes: 2 characters 00
The name parsing is provided by the parse_filename function:
End of explanation
<END_TASK>