You can find below the Technical Documentation produced from the AI4Copernicus project:
Technical Documentation June 2021
AI4Copernicus provides users with access to DIAS platforms and AI4EU resources in a streamlined way, offering support along the way.
The expected roadmap for interacting with AI4Copernicus resources is:
- Develop (CREODIAS or local/private resource)
- Optional: Onboard onto the AI4EU Experiments platform
- Refine making use of other Experiments resources
- Deploy the new workflow on CREODIAS or elsewhere
- Publish onto the public AI4EU catalogue
(1) We anticipate that, as a developer, you will develop your solution and models on CREODIAS – integration of more DIAS platforms is currently underway. (Alternative or local resources may also be used for development, however in this case support and the integration of Copernicus data will not be provided by AI4Copernicus.) CREODIAS will provide you with access to necessary cloud resources as well as to Copernicus datasets and other products.
(2) For exploring and experimenting AI4Copernicus provides a set of additional services and tools, outlined below, e.g. semantic searching.
(3) Once you have designed and built your model you can optionally onboard and publish it on the AI4EU public marketplace. This set of resources provide workflow and sharing functionality.
(4) As a final step, you are encouraged to publish your work on the AI4EU catalogue for other interested parties, researchers and practitioners to be able to discover it.
AI4EU can be reached here. The AI4EU catalogue is reachable here. Successful bidders are expected to publish their finished products on this catalogue – more information will be provided upon success. The AI4EU experiments platform is reachable here, where interested parties can register and documentation is provided. Further information and tutorials can be found at the relevant GitHub repository.
You can register on CREODIAS. As a registered user, you are entitled to a free trial. Successful AI4Copernicus bidders will be provided with credits to develop and publish their products.
More information about the participating platforms can be found on the website of AI4Copernicus.
Let’s consider a use case where we calculate the maximum value of a CO column in a given AOI (Area of Interest) and for a certain time period. To achieve this goal the user has access to AI tools to set up a pipeline which runs a workflow in an automated way. The result of this use-case is the value presented on the port of a container deployed in the Kubernetes cluster that is set up in the CREODIAS environment.
1. Knowledge base for the example CO processing
- Finder API, to look for relevant data. Documentation here.
- Help on using the S3 API can be found at the following resources:
i. HOW TO ACCESS/LIST EO DATA USING BOTO3: Find here
ii. HOW TO DOWNLOAD EO DATA FILE USING BOTO3: Find here.
iii. EO DATA ACCESS – S3 OR NFS?: Find here.
iv. HOW TO ACCESS EO DATA AND OBJECT STORAGE USING S3CMD (LINUX): See here.
- Interacting with the experimental AI4EU experiments requires knowledge on GRPC communication standards. This is provided here
- For data acquisition in NetCDF format please refer here.
2. CREODIAS Kubernetes cluster
- kubectl setup for CREODIAS Kubernetes cluster. Configuration is available after login to CREODIAS under ~/.kube/config
3. Python 3: This use-case has been implemented in Python 3.
At the beginning the user needs to prepare a data broker to find relevant data products, and an analyzer to read data from product files. The user then needs to publish the relevant docker images in the image registry.
1. Information on model onboarding is presented at:
2. Pipeline composition and local download:
3. Configuration of kubectl to point execution environment in CREODIAS
4. Deployment and execution of the solution pipeline
For this particular use-case we prepare:
1. An EO Data broker that searches for Sentinel-5p’s carbon monoxide data via the EO Finder API. This broker returns the file metadata to the pipeline designed on AI4EU experiments
2. A CO Max analyzer makes use of the EO data files using S3 API and makes available the calculated value via a REST API
The figure below outlines the procedure:
AI4Copernicus Services Overview
As part of our technical work in AI4Copernicus we will provide semantic searching over Copernicus data as well as pre-trained ML and related models to be used by our users and bidders. More information on how to access and use them will be provided here soon.
Contacts for technical information
AI4EU Experiments: Antonis Koukourikos (NCSR-D): firstname.lastname@example.org