Deploy C2D
This chapter will present how to deploy the C2D component of the Ocean stack. As mentioned in the C2D Architecture chapter, the Compute-to-Data component uses Kubernetes to orchestrate the creation and deletion of the pods in which the C2D jobs are run.
For the ones that do not have a Kubernetes environment available, we added to this guide instructions on how to install Minikube, which is a lightweight Kubernetes implementation that creates a VM on your local machine and deploys a simple cluster containing only one node. In case you have a Kubernetes environment in place, please skip directly to step 4 of this guide.
Requirements
Communications: a functioning internet-accessible provider service
Hardware: a server capable of running compute jobs (e.g. we used a machine with 8 CPUs, 16 GB Ram, 100GB SSD, and a fast internet connection). See this guide for how to create a server;
Operating system: Ubuntu 22.04 LTS
Steps
Install Docker and Git
Install Minikube
Start Minikube
The first command is important and solves a PersistentVolumeClaims problem.
Depending on the number of available CPUs, RAM, and the required resources for running the job, consider adding options --cpu
, --memory
, and --disk-size
to avoid runtime issues.
For other options to run minikube refer to this link
Install the Kubernetes command line tool (kubectl)
Wait until all the defaults are running (1/1).
Run the IPFS host (optional)
To store the results and the logs of the C2D jobs, you can use either an AWS S3 bucket or IPFS.
In case you want to use IPFS you need to run an IPFS host, as presented below.
Update the storage class
The storage class is used by Kubernetes to create the temporary volumes on which the data used by the algorithm will be stored.
Please ensure that your class allocates volumes in the same region and zone where you are running your pods.
You need to consider the storage class available for your environment.
For Minikube, you can use the default 'standard' class.
In AWS, we created our own 'standard' class:
For more information, please visit https://kubernetes.io/docs/concepts/storage/storage-classes/
Download and Configure Operator Service
Open a new terminal and run the command below.
Edit operator-service/kubernetes/postgres-configmap.yaml
. Change POSTGRES_PASSWORD
to a nice long random password.
Edit operator-service/kubernetes/deployment.yaml
. Optionally change:
ALGO_POD_TIMEOUT
add
requests_cpu
add
requests_memory
add
limits_cpu
add
limits_memory
Download and Configure Operator Engine
Check the README section of the operator engine to customize your deployment.
At a minimum, you should add your IPFS URLs or AWS settings, and add (or remove) notification URLs.
Create namespaces
Deploy Operator Service
Deploy Operator Engine
Optional: For production enviroments, it's safer to block access to metadata. To do so run the below command:
Expose Operator Service
Run a port forward or create your ingress service and setup DNS and certificates (not covered here):
Alternatively you could use another method to communicate between the C2D Environment and the provider, such as an SSH tunnel.
Initialize database
If your Minikube is running on compute.example.com:
Update Provider
Update your provider service by updating the operator_service.url
value in config.ini
Restart your provider service.
Last updated