Kubernetes Clusters impacted by KubeFlow Crypto mining Attack

Microsoft has released a study describing an ongoing series of attacks on Kubeflow, a toolkit on top of Kubernetes clusters for running machine learning ( ML) operations. According to Microsoft, Kubernetes was at the forefront of a more recent large-scale XMRIG initiative. Misconfigured dashboards are at the center of a widespread campaign for the XMRIG Monero-mining. Closer inspection showed the image running a common open-source crypto-jacking malware that mines the virtual currency of Monero, known as XMRIG. The attacks have been ongoing since April 2020. The end goal of Microsoft is to deploy a cryptocurrency miner on K8s clusters running internet-exposed Kubeflow instances.

According to a security researcher, the business has found such kinds of attacks against ten of Kubernetes clusters running ML toolkit Kubeflow. Although the number of compromised clusters is small relative to previous attacks by Kubernetes (k8s). Server owners’ profits for crooks and financial losses are more likely than previously seen.

Nodes used to perform ML tasks are often fairly efficient, and often include GPUs. This makes Kubernetes clusters a great target for crypto-mining campaigns. They are also for ML tasks, which was the purpose of this attack.

KubeFlow – Cloud-Native Machine Learning toolkit for Kubernetes

It is an open-source project that integrates the most common resources of data science in one place. It’s an exciting toolkit with Kubernetes under the hood which can make life easier for Data Scientists and Data Engineers.

Kubeflow components

When you have a Kubeflow case going, though, you will get an integrated component. This shows that it will allow you to create machine learning models and bring them into production. Kubeflow is a project-based in the cloud. It can be implemented using various providers of cloud platforms: Azure, Google Cloud Platform (GCP), and Amazon Web Services ( AWS). It can also be installed on your local machine or on-prem infrastructure.

1. Notebooks: Notebooks are an industry norm for the prototyping and sharing of machine learning models across teams. You can deploy a Jupyter Lab instance easily in Kubeflow, and start building your ML models.
2. Model training: Kubeflow has built multiple model training options into it. From local Jupyter Notebook training, through managed services such as the Google Cloud AI Platform (formerly known as the Cloud Machine Learning Engine), to TensorFlow Training (TFJob) custom distributed training. To analyze your model you can use TensorBoard or TensorFlow Model Analysis (TFMA) with ease.
3. Fairing: Kubeflow Fairing is a Python package allowing you to train the ML model in a cloud environment that is hybrid. You can choose whether to train the model locally (i.e. in the Jupyter Notebook) or cloud-based.
4. Hyperparameter tuning: Kubeflow has incorporated Katib — a hyperparameter tuning system that runs on Kubernetes and implements Google Vizier.
5. Pipelines: Kubeflow pipelines are a modern method for the development of end-to-end ML workflow. Every stage of a pipeline is a Docker container and a.yaml file (using Argo) describes the pipeline.
6. Experiments: However, kubeflow offers a convenient way of comparing different pipeline speeds.
7. Model serving: Options are available for multiple ML models: like TensorFlow Serving + Istio, TensorFlow Batch Prediction, Seldon Serving, NVIDIA TensorRT Inference Server, PyTorch Serving.

Popular attack vectors used to detect compromised Kubernetes clusters

Firstly, Kubernetes is a system for running and coordinating containerized applications across a machine cluster. It is a platform to use methods that provide predictability, scalability, and high availability to manage the life cycle of container type applications and services. The data gets encrypted when writing into etcd. Any newly created or updated secret should be encrypted when stored until your Kube-API server has been restarted. To search, you can use the command line software etcdctl to get back the contents of your file. KMS provider uses gRPC to interact with a plugin unique to KMS. This is responsible for all remote KMS contact.

Popular attack vectors used to detect compromised Kubernetes clusters are mentioned below:

Azure Security Center and Kubernetes clusters 

These activities typically run in insecure containers, such as web applications, which exploit known vulnerabilities. Azure Security Center recently found a new campaign of crypto-mining that specifically targets Kubernetes environments. What separates this attack from other crypto-mining attacks is its scale: In just two hours, a malicious container will be deployed on tens of Kubernetes clusters. The containers were running a image in a public repository: kannix / monero-miner. Also the same actor who deployed the crypto mining containers identified the cluster resources including the secrets of Kubernetes. To be safe, it is necessary to carefully configure the Kubernetes environment to ensure that no container-focus surface doors are left open to attackers.cs, such as a privileged container detected.

Misconfigured workloads in Kubeflow pose a security risk to Kubernetes Clusters

Azure Security Center tracks and resists attacks of 1000+ Kubernetes clusters that run on top of the Azure Kubernetes Service. Although, they have published a blog post related to large-scale campaign against k8s clusters that abused exposed dashboards to deploy cryptocurrency miners. Kubeflow, a Kubernetes ML workflow toolkit, found that this attack occurred on many of Kubernetes clusters.

Kubeflow, a free and open-source machine learning platform, started with Kubernetes as a project to run TensorFlow work. This fact however makes clusters of Kubernetes used for Machine Learning tasks. Kubeflow Dashboards are the exact focus for crypto-mining campaigns. In many different k8s clusters a suspicious image from a public repo was observed during April.

Kubeflow is a container type service. Few of the services include training model frameworks, notebook server Jupyter and Katib, and more. the different tasks in the cluster are running as containers. So if attackers acquire access Kubeflow. They have many doors to run malicious container images in the k8s cluster.

Once attackers acquire access to dashboard, they can upload backdoor container images in the k8s cluster using multiple methods. For example:

• Kubeflow helps users to build a server on the Jupyter notebook. Kubeflow can allow users to pick a notebook server image, including the option to select a custom container image.
• Another tool that attackers can utilize is to install a malicious image from a real Jupyter notebook. Also, Attackers can use Python code to run a new or existing notebook.

Where to test if they affect your Kubernetes clusters?

• Verify that there is no malicious container image/pod into the cluster. The command below will help you test it is: kubectl get pods -A -o jsonpath=”{.items[].spec.containers[].image}”
• If Kubeflow is deployed in the cluster, ensure its dashboard is not exposed to the internet. Search the Istio incoming service form by the below command and make sure it does not have a load-balancer with a publicIP : kubectl get -n istio-system istio-ingress gateway service

Conclusion

Since the past, Azure Security Center has identified several campaigns against K8s clusters which have a common vector of access: an exposed internet connection. This is however the first time we have detected an attack directly targeting Kubeflow environments. When deploying a service such as Kubeflow in a cluster it is essential to be aware of security aspects such as:

• User Authentication and Access Control.
• Monitor the cluster endpoints which are open to the public. Moreover, it ensures that vulnerable devices are not exposing to the Internet in an insecure way.
• Track the Kubernetes runtime environment periodically. That includes monitoring of the containers running, images, and processes they run.
• Make it easy to deploy only trustworthy images and find vulnerabilities in your files. Although using Azure Policy you can restrict the images that are included in the cluster.

Moreover, the Kubernetes threat matrix includes techniques that attackers can use to target the Kubernetes cluster. The intruder used an exposed dashboard to obtain initial access to the cluster. The execution and durability of the cluster were achieved by establishing a container within the cluster that was installed inside the cluster.