How to share storage between Kubernetes pods?

问题:

I am evaluating Kubernetes as a platform for our new application. For now, it looks all very exciting! However, I’m running into a problem: I’m hosting my cluster on GCE and I need some mechanism to share storage between two pods - the continous integration server and my application server. What’s the best way for doing this with kubernetes? None of the volume types seems to fit my needs, since GCE disks can’t be shared if one pod needs to write to the disk. NFS would be perfect, but seems to require special build options for the kubernetes cluster?

EDIT: Sharing storage seems to be a problem that I have encountered multiple times now using Kubernetes. There are multiple use cases where I'd just like to have one volume and hook it up to multiple pods (with write access). I can only assume that this would be a common use case, no?

EDIT2: For example, this page describes how to set up an Elasticsearch cluster, but wiring it up with persistent storage is impossible (as described here), which kind of renders it pointless :(

回答1:

A bit late to answer this question but from my experience thus far of Kubernetes / MSA, the issue here is more in your design pattern. One of the fundamental design patterns that continues to come up quite often in MSA is the proper encapsulation of your services, which also includes its data.

Your service should look after the data that is related to its area of concern and, much like OOP, should allow access to this data to other services via an interface (an API, PUBSUB message etc). Multi-service access to data is an anti-pattern akin to global variables in OOP.

I assume that Google have the same opinion as well and this is why Kubernetes is set up in this fashion.

As an example, if you where looking to write logs, you should have a log service which each service can call with the relevant data it needs to log. Writing directly to a shared disk means that you'd need to update every container if you change your log directory structure etc or decided to add extra functionality like emails on errors.

回答2:

NFS is a built-in volume plugin and supports multiple pod writers. There are no special build options to get NFS working in Kube.

I work at Red Hat on Kubernetes, focused mainly on storage.

回答3:

First of all. Kubernetes doesn't have integrated functionality to share storage between hosts. There are several options below. But first how to share storage if you already have some volumes set up.

To share a volume between multiple pods you'd need to create a PVC with access mode ReadWriteMany

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
    name: my-pvc
spec:
    accessModes:
      - ReadWriteMany
    storageClassName: myvolume
    resources:
        requests:
            storage: 1Gi

After that you can mount it to multiple pods:

apiVersion: v1
kind: Pod
metadata:
  name: myapp1
spec:
  containers:
...
      volumeMounts:
        - mountPath: /data
          name: data
          subPath: app1
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: 'my-pvc'
---
apiVersion: v1
kind: Pod
metadata:
  name: myapp2
spec:
  containers:
...
      volumeMounts:
        - mountPath: /data
          name: data
          subPath: app2
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: 'my-pvc'

Of course, persistent volume must be accessible via network. Otherwise you'd need to make sure that all the pods are scheduled to the node with that volume.

There are several volume types that are suitable for that and not tied to any cloud provider:

• NFS
• RBD (Ceph Block Device)
• CephFS
• Glusterfs
• Portworx Volumes

Of course, to use a volume you need to have it first. That is, if you want to consume NFS you need to setup NFS on all nodes in K8s cluster. If you want to consume Ceph, you need to setup Ceph cluster and so on.

The only volume type that supports Kubernetes out of the box is Portworks. There are instruction on how to set it up in GKE.

To setup Ceph cluster in K8s there's a project in development called Rook.

But this is all overkill if you just want a folder from one node to be available in another node. In this case just setup NFS server. It wouldn't be harder than provisioning other volume types and will consume much less cpu/memory/disk resources.

回答4:

Update: The best choice is probably Cloud Filestore, a managed NFS system. This gives you full random read/write access to files, unlike GCS which only supports upload/download. See docs here.

Original: Have you tried Google Cloud Storage? You might even be able to use the FUSE adapter to map it like a network disk.

回答5:

If it is logs that you are looking to write to disk, I suggest you look at logspout https://github.com/gliderlabs/logspout. This will collect each pod's logging and then you can use google cloud platforms' fairly new logging service that uses fluentd. That way all logs from each pod are collected into a single place.

If it is data that would normally write to a database or something of that nature, I recommend having a separate server outside of the kubernetes cluster that runs the database.

EDIT

For sharing files amongst pods, I recommend mounting a google cloud storage drive to each node in your kubernetes cluster, then setting that up as a volume into each pod that mounts to that mounted directory on the node and not directly to the drive. Having it mount to each node is good because pods do not run on designated nodes, so it's best to centralize it in that case.

回答6:

Have you looked at kubernetes Volumes ? You are probably looking at creating a gcePersistentDisk

A gcePersistentDisk volume mounts a Google Compute Engine (GCE) Persistent Disk into your pod. Unlike emptyDir, which is erased when a Pod is removed, the contents of a PD are preserved and the volume is merely unmounted. This means that a PD can be pre-populated with data, and that data can be “handed off” between pods. Important: You must create a PD using gcloud or the GCE API or UI before you can use it There are some restrictions when using a gcePersistentDisk: the nodes on which pods are running must be GCE VMs those VMs need to be in the same GCE project and zone as the PD A feature of PD is that they can be mounted as read-only by multiple consumers simultaneously. This means that you can pre-populate a PD with your dataset and then serve it in parallel from as many pods as you need. Unfortunately, PDs can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed. Using a PD on a pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1.

To support multiple writes from various pods you will probably need to create one beefy pod which exposes a thrift or socket types service which exposes readFromDisk and WriteToDisk methods.

回答7:

Google recently released cloud filestore, with a tutorial here: https://cloud.google.com/filestore/docs/accessing-fileshares

Might be a good alternative to cloud storage/buckets for some scenarios.

回答8:

@Marco - in regards to the Maven related question my advice would be to stop looking at this as a centralized storage problem and perhaps think of it as a service issue.

I've run Maven repositories under HTTP in the past (read-only). I would simply create a Maven repo and expose it over Apache/Nginx in its own pod (docker container) with what ever dedicated storage you need for just that pod and then use service discovery to link it to your application and build systems.