Elastic Search 在 Kubernetes 上升级到 v8

提问者：小点点

Elastic Search 在 Kubernetes 上升级到 v8

我在使用 7.x 图表部署的 Microsoft Kubernetes 集群上进行弹性搜索部署，并将图像更改为 8.x。此升级有效，并且 elastic 和 Kibana 都可以访问，但现在我需要启用从现在开始包含在基本许可证中的新安全功能。安全性背后的原因首先来自启用APM服务器/代理的要求。

我有以下值：

- name: cluster.initial_master_nodes
  value: elasticsearch-master-0,
- name: discovery.seed_hosts
  value: elasticsearch-master-headless
- name: cluster.name
  value: elasticsearch
- name: network.host
  value: 0.0.0.0
- name: cluster.deprecation_indexing.enabled
  value: 'false'
- name: node.roles
  value: data,ingest,master,ml,remote_cluster_client

弹性搜索和 kibana pod 能够启动，但由于安全性，我无法设置 APM 集成。因此，我使用以下值启用安全性：

- name: xpack.security.enabled
  value: 'true'

然后我从弹性搜索窗格中收到一个错误日志：“如果启用了安全性，则必须启用传输SSL。请将 [xpack.security.transport.ssl.enabled] 设置为 [true] 或通过将 [xpack.security.enabled] 设置为 [false]“ 来禁用安全性。所以我使用以下值启用 ssl：

- name: xpack.security.transport.ssl.enabled
  value: 'true'

然后我从弹性搜索 pod 收到一个错误日志：“xpack.security.transport.ssl 的 SSL 配置无效 - 服务器 SSL 配置需要密钥和证书，但尚未配置;您必须设置 [xpack.security.transport.ssl.keystore.path]（p12 文件），或同时设置 [xpack.security.transport.ssl.key]（PEM 文件）和 [xpack.security.transport.ssl.certificate]（PEM 密钥文件）”。

我从选项 1 开始，我使用以下命令创建密钥（无密码/输入、输入/输入、输入、输入），并将它们复制到一个持久文件夹：

./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
cp elastic-stack-ca.p12 data/elastic-stack-ca.p12
cp elastic-certificates.p12 data/elastic-certificates.p12

此外，我还在配置以下值：

- name: xpack.security.transport.ssl.truststore.path
  value: '/usr/share/elasticsearch/data/elastic-certificates.p12'
- name: xpack.security.transport.ssl.keystore.path
  value: '/usr/share/elasticsearch/data/elastic-certificates.p12'

但是 pod 仍在初始化中，如果使用密码生成证书。然后，我从弹性搜索 Pod 收到错误日志：“无法读取配置的 [PKCS12] 密钥库（作为信任库）[/usr/share/elasticsearch/data/elastic-certificate.p12] - 这通常是由不正确的密码引起的;（未提供密码）”

然后我转到 Option2，我正在使用以下命令创建密钥，并将它们复制到一个持久文件夹

./bin/elasticsearch-certutil ca --pem
unzip elastic-stack-ca.zip –d

cp ca.crt data/ca.crt
cp ca.key data/ca.key

此外，我还在配置以下值：

- name: xpack.security.transport.ssl.key
  value: '/usr/share/elasticsearch/data/ca.key'
- name: xpack.security.transport.ssl.certificate
  value: '/usr/share/elasticsearch/data/ca.crt'

最后，我在eleastic搜索社区中找到了相同的问题，没有任何回应：https://discuss.elastic.co/t/elasticsearch-pods-are-not-ready-when-xpack-security-enabled-is-configured/281709?u=s19k15

这是我的状态完整集

status:
  observedGeneration: 169
  replicas: 1
  updatedReplicas: 1
  currentRevision: elasticsearch-master-7449d7bd69
  updateRevision: elasticsearch-master-7d8c7b6997
  collisionCount: 0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch-master
  template:
    metadata:
      name: elasticsearch-master
      creationTimestamp: null
      labels:
        app: elasticsearch-master
        chart: elasticsearch
        release: platform
    spec:
      initContainers:
        - name: configure-sysctl
          image: docker.elastic.co/elasticsearch/elasticsearch:8.1.2
          command:
            - sysctl
            - '-w'
            - vm.max_map_count=262144
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            runAsUser: 0
      containers:
        - name: elasticsearch
          image: docker.elastic.co/elasticsearch/elasticsearch:8.1.2
          ports:
            - name: http
              containerPort: 9200
              protocol: TCP
            - name: transport
              containerPort: 9300
              protocol: TCP
          env:
            - name: node.name
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: cluster.initial_master_nodes
              value: elasticsearch-master-0,
            - name: discovery.seed_hosts
              value: elasticsearch-master-headless
            - name: cluster.name
              value: elasticsearch
            - name: cluster.deprecation_indexing.enabled
              value: 'false'
            - name: ES_JAVA_OPTS
              value: '-Xmx512m -Xms512m'
            - name: node.roles
              value: data,ingest,master,ml,remote_cluster_client
            - name: xpack.license.self_generated.type
              value: basic
            - name: xpack.security.enabled
              value: 'true'
            - name: xpack.security.transport.ssl.enabled
              value: 'true'
            - name: xpack.security.transport.ssl.truststore.path
              value: /usr/share/elasticsearch/data/elastic-certificates.p12
            - name: xpack.security.transport.ssl.keystore.path
              value: /usr/share/elasticsearch/data/elastic-certificates.p12
            - name: xpack.security.http.ssl.enabled
              value: 'true'
            - name: xpack.security.http.ssl.truststore.path
              value: /usr/share/elasticsearch/data/elastic-certificates.p12
            - name: xpack.security.http.ssl.keystore.path
              value: /usr/share/elasticsearch/data/elastic-certificates.p12
            - name: logger.org.elasticsearch.discovery
              value: debug
            - name: path.logs
              value: /usr/share/elasticsearch/data
            - name: xpack.security.enrollment.enabled
              value: 'true'
          resources:
            limits:
              cpu: '1'
              memory: 2Gi
            requests:
              cpu: 100m
              memory: 512Mi
          volumeMounts:
            - name: elasticsearch-master
              mountPath: /usr/share/elasticsearch/data
          readinessProbe:
            exec:
              command:
                - bash
                - '-c'
                - >
                  set -e

                  # If the node is starting up wait for the cluster to be ready
                  (request params: "wait_for_status=green&timeout=1s" )

                  # Once it has started only check that the node itself is
                  responding

                  START_FILE=/tmp/.es_start_file


                  # Disable nss cache to avoid filling dentry cache when calling
                  curl

                  # This is required with Elasticsearch Docker using nss < 3.52

                  export NSS_SDB_USE_CACHE=no


                  http () {
                    local path="${1}"
                    local args="${2}"
                    set -- -XGET -s

                    if [ "$args" != "" ]; then
                      set -- "$@" $args
                    fi

                    if [ -n "${ELASTIC_PASSWORD}" ]; then
                      set -- "$@" -u "elastic:${ELASTIC_PASSWORD}"
                    fi

                    curl --output /dev/null -k "$@" "http://127.0.0.1:9200${path}"
                  }


                  if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy'
                    HTTP_CODE=$(http "/" "-w %{http_code}")
                    RC=$?
                    if [[ ${RC} -ne 0 ]]; then
                      echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with RC ${RC}"
                      exit ${RC}
                    fi
                    # ready if HTTP code 200, 503 is tolerable if ES version is 6.x
                    if [[ ${HTTP_CODE} == "200" ]]; then
                      exit 0
                    elif [[ ${HTTP_CODE} == "503" && "8" == "6" ]]; then
                      exit 0
                    else
                      echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with HTTP code ${HTTP_CODE}"
                      exit 1
                    fi

                  else
                    echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" "--fail" ; then
                      touch ${START_FILE}
                      exit 0
                    else
                      echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                      exit 1
                    fi
                  fi
            initialDelaySeconds: 10
            timeoutSeconds: 5
            periodSeconds: 10
            successThreshold: 3
            failureThreshold: 3
          lifecycle:
            postStart:
              exec:
                command:
                  - bash
                  - '-c'
                  - >
                    #!/bin/bash

                    # Create the
                    dev.general.logcreation.elasticsearchlogobject.v1.json index

                    ES_URL=http://localhost:9200

                    while [[ "$(curl -s -o /dev/null -w '%{http_code}\n'
                    $ES_URL)" != "200" ]]; do sleep 1; done

                    curl --request PUT --header 'Content-Type: application/json'
                    "$ES_URL/dev.general.logcreation.elasticsearchlogobject.v1.json/"
                    --data
                    '{"mappings":{"properties":{"Properties":{"properties":{"StatusCode":{"type":"text"}}}}},"settings":{"index":{"number_of_shards":"1","number_of_replicas":"0"}}}'
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              drop:
                - ALL
            runAsUser: 1000
            runAsNonRoot: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 120
      dnsPolicy: ClusterFirst
      automountServiceAccountToken: true
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - elasticsearch-master
              topologyKey: kubernetes.io/hostname
      schedulerName: default-scheduler
      enableServiceLinks: true
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: elasticsearch-master
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 4Gi
        volumeMode: Filesystem
      status:
        phase: Pending
  serviceName: elasticsearch-master-headless
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  revisionHistoryLimit: 10

有什么想法吗？

共1个答案

匿名用户

终于找到了答案，也许这对很多人都有帮助，以防他们遇到类似的事情。当 pod 无休止地初始化时，就像睡觉一样。就我而言，当启用安全性时，我的图表 StatefullSet 中的一个奇怪代码开始导致此问题。

while [[ "$(curl -s -o /dev/null -w '%{http_code}\n'
                    $ES_URL)" != "200" ]]; do sleep 1; done

这不会返回 200，因为现在 http 也除了一个用户和一个密码进行身份验证，因此要睡觉。

因此，请确保在 pod 处于初始化状态并保持在那里的情况下，没有任何 while/sleep