Spring Boot 3.2 - 使用 NodePort 的多集群 MongoDB 连接-解网

问：

我有一个基于 Spring Boot 3.2 （RC2）的应用程序，它连接到 MongoDB。我的 MongoDB 是部署在两个 Kubernetes 集群上的 ReplicaSet：

集群 A 上的三个节点（和一个仲裁服务器）
集群 B 上的三个节点节点（因此 Pod）通过 NodePort 公开，因此 ReplicaSet 的所有同步都通过 NodePort 进行。

我的rs.status（）给出的结果类似于以下内容：

{
  set: 'rsTest',
  date: ISODate("2023-11-17T08:12:31.374Z"),
  myState: 1,
  term: Long("2"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 4,
  writeMajorityCount: 4,
  votingMembersCount: 7,
  writableVotingMembersCount: 6,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1700208749, i: 2 }), t: Long("2") },
    lastCommittedWallTime: ISODate("2023-11-17T08:12:29.165Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1700208749, i: 2 }), t: Long("2") },
    appliedOpTime: { ts: Timestamp({ t: 1700208750, i: 6 }), t: Long("2") },
    durableOpTime: { ts: Timestamp({ t: 1700208750, i: 6 }), t: Long("2") },
    lastAppliedWallTime: ISODate("2023-11-17T08:12:30.553Z"),
    lastDurableWallTime: ISODate("2023-11-17T08:12:30.553Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1700208716, i: 7 }),
  electionCandidateMetrics: {
    lastElectionReason: 'electionTimeout',
    lastElectionDate: ISODate("2023-11-10T11:37:02.387Z"),
    electionTerm: Long("2"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1699616220, i: 9 }), t: Long("1") },
    numVotesNeeded: 1,
    priorityAtElection: 5,
    electionTimeoutMillis: Long("10000"),
    newTermStartDate: ISODate("2023-11-10T11:37:02.392Z"),
    wMajorityWriteAvailabilityDate: ISODate("2023-11-10T11:37:02.399Z")
  },
  members: [
    {
      _id: 0,
      name: '<WORKER-NODE-CLUSTER-A-IP-1>:30100',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 592530,
      optime: { ts: Timestamp({ t: 1700208750, i: 6 }), t: Long("2") },
      optimeDate: ISODate("2023-11-17T08:12:30.000Z"),
      lastAppliedWallTime: ISODate("2023-11-17T08:12:30.553Z"),
      lastDurableWallTime: ISODate("2023-11-17T08:12:30.553Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1699616222, i: 1 }),
      electionDate: ISODate("2023-11-10T11:37:02.000Z"),
      configVersion: 17,
      configTerm: 2,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: '<ARBITER-SVC>:27017',
      health: 1,
      state: 7,
      stateStr: 'ARBITER',
      uptime: 592517,
      lastHeartbeat: ISODate("2023-11-17T08:12:30.975Z"),
      lastHeartbeatRecv: ISODate("2023-11-17T08:12:30.977Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 17,
      configTerm: 2
    },
    {
      _id: 2,
      name: '<WORKER-NODE-CLUSTER-A-IP-2>:30101',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 592478,
      optime: { ts: Timestamp({ t: 1700208749, i: 6 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1700208749, i: 6 }), t: Long("2") },
      optimeDate: ISODate("2023-11-17T08:12:29.000Z"),
      optimeDurableDate: ISODate("2023-11-17T08:12:29.000Z"),
      lastAppliedWallTime: ISODate("2023-11-17T08:12:30.553Z"),
      lastDurableWallTime: ISODate("2023-11-17T08:12:30.553Z"),
      lastHeartbeat: ISODate("2023-11-17T08:12:30.080Z"),
      lastHeartbeatRecv: ISODate("2023-11-17T08:12:31.179Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: '<WORKER-NODE-CLUSTER-A-IP-1>:30100',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 17,
      configTerm: 2
    },
    {
      _id: 3,
      name: '<WORKER-NODE-CLUSTER-A-IP-3>:30102',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 592447,
      optime: { ts: Timestamp({ t: 1700208749, i: 6 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1700208749, i: 6 }), t: Long("2") },
      optimeDate: ISODate("2023-11-17T08:12:29.000Z"),
      optimeDurableDate: ISODate("2023-11-17T08:12:29.000Z"),
      lastAppliedWallTime: ISODate("2023-11-17T08:12:30.553Z"),
      lastDurableWallTime: ISODate("2023-11-17T08:12:30.553Z"),
      lastHeartbeat: ISODate("2023-11-17T08:12:30.190Z"),
      lastHeartbeatRecv: ISODate("2023-11-17T08:12:30.775Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: '<WORKER-NODE-CLUSTER-A-IP-2>:30101',
      syncSourceId: 2,
      infoMessage: '',
      configVersion: 17,
      configTerm: 2
    },
    {
      _id: 4,
      name: '<WORKER-NODE-CLUSTER-B-IP-1>:30100',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 25736,
      optime: { ts: Timestamp({ t: 1700208749, i: 2 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1700208749, i: 2 }), t: Long("2") },
      optimeDate: ISODate("2023-11-17T08:12:29.000Z"),
      optimeDurableDate: ISODate("2023-11-17T08:12:29.000Z"),
      lastAppliedWallTime: ISODate("2023-11-17T08:12:29.165Z"),
      lastDurableWallTime: ISODate("2023-11-17T08:12:29.165Z"),
      lastHeartbeat: ISODate("2023-11-17T08:12:30.370Z"),
      lastHeartbeatRecv: ISODate("2023-11-17T08:12:28.906Z"),
      pingMs: Long("596"),
      lastHeartbeatMessage: '',
      syncSourceHost: '<WORKER-NODE-CLUSTER-A-IP-2>:30101',
      syncSourceId: 2,
      infoMessage: '',
      configVersion: 17,
      configTerm: 2
    },
    {
      _id: 5,
      name: '<WORKER-NODE-CLUSTER-B-IP-2>:30101',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 21862,
      optime: { ts: Timestamp({ t: 1700208749, i: 2 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1700208749, i: 2 }), t: Long("2") },
      optimeDate: ISODate("2023-11-17T08:12:29.000Z"),
      optimeDurableDate: ISODate("2023-11-17T08:12:29.000Z"),
      lastAppliedWallTime: ISODate("2023-11-17T08:12:29.165Z"),
      lastDurableWallTime: ISODate("2023-11-17T08:12:29.165Z"),
      lastHeartbeat: ISODate("2023-11-17T08:12:30.790Z"),
      lastHeartbeatRecv: ISODate("2023-11-17T08:12:29.443Z"),
      pingMs: Long("175"),
      lastHeartbeatMessage: '',
      syncSourceHost: '<WORKER-NODE-CLUSTER-A-IP-2>:30101',
      syncSourceId: 2,
      infoMessage: '',
      configVersion: 17,
      configTerm: 2
    },
    {
      _id: 6,
      name: '<WORKER-NODE-CLUSTER-B-IP-2>:30102',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 41858,
      optime: { ts: Timestamp({ t: 1700208742, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1700208741, i: 5 }), t: Long("2") },
      optimeDate: ISODate("2023-11-17T08:12:22.000Z"),
      optimeDurableDate: ISODate("2023-11-17T08:12:21.000Z"),
      lastAppliedWallTime: ISODate("2023-11-17T08:12:22.457Z"),
      lastDurableWallTime: ISODate("2023-11-17T08:12:21.511Z"),
      lastHeartbeat: ISODate("2023-11-17T08:12:29.040Z"),
      lastHeartbeatRecv: ISODate("2023-11-17T08:12:28.382Z"),
      pingMs: Long("1115"),
      lastHeartbeatMessage: '',
      syncSourceHost: '<WORKER-NODE-CLUSTER-A-IP-1>:30100',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 17,
      configTerm: 2
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1700208750, i: 6 }),
    signature: {
      hash: Binary.createFromBase64("3pWba14pRUH9FpER+6qWF6I=", 0),
      keyId: Long("7299796089241075715")
    }
  },
  operationTime: Timestamp({ t: 1700208750, i: 6 })
}

因此，ReplicaSet 的成员被随机分配到我的 Kubernetes 集群的工作节点。目前为止，一切都好。

在Spring Boot应用程序中，spring.data.mongodb.uri的值如下：

mongodb://my_user:my_pwd@
<MASTER-NODE-CLUSTER-A-IP-1>:30100,
<MASTER-NODE-CLUSTER-A-IP-2>:30101,
<MASTER-NODE-CLUSTER-A-IP-3>:30102,
<INFRA-NODE-CLUSTER-A-IP-1>:30100,
<INFRA-NODE-CLUSTER-A-IP-2>:30101,
<INFRA-NODE-CLUSTER-A-IP-3>:30102,
<MASTER-NODE-CLUSTER-B-IP-1>:30100,
<MASTER-NODE-CLUSTER-B-IP-2>:30101,
<MASTER-NODE-CLUSTER-B-IP-3>:30102,
<INFRA-NODE-CLUSTER-B-IP-1>:30100,
<INFRA-NODE-CLUSTER-B-IP-2>:30101,
<INFRA-NODE-CLUSTER-B-IP-3>:30102
/my_db?replicaSet=rsTest&authSource=my_db

我没有连接到分配给 ReplicaSet 的节点的 IP，而是使用 NodePort 来使用 Kubernetes 集群的主节点和基础设施节点（更稳定）的 IP。

结果是，有时应用程序可以正确启动并工作，尽管我正确地看到这样的日志：

<主节点-..> 不再是副本集的成员。从客户端中删除集群视图

但由于 MongoHealthIndicator 超时，它通常不会启动：

o.s.b.a.health.HealthEndpointSupport：运行状况贡献者 org.springframework.boot.actuate.data.mongo.MongoHealthIndicator （mongo）花了 30103ms 来响应

引起：

30000 毫秒后超时，等待匹配的服务器 ReadPreferenceServerSelector{readPreference=primary}。客户端视图集群状态为 {type=REPLICA_SET，服务器=[{address=WORKER-NODE-CLUSTER-A-IP-3：30102，类型=REPLICA_SET_SECONDARY，roundTripTime=20.0 毫秒，状态=CONNECTED}， {地址=ARBITER-SVC：27017，状态=已连接}， {地址=WORKER-NODE-CLUSTER-A-IP-2：30101， type=REPLICA_SET_SECONDARY， roundTripTime=20.7 毫秒，状态 = CONNECTED}， {地址=WORKER-NODE-CLUSTER-A-IP-1：30100，类型=未知，状态=连接}， {地址=WORKER-NODE-CLUSTER-B-IP-3：30102，类型=REPLICA_SET_SECONDARY，roundTripTime=715.3 毫秒，状态=CONNECTED}， {地址=WORKER-NODE-CLUSTER-B-IP-2：30101， type=REPLICA_SET_SECONDARY， roundTripTime=131.8 毫秒，状态 = CONNECTED}， {地址=WORKER-NODE-CLUSTER-B-IP-1：30100， type=REPLICA_SET_SECONDARY， roundTripTime=793.6 毫秒，状态 = CONNECTED}]

显然，它无法连接到主数据库。

谁能为我指出正确的方向？为什么有时应用程序启动没有问题，但我经常收到这些错误并且应用程序无法启动？

提前感谢大家！

mongodb spring-boot spring-data-mongodb kubernetes-nodeport

0赞 Markus W Mahlberg 11/20/2023

首先：您的 MongoDB 集群设置没有意义。如果 DC1 发生故障，整个集群就会关闭，那么为什么首先要有第二个 DC。为什么不将仲裁器放入第三个数据中心呢？话虽如此：每个集群的 IP 地址是否彼此已知？您可以从 DC2 ping DC1 中的所有地址，反之亦然吗？

0赞 fuerteVentura22 11/21/2023

为什么在 DC1 关闭的情况下整个集群会失败？DC2 节点将成为主节点。话虽如此：是的，所有 IP 都是已知的。无论如何，回滚到 Spring Boot M2 解决了启动问题。这可能是 Spring Boot 3.2 RC2 错误。

0赞 Markus W Mahlberg 12/2/2023

Njet的。仲裁是根据初始投票节点计算得出的。如果 DC1 关闭，则剩余的投票节点（3）无法创建仲裁（4）。集群甚至告诉你：majorityVoteCount: 4, writeMajorityCount: 4, votingMembersCount: 7,

答： 暂无答案

上一个：在 Kubernetes 上运行 Spark 教程 [已关闭]

下一个：由于 Webhook 部署上的证书问题，外部密钥操作员无法创建外部密钥

Spring Boot 3.2 - 使用 NodePort 的多集群 MongoDB 连接

Spring Boot 3.2 - Multicluster MongoDB Connection with NodePort

评论