Kubernetes搭建spinnaker服务

Kubernetes搭建spinnaker服务

背景：

2017-2018年左右的吧，不记得看什么了看到了spinnaker，但是当时真的安装不起来。各种被墙裂。2020年底学习了泽阳大佬的[spinnaker实践课程]。通过Halyard方式搭建了spinnaker的集群，并与jenkins gitlab harbor k8s完成了集成。2021年初稍微玩了一下，就去整别的事情去了，没有能应用于线上环境。下半年了，jenkins k8s这些的流程现在基本都是清晰了。想把cd从jenkins中剥离出来教给spinnaker了，就重新温习一下spinnaker吧！

关于spinnaker

spinnaker是Netfix公司开源的一款持续部署工具，采用java语言编写，遵循微服务的设计思想，目标是为团队提供灵活的持续部署流水线并提供软件的部署效率

spinnaker的优势

Kubernetes搭建spinnaker服务

注：spinnaker的安装方式有helm 和halyard的本地部署方式 这里采用了halyard的方式！。基本过程参照泽阳大佬的spinnaker课程！本人集群环境为kubernetes1.20.6 rutime使用了containerd并没有采用docker。中间过程尝试了很多次各种失败，先基于docker的方式做一次安装部署。后面剖析一下containerd方式！

基本环境

腾讯云同一vpc内服务器，内网互通，ip为内网地址

注：个人尝试containerd运行halyard未能成功，最终使用docker方式运行halyard

基于docker runtime方式部署halyard的方式部署spinnaker

下载镜像，挂载本地配置文件目录，并启动容器

[root@k8s-01 ~]# docker pull registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 ####创建.hall文件夹后面持久化存储spinnaker生成文件 [root@k8s-01 ~]# mkdir -p /home/spinnaker/.hal ###创建.kube文件夹并将集群中的config文件上传到此目录 [root@k8s-01 ~]# mkdir -p /home/spinnaker/.kube [root@k8s-01 ~]# ls /home/spinnaker/.kube config ####启动halyard容器 [root@k8s-01 ~]# docker run -itd --name halyard -v /home/spinnaker/.hal:/home/spinnaker/.hal -v /home/spinnaker/.kube:/home/spinnaker/.kube registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0

特权身份进入容器关闭gcs

## 以root身份进入容器，修改配置文件 [root@k8s-01 .kube]# docker exec -it -u root halyard bash bash-5.0#

## 修改spinnaker.config.input.gcs.enabled = false 。 vi /opt/halyard/config/halyard.yml   spinnaker:   artifacts:     debian: https://dl.bintray.com/spinnaker-releases/debians     docker: gcr.io/spinnaker-marketplace   config:     input:       gcs:         enabled: false       writerEnabled: false       bucket: halconfig

重新启动halyard容器

## 需要重启容器（如果此命令未重启，则需要退出容器然后 docker restart halyard） bash-5.0# hal shutdown Halyard Daemon Response: Shutting down, bye... ##重启容器 [root@k8s-01 .kube]# docker start halyard halyard

上传boms文件到服务器

###通过rz命令上传制品库到运行halyard的服务器，并解压压缩包 [root@k8s-01 work]# ls 1.26.6-Install-Scripts.zip [root@k8s-01 work]# unzip 1.26.6-Install-Scripts.zip

[root@k8s-01 1.26.6]# ls .boms/ bom clouddriver deck echo fiat front50 gate igor kayenta monitoring-daemon orca rosco [root@k8s-01 1.26.6]# cp -Ra .boms/ /home/spinnaker/.hal/ [root@k8s-01 1.26.6]# ls /home/spinnaker/.hal/.boms/ bom clouddriver deck echo fiat front50 gate igor kayenta monitoring-daemon orca rosco

关于镜像的下载

镜像下载泽阳大佬的制品库下载中有下载镜像的脚本：

#!/bin/bash S_REGISTRY="gcr.io/spinnaker-marketplace" #T_REGISTRY="registry.cn-beijing.aliyuncs.com/spinnaker-cd" T_REGISTRY="docker.io/spinnakercd" NODES="node01.zy.com node02.zy.com" ## 下载镜像 function GetImages(){ echo -e "\033[43;34m =====GetImg===== \033[0m" IMAGES=$( cat tagfile.txt) for image in ${IMAGES} do for node in ${NODES} do echo -e "\033[32m ${node} ---> pull ---> ${image} \033[0m" ssh ${node} "docker pull ${T_REGISTRY}/${image}" echo -e "\033[32m ${node} ---> tag ---> ${image} \033[0m" ssh ${node} "docker tag ${T_REGISTRY}/${image} ${S_REGISTRY}/${image}" done done for node in ${NODES} do echo -e "\033[43;34m =====${node}===镜像信息===== \033[0m" ssh ${node} "docker images | grep 'spinnaker-marketplace' " done } GetImages

But 我的集群的运行时是containerd。ctr crictl两个命令的区别有必要重新温习一下。crictl也没法修改标签啊？

#!/bin/bash S_REGISTRY="gcr.io/spinnaker-marketplace" #T_REGISTRY="registry.cn-beijing.aliyuncs.com/spinnaker-cd" T_REGISTRY="docker.io/spinnakercd" NODES="10.0.4.18 10.0.4.49 10.0.4.48 10.0.4.23 10.0.4.47 10.0.4.32" ## 下载镜像 function GetImages(){ echo -e "\033[43;34m =====GetImg===== \033[0m" IMAGES=$( cat tagfile.txt) for image in ${IMAGES} do for node in ${NODES} do echo -e "\033[32m ${node} ---> pull ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl pull ${T_REGISTRY}/${image}" echo -e "\033[32m ${node} ---> tag ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl images ${T_REGISTRY}/${image} ${S_REGISTRY}/${image}" done done for node in ${NODES} do echo -e "\033[43;34m =====${node}===镜像信息===== \033[0m" ssh -p 36000 ${node} "crictl images ls| grep 'spinnaker-marketplace' " done } GetImages

[root@k8s-2 .hal]# mkdir -p /home/spinnaker/.hal/default/service-settings [root@k8s-2 .hal]# cd /home/spinnaker/.hal/default/service-settings [root@k8s-2 service-settings]# pwd /home/spinnaker/.hal/default/service-settings [root@k8s-2 service-settings]# ls clouddriver.yml deck.yml echo.yml fiat.yml front50.yml gate.yml igor.yml kayenta.yml orca.yml rosco.yml [root@k8s-2 service-settings]# cat * artifactId: docker.io/spinnakercd/clouddriver:8.0.4-20210625060028 artifactId: docker.io/spinnakercd/deck:3.7.2-20210614020020 artifactId: docker.io/spinnakercd/echo:2.17.1-20210429125836 artifactId: docker.io/spinnakercd/fiat:1.16.0-20210422230020 artifactId: docker.io/spinnakercd/front50:0.27.1-20210625161956 artifactId: docker.io/spinnakercd/gate:1.22.1-20210603020019 artifactId: docker.io/spinnakercd/igor:1.16.0-20210422230020 artifactId: docker.io/spinnakercd/kayenta:0.21.0-20210322140019 artifactId: docker.io/spinnakercd/orca:2.20.3-20210630022216 artifactId: docker.io/spinnakercd/rosco:0.25.0-20210422230020

Halyard配置管理

注： halyard的配置都在k8s-01节点执行默认在halyard容器内

设置Spinnaker版本，--version 指定版本

[root@k8s-01 .kube]# docker exec -it -u root halyard bash bash-5.0$ hal config version edit --version local:1.26.6 + Get current deployment Success - Edit Spinnaker version Failure Validation in Global: ! ERROR Failure writing your halconfig to path "/home/spinnaker/.hal/config": /home/spinnaker/.hal/config - Failed to update version.

[root@k8s-01 1.26.6]# chmod 777 -R /home/spinnaker/.hal/ [root@k8s-01 1.26.6]#

继续指定spinnaker版本并生成配置文件

bash-5.0$ hal config version edit --version local:1.26.6 + Get current deployment Success + Edit Spinnaker version Success + Spinnaker has been configured to update/install version "local:1.26.6". Deploy this version of Spinnaker with `hal deploy apply`. bash-5.0$ ls config default bash-5.0$ cat config currentDeployment: default deploymentConfigurations: - name: default version: local:1.26.6 providers: appengine: enabled: false accounts: [] aws: enabled: false accounts: [] bakeryDefaults: baseImages: [] defaultKeyPairTemplate: '{{name}}-keypair' defaultRegions: - name: us-west-2 defaults: iamRole: BaseIAMRole ecs: enabled: false accounts: [] azure: enabled: false accounts: [] bakeryDefaults: templateFile: azure-linux.json baseImages: [] dcos: enabled: false accounts: [] clusters: [] dockerRegistry: enabled: false accounts: [] google: enabled: false accounts: [] bakeryDefaults: templateFile: gce.json baseImages: [] zone: us-central1-f network: default useInternalIp: false huaweicloud: enabled: false accounts: [] bakeryDefaults: baseImages: [] kubernetes: enabled: false accounts: [] tencentcloud: enabled: false accounts: [] bakeryDefaults: baseImages: [] oracle: enabled: false accounts: [] bakeryDefaults: templateFile: oci.json baseImages: [] cloudfoundry: enabled: false accounts: [] deploymentEnvironment: size: SMALL type: LocalDebian imageVariant: SLIM updateVersions: true consul: enabled: false vault: enabled: false customSizing: {} sidecars: {} initContainers: {} hostAliases: {} affinity: {} tolerations: {} nodeSelectors: {} gitConfig: upstreamUser: spinnaker livenessProbeConfig: enabled: false haServices: clouddriver: enabled: false disableClouddriverRoDeck: false echo: enabled: false persistentStorage: azs: {} gcs: rootFolder: front50 redis: {} s3: rootFolder: front50 oracle: {} features: auth: false fiat: false chaos: false entityTags: false metricStores: datadog: enabled: false tags: [] prometheus: enabled: false add_source_metalabels: true stackdriver: enabled: false newrelic: enabled: false tags: [] period: 30 enabled: false notifications: slack: enabled: false twilio: enabled: false baseUrl: https://api.twilio.com/ github-status: enabled: false timezone: America/Los_Angeles ci: jenkins: enabled: false masters: [] travis: enabled: false masters: [] wercker: enabled: false masters: [] concourse: enabled: false masters: [] gcb: enabled: false accounts: [] codebuild: enabled: false accounts: [] repository: artifactory: enabled: false searches: [] security: apiSecurity: ssl: enabled: false uiSecurity: ssl: enabled: false authn: oauth2: enabled: false client: {} resource: {} userInfoMapping: {} saml: enabled: false userAttributeMapping: {} ldap: enabled: false x509: enabled: false iap: enabled: false enabled: false authz: groupMembership: service: EXTERNAL google: roleProviderType: GOOGLE github: roleProviderType: GITHUB file: roleProviderType: FILE ldap: roleProviderType: LDAP enabled: false artifacts: bitbucket: enabled: false accounts: [] gcs: enabled: false accounts: [] oracle: enabled: false accounts: [] github: enabled: false accounts: [] gitlab: enabled: false accounts: [] gitrepo: enabled: false accounts: [] http: enabled: false accounts: [] helm: enabled: false accounts: [] s3: enabled: false accounts: [] maven: enabled: false accounts: [] templates: [] pubsub: enabled: false google: enabled: false pubsubType: GOOGLE subscriptions: [] publishers: [] canary: enabled: false serviceIntegrations: - name: google enabled: false accounts: [] gcsEnabled: false stackdriverEnabled: false - name: prometheus enabled: false accounts: [] - name: datadog enabled: false accounts: [] - name: signalfx enabled: false accounts: [] - name: aws enabled: false accounts: [] s3Enabled: false - name: newrelic enabled: false accounts: [] reduxLoggerEnabled: true defaultJudge: NetflixACAJudge-v1.0 stagesEnabled: true templatesEnabled: true showAllConfigsEnabled: true spinnaker: extensibility: plugins: {} repositories: {} webhook: trust: enabled: false stats: enabled: true endpoint: https://stats.spinnaker.io instanceId: 01FKDR1B3P8PF35RRC93XTE9AS deploymentMethod: {} connectionTimeoutMillis: 3000 readTimeoutMillis: 5000 bash-5.0$

设置时区

# 设置时区 hal config edit --timezone Asia/Shanghai

S3--no-validate

# 设置存储为s3(后面不用，但是必须配置bug) hal config storage edit --type s3 --no-validate

访问方式,设置deck与gate的域名

# 访问方式：设置deck与gate的域名 hal config security ui edit --override-base-url http://spinnaker.xxxx.com hal config security api edit --override-base-url hal config provider docker-registry enable --no-validate + Get current deployment Success + Edit the dockerRegistry provider Success + Successfully enabled dockerRegistry bash-5.0$ hal config provider docker-registry account add my-harbor-registry \ > --address \ > --username xxxx \ > --password xxxx + Get current deployment Success + Add the my-harbor-registry account Success Validation in default.provider.dockerRegistry.my-harbor-registry: - WARNING Your docker registry has no repositories specified, and the registry's catalog is empty. Spinnaker will not be able to deploy any images until some are pushed to this registry. ? Manually specify some repositories for this docker registry to index. + Successfully added account my-harbor-registry for provider dockerRegistry.

开启kubernetes配置并添加account

bash-5.0$ hal config provider kubernetes enable + Get current deployment Success + Edit the kubernetes provider Success Validation in default.provider.kubernetes: - WARNING Provider kubernetes is enabled, but no accounts have been configured. + Successfully enabled kubernetes bash-5.0$ hal config provider kubernetes account add default \ > --docker-registries my-harbor-registry \ > --context $(kubectl config current-context) \ > --service-account true \ > --omit-namespaces=kube-system,kube-public \ > --provider-version v2 \ > --no-validate + Get current deployment Success + Add the default account Success + Successfully added account default for provider kubernetes.

指定部署使用account和命名空间，部署方式distributed（分布式）

​

bash-5.0$ hal config deploy edit \ > --account-name default \ > --type distributed \ > --location spinnaker

开启一些主要的功能（后期可以再追加）

bash-5.0$ hal config features edit --pipeline-templates true bash-5.0$ hal config features edit --artifacts true bash-5.0$ hal config features edit --managed-pipeline-templates-v2-ui true

配置与jenkins CI集成

# 配置Jenkins hal config ci jenkins enable ### JenkinsServer 需要用到账号和密码 hal config ci jenkins master add my-jenkins-master-01 \ --address \ --username zhangpeng \ --password xxxx ### 启用csrf hal config ci jenkins master edit my-jenkins-master-01 --csrf true

配置GitHub/GitLab集成

github的是泽阳大佬的。我这里就只集成了gitlab。github仅供参考在配置文件中也生成一下。方便对比配置文件。token的生成就不用做过多的赘述了！

# GitHub ## 参考：https://spinnaker.io/setup/artifacts/github/ ## 创建token https://github.com/settings/tokens hal config artifact github enable hal config artifact github account add my-github-account \ --token xxxxxxxxxxxxxxxxxxxxxxx \ --username zeyangli # GitLab ## https://spinnaker.io/setup/artifacts/gitlab/ ## 创建一个个人的token（admin） hal config artifact gitlab enable hal config artifact gitlab account add my-gitlab-account \ --token xxxxxxxxxxxxxx

使用外部redis集群

关于redis我是使用的腾讯云的云redis。正常该搞一个密码的。但是没有去仔细看下官方文档，就直接使用了免密的方式！

## service-settings bash-5.0$ pwd /home/spinnaker/.hal/default/service-settings vi .hal/default/service-settings/redis.yml overrideBaseUrl: redis://10.0.0.31:6379 skipLifeCycleManagement: true ## profiles ## /home/spinnaker/.hal/default/profiless bash-5.0$ pwd /home/spinnaker/.hal/default bash-5.0$ mkdir /home/spinnaker/.hal/default/profiles bash-5.0$ cd profiles/ bash-5.0$ vi gate-local.yml redis: configuration: secure: true

使用SQL数据库

Clouddriver服务

创建数据库：

CREATE DATABASE `clouddriver` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW ON `clouddriver`.* TO 'clouddriver_service'@'%' IDENTIFIED BY 'clouddriver@spinnaker.com'; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW ON `clouddriver`.* TO 'clouddriver_migrate'@'%' IDENTIFIED BY 'clouddriver@spinnaker.com';

修改配置文件：

bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ vi clouddriver-local.yml sql: enabled: true # read-only boolean toggles `SELECT` or `DELETE` health checks for all pools. # Especially relevant for clouddriver-ro and clouddriver-ro-deck which can # target a SQL read replica in their default pools. read-only: false taskRepository: enabled: true cache: enabled: true # These parameters were determined to be optimal via benchmark comparisons # in the Netflix production environment with Aurora. Setting these too low # or high may negatively impact performance. These values may be sub-optimal # in some environments. readBatchSize: 500 writeBatchSize: 300 scheduler: enabled: true # Enable clouddriver-caching's clean up agent to periodically purge old # clusters and accounts. Set to true when using the Kubernetes provider. unknown-agent-cleanup-agent: enabled: false connectionPools: default: # additional connection pool parameters are available here, # for more detail and to view defaults, see: # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt default: true jdbcUrl: jdbc:mysql://10.0.4.22:3306/clouddriver user: clouddriver_service password: clouddriver@spinnaker.com # The following tasks connection pool is optional. At Netflix, clouddriver # instances pointed to Aurora read replicas have a tasks pool pointed at the # master. Instances where the default pool is pointed to the master omit a # separate tasks pool. tasks: user: clouddriver_service jdbcUrl: jdbc:mysql://10.0.4.22:3306/clouddriver password: clouddriver@spinnaker.com migration: user: clouddriver_migrate jdbcUrl: jdbc:mysql://10.0.4.22:3306/clouddriver password: clouddriver@spinnaker.com redis: enabled: false cache: enabled: false scheduler: enabled: false taskRepository: enabled: false

Front50服务

创建数据库

CREATE DATABASE `front50` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW ON `front50`.* TO 'front50_service'@'%' IDENTIFIED BY "front50@spinnaker.com"; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW ON `front50`.* TO 'front50_migrate'@'%' IDENTIFIED BY "front50@spinnaker.com";

修改配置文件

bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ vi front50-local.yml spinnaker: s3: enabled: false sql: enabled: true connectionPools: default: # additional connection pool parameters are available here, # for more detail and to view defaults, see: # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt default: true jdbcUrl: jdbc:mysql://10.0.4.22:3306/front50 user: front50_service password: front50@spinnaker.com migration: user: front50_migrate jdbcUrl: jdbc:mysql://10.0.4.22:3306/front50 password: front50@spinnaker.com

Orca服务

创建数据库

set tx_isolation = 'REPEATABLE-READ'; CREATE SCHEMA `orca` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW ON `orca`.* TO 'orca_service'@'%' IDENTIFIED BY "orca@spinnaker.com" ; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW ON `orca`.* TO 'orca_migrate'@'%' IDENTIFIED BY "orca@spinnaker.com" ;

修改配置文件

bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ vi front50-local.yml bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ vi orca-local.yml tasks: useManagedServiceAccounts: true sql: enabled: true connectionPool: jdbcUrl: jdbc:mysql://10.0.4.22:3306/orca user: orca_service password: orca@spinnaker.com connectionTimeout: 5000 maxLifetime: 30000 # MariaDB-specific: maxPoolSize: 50 migration: jdbcUrl: jdbc:mysql://10.0.4.22:3306/orca user: orca_migrate password: orca@spinnaker.com # Ensure we're only using SQL for accessing execution state executionRepository: sql: enabled: true redis: enabled: false # Reporting on active execution metrics will be handled by SQL monitor: activeExecutions: redis: false # Use SQL for Orca's work queue # Settings from Netflix and may require adjustment for your environment # Only validated with AWS Aurora MySQL 5.7 # Please PR if you have success with other databases keiko: queue: sql: enabled: true redis: enabled: false queue: zombieCheck: enabled: true pendingExecutionService: sql: enabled: true redis: enabled: false

部署服务

bash-5.0$ hal deploy apply --no-validate

创建Ingress访问web测试

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: spinnaker-service namespace: spinnaker annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/router.entrypoints: web spec: rules: - host: spinnaker.xxxx.com http: paths: - pathType: Prefix path: / backend: service: name: spin-deck port: number: 9000 - host: spin-gate.xxxx.com http: paths: - pathType: Prefix path: / backend: service: name: spin-gate port: number: 8084

集成ldap:

hal config security authn ldap edit \ --user-search-base 'ou=devops,dc=zy,dc=com' \ --url 'ldap://192.168.1.200:389' \ --user-search-filter 'cn={0}' \ --manager-dn 'cn=admin,dc=zy,dc=com' \ --manager-password '12345678' hal config security authn ldap enable

bash-5.0$ cd /home/spinnaker/.hal/ bash-5.0$ pwd /home/spinnaker/.hal bash-5.0$ cat config

bash-5.0$ hal deploy apply --no-validate

[root@k8s-master-01 ~]# kubectl get pods -n spinnaker

关于授权

首先登陆ldap web管理页面两个用户组 groupOfUniqueNames yunwenzu devops两个组,根据ldap中组进行授权。

ldap创建用户组与用户

yunweizu-用户zhangpeng

devop用户组-用户huozhonghao

halyard中配置：

开启ldap security 配置。并增加相关配置：

hal config security authz ldap edit \ --url 'ldap://172.19.252.28:389/dc=xxxx,dc=com' \ --manager-dn 'cn=admin,dc=xxxx,dc=com' \ --manager-password 'xxxxxx' \ --user-dn-pattern 'cn={0}' \ --group-search-base 'ou=devops' \ --group-search-filter 'uniqueMember={0}' \ --group-role-attributes 'cn' \ --user-search-filter 'cn={0}' hal config security authz edit --type ldap hal config security authz enable

设置那些用户可以访问集群账户、镜像仓库、应用程序

## 配置yunweizu和group02角色的用户可以使用default这个集群账户 hal config provider kubernetes account edit default \ --add-read-permission yunweizu,group02  \ --add-write-permission yunweizu    ## 配置yunweizu角色的用户可以使用my-harbor-registry账户 hal config provider docker-registry account edit my-harbor-registry \ --read-permissions yunweizu \ --write-permissions yunweizu ##更新部署 hal deploy apply

登陆spinnaker web尝试：

创建一个新的用户组platform将huozhonghao用户加入

spinnaker web登陆huozhonghao用户

开启管道权限

halyard容器中操作：

bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ cat /home/spinnaker/.hal/default/profiles/orca-local.yml tasks: useManagedServiceAccounts: true bash-5.0$ cat ~/.hal/default/profiles/settings-local.js window.spinnakerSettings.feature.managedServiceAccounts = true; bash-5.0$ hal deploy apply --no-validate

权限的一些测试

bash-5.0$ hal deploy apply --no-validate

[root@k8s-master-01 develop]# kubectl get pods -n spinnaker

[root@k8s-master-01 develop]#kubectl get svc -n spinnaker [root@k8s-master-01 develop]# curl -X POST http://172.19.254.33:7003/roles/sync [root@k8s-master-01 develop]#curl 172.19.254.33:7003/authorize/huozhonghao

bash-5.0$ vi config bash-5.0$ hal deploy apply --no-validate

一些失败的尝试（还是没有成功）

1. 下载Halyard 镜像并启动容器---ctr各种命令的复习

ctr pull

[root@k8s-master-01 ~]# ctr image pull registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 [root@k8s-master-01 ~]# mkdir /root/.hal

docker run -itd --name halyard \ -v /root/.hal:/home/spinnaker/.hal \ -v /root/.kube:/home/spinnaker/.kube \ registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0

ctr run

依着葫芦画瓢一下？

ctr run -itd --name halyard \ -v /root/.hal:/home/spinnaker/.hal \ -v /root/.kube:/home/spinnaker/.kube \ registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0

中间尝试了很多次各种 ctr命令确实没有搞明白......参考了使用ctr 命令管理 Containerd 容器我觉得使用containerd安装spinnaker 这真的是可以复习ctr critical命令了

ctr create

[root@k8s-master-01 1.26.6]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/root/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 1.26.6]# ctr c ls CONTAINER IMAGE RUNTIME halyard registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 io.containerd.runc.v2

ctr t start

[root@k8s-master-01 1.26.6]# ctr t start -d halyard [root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS halyard 1729924 RUNNING

ctr tasks exec -t --exec-id

[root@k8s-master-01 1.26.6]# ctr tasks list TASK PID STATUS halyard 1729924 RUNNING [root@k8s-master-01 1.26.6]# ctr tasks exec -t --exec-id 1729924 halyard sh / $

ctr c rm ctr c kill----读写权限没有搞明白 只能采用挂载本地文件的方式重新搞一波了

[root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS halyard 4184764 RUNNING [root@k8s-master-01 1.26.6]# ctr t kill halyard [root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS halyard 4184764 STOPPED [root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS halyard 4184764 STOPPED [root@k8s-master-01 1.26.6]# ctr c rm halyard [root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS

[root@k8s-master-01 1.26.6]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/root/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 1.26.6]# ctr c ls CONTAINER IMAGE RUNTIME halyard registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 io.containerd.runc.v2 [root@k8s-master-01 1.26.6] # ctr t start -d halyard [root@k8s-master-01 1.26.6] # ctr t ls TASK PID STATUS halyard 1729924 RUNNING [root@k8s-master-01 1.26.6] # ctr tasks exec -t --exec-id 1729924 halyard sh

下载镜像的尝试：

小伙伴们觉得下载镜像应该用下面哪个脚本？用ctr or crictl呢？最终使用镜像的是要kubernetes....应该是用crictl的。 ctr搞了kubernetes集群应用是发现不了镜像的！

#!/bin/bash S_REGISTRY="gcr.io/spinnaker-marketplace" #T_REGISTRY="registry.cn-beijing.aliyuncs.com/spinnaker-cd" T_REGISTRY="docker.io/spinnakercd" NODES="10.0.4.18 10.0.4.49 10.0.4.48 10.0.4.23 10.0.4.47 10.0.4.32" ## 下载镜像 function GetImages(){ echo -e "\033[43;34m =====GetImg===== \033[0m" IMAGES=$( cat tagfile.txt) for image in ${IMAGES} do for node in ${NODES} do echo -e "\033[32m ${node} ---> pull ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl pull ${T_REGISTRY}/${image}" echo -e "\033[32m ${node} ---> tag ---> ${image} \033[0m" ssh -p 36000 ${node} "ctr image tag ${T_REGISTRY}/${image} ${S_REGISTRY}/${image}" done done for node in ${NODES} do echo -e "\033[43;34m =====${node}===镜像信息===== \033[0m" ssh -p 36000 ${node} "ctr image ls | grep 'spinnaker-marketplace' " done } GetImages

#!/bin/bash S_REGISTRY="gcr.io/spinnaker-marketplace" #T_REGISTRY="registry.cn-beijing.aliyuncs.com/spinnaker-cd" T_REGISTRY="docker.io/spinnakercd" NODES="10.0.4.18 10.0.4.49 10.0.4.48 10.0.4.23 10.0.4.47 10.0.4.32" ## 下载镜像 function GetImages(){ echo -e "\033[43;34m =====GetImg===== \033[0m" IMAGES=$( cat tagfile.txt) for image in ${IMAGES} do for node in ${NODES} do echo -e "\033[32m ${node} ---> pull ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl pull ${T_REGISTRY}/${image}" echo -e "\033[32m ${node} ---> tag ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl images ${T_REGISTRY}/${image} ${S_REGISTRY}/${image}" done done for node in ${NODES} do echo -e "\033[43;34m =====${node}===镜像信息===== \033[0m" ssh -p 36000 ${node} "crictl images ls| grep 'spinnaker-marketplace' " done } GetImages

当然了还有一个问题就是 crictl 可以更改镜像名字吗？貌似是不可以的...然后此方式就失败了。

各种失败的尝试-containerd下：

[root@k8s-master-01 .boms]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/root/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 .boms]# ctr c ls CONTAINER IMAGE RUNTIME halyard registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 io.containerd.runc.v2 [root@k8s-master-01 .boms]# ctr t start -d halyard [root@k8s-master-01 .boms]# ctr t ls TASK PID STATUS halyard 1775521 RUNNING [root@k8s-master-01 .boms]# ctr tasks exec -t --exec-id 1729924 halyard sh / $ hal config version edit --version local:1.26.6 ~ $ cd /home/spinnaker/.hal/ vi config

timezone: America/Los_Angeles timezone: Asia/Shanghai

hal config storage edit --type s3 --no-validate

hal config security ui edit --override-base-url http://spinnaker.layame.com hal config security api edit --override-base-url .boms]# ctr t kill --signal 9 halyard [root@k8s-master-01 .boms]# ctr c rm halyard

[root@k8s-master-01 .boms]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/root/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 .boms]# ctr c ls CONTAINER IMAGE RUNTIME halyard registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 io.containerd.runc.v2 [root@k8s-master-01 .boms]# ctr t start -d halyard [root@k8s-master-01 .boms]# ctr t ls TASK PID STATUS halyard 1832934 RUNNING [root@k8s-master-01 .boms]# ctr tasks exec -t --exec-id 1832934 halyard sh ~ $ cd /home/spinnaker/.hal/ ~/.hal $ cat config |grep time timezone: Asia/Shanghai ~/.hal $ cat config |grep s3 persistentStoreType: s3 s3: s3: s3Enabled: true ~/.hal $ cat config |grep com baseUrl: https://api.twilio.com/ overrideBaseUrl: http://spin-gate.layame.com overrideBaseUrl: $ hal config provider kubernetes enable ~/.hal $ hal config provider kubernetes account add default \ --docker-registries my-harbor-registry \ --context $(kubectl config current-context) \ --service-account true \ --omit-namespaces=kube-system,kube-public \ --provider-version v2 \ --no-validate

hal config deploy edit \ --account-name default \ --type distributed \ --location spinnaker

hal config features edit --pipeline-templates true hal config features edit --artifacts true hal config features edit --managed-pipeline-templates-v2-ui true

[root@k8s-node-01 home]# mkdir -p /home/spinnaker/.hal [root@k8s-node-01 home]# mkdir -p /opt/halyard/config [root@k8s-node-01 home]# mkdir -p /home/spinnaker/.kube [root@k8s-node-01 home]# crictl pull registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 Image is up to date for sha256:8673f1670b8768138cd8349b7d9843eb4fd451658227d2e9f02d5fbe454c500d [root@k8s-node-01 home]# cd /home/spinnaker/.kube [root@k8s-node-01 .kube]# rz [root@k8s-node-01 .kube]# ls config [root@k8s-node-01 .kube]# ctr image pull registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 [root@k8s-node-01 .kube]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/home/spinnaker/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/home/spinnaker/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw

[root@k8s-node-01 .boms]# pwd /home/spinnaker/.hal/.boms [root@k8s-node-01 .boms]# ls bom clouddriver deck echo fiat front50 gate igor kayenta monitoring-daemon orca rosco [root@k8s-node-01 .boms]# cd /opt/halyard/config/ [root@k8s-node-01 config]# cat halyard.yaml

[root@k8s-node-01 ~]# ctr t ls TASK PID STATUS [root@k8s-node-01 ~]# ctr t start -d halyard [root@k8s-node-01 ~]# ctr t ls TASK PID STATUS halyard 3910255 RUNNING [root@k8s-node-01 ~]# ctr tasks exec -t --exec-id 3910255 halyard sh / $ hal config version edit --version local:1.26.6 + Get current deployment Success - Edit Spinnaker version Failure Validation in Global: ! ERROR Failure writing your halconfig to path "/home/spinnaker/.hal/config": /home/spinnaker/.hal/config - Failed to update version. / $ hal config version edit --version local:1.26.6 + Get current deployment Success + Edit Spinnaker version Success + Spinnaker has been configured to update/install version "local:1.26.6". Deploy this version of Spinnaker with `hal deploy apply`. / $ hal config edit --timezone Asia/Shanghai ********又tmd sb了 不知道怎么回事不试了。直接改好配置文件直接启动了！

总结以上失败 执行啥也不行...最后决定直接把docker环境面config文件以及其他制品搞过来试试！

my config文件：

currentDeployment: default deploymentConfigurations: - name: default version: local:1.26.6 providers: appengine: enabled: false accounts: [] aws: enabled: false accounts: [] bakeryDefaults: baseImages: [] defaultKeyPairTemplate: '{{name}}-keypair' defaultRegions: - name: us-west-2 defaults: iamRole: BaseIAMRole ecs: enabled: false accounts: [] azure: enabled: false accounts: [] bakeryDefaults: templateFile: azure-linux.json baseImages: [] dcos: enabled: false accounts: [] clusters: [] dockerRegistry: enabled: true accounts: - name: my-harbor-registry requiredGroupMembership: [] providerVersion: V1 permissions: READ: - yunweizu WRITE: - yunweizu address: https://harbor.layame.com username: zhangpeng password: xxxx email: fake.email@spinnaker.io cacheIntervalSeconds: 30 clientTimeoutMillis: 60000 cacheThreads: 1 paginateSize: 100 sortTagsByDate: false trackDigests: false insecureRegistry: false repositories: [] primaryAccount: my-harbor-registry google: enabled: false accounts: [] bakeryDefaults: templateFile: gce.json baseImages: [] zone: us-central1-f network: default useInternalIp: false huaweicloud: enabled: false accounts: [] bakeryDefaults: baseImages: [] kubernetes: enabled: true accounts: - name: default requiredGroupMembership: [] providerVersion: V2 permissions: READ: - yunweizu,group02  - devops WRITE: - yunweizu - devops dockerRegistries: - accountName: my-harbor-registry namespaces: [] context: kubernetes-admin@kubernetes configureImagePullSecrets: true serviceAccount: true cacheThreads: 1 namespaces: [] omitNamespaces: - kube-system - kube-public kinds: [] omitKinds: [] customResources: [] cachingPolicies: [] oAuthScopes: [] onlySpinnakerManaged: false primaryAccount: default tencentcloud: enabled: false accounts: [] bakeryDefaults: baseImages: [] oracle: enabled: false accounts: [] bakeryDefaults: templateFile: oci.json baseImages: [] cloudfoundry: enabled: false accounts: [] deploymentEnvironment: size: SMALL type: Distributed accountName: default imageVariant: SLIM updateVersions: true consul: enabled: false vault: enabled: false location: spinnaker customSizing: {} sidecars: {} initContainers: {} hostAliases: {} affinity: {} tolerations: {} nodeSelectors: {} gitConfig: upstreamUser: spinnaker livenessProbeConfig: enabled: false haServices: clouddriver: enabled: false disableClouddriverRoDeck: false echo: enabled: false persistentStorage: persistentStoreType: s3 azs: {} gcs: rootFolder: front50 redis: {} s3: rootFolder: front50 oracle: {} features: auth: false fiat: false chaos: false entityTags: false pipelineTemplates: true artifacts: true managedPipelineTemplatesV2UI: true metricStores: datadog: enabled: false tags: [] prometheus: enabled: false add_source_metalabels: true stackdriver: enabled: false newrelic: enabled: false tags: [] period: 30 enabled: false notifications: slack: enabled: false twilio: enabled: false baseUrl: https://api.twilio.com/ github-status: enabled: false timezone: Asia/Shanghai ci: jenkins: enabled: true masters: - name: my-jenkins-master-01 permissions: {} address: https://jenkins.xxxx.com username: zhangpeng password: xxxxx csrf: true travis: enabled: false masters: [] wercker: enabled: false masters: [] concourse: enabled: false masters: [] gcb: enabled: false accounts: [] codebuild: enabled: false accounts: [] repository: artifactory: enabled: false searches: [] security: apiSecurity: ssl: enabled: false overrideBaseUrl: https://spin-gate.xxxx.com uiSecurity: ssl: enabled: false overrideBaseUrl: https://spinnaker.xxxx.com authn: oauth2: enabled: false client: {} resource: {} userInfoMapping: {} saml: enabled: false userAttributeMapping: {} ldap: enabled: true url: ldap://172.19.252.28:389 userSearchBase: ou=devops,dc=xxxx,dc=com userSearchFilter: cn={0} managerDn: cn=admin,dc=xxxx,dc=com managerPassword: xxxx x509: enabled: false iap: enabled: false enabled: true authz: groupMembership: service: LDAP google: roleProviderType: GOOGLE github: roleProviderType: GITHUB file: roleProviderType: FILE path: /home/spinnaker/.hal/userrole.yml ldap: roleProviderType: LDAP url: ldap://172.19.252.28:389/dc=xxxx,dc=com managerDn: cn=admin,dc=xxxx,dc=com managerPassword: xxxx userDnPattern: cn={0} groupSearchBase: ou=devops userSearchFilter: cn={0} groupSearchFilter: uniqueMember={0} groupRoleAttributes: cn enabled: true artifacts: bitbucket: enabled: false accounts: [] gcs: enabled: false accounts: [] oracle: enabled: false accounts: [] github: enabled: true accounts: - name: my-github-account username: zeyangli token: xxxx gitlab: enabled: true accounts: - name: my-gitlab-account token: xxxx gitrepo: enabled: false accounts: [] http: enabled: false accounts: [] helm: enabled: false accounts: [] s3: enabled: false accounts: [] maven: enabled: false accounts: [] templates: [] pubsub: enabled: false google: enabled: false pubsubType: GOOGLE subscriptions: [] publishers: [] canary: enabled: false serviceIntegrations: - name: google enabled: false accounts: [] gcsEnabled: false stackdriverEnabled: false - name: prometheus enabled: false accounts: [] - name: datadog enabled: false accounts: [] - name: signalfx enabled: false accounts: [] - name: aws enabled: false accounts: [] s3Enabled: false - name: newrelic enabled: false accounts: [] reduxLoggerEnabled: true defaultJudge: NetflixACAJudge-v1.0 stagesEnabled: true templatesEnabled: true showAllConfigsEnabled: true spinnaker: extensibility: plugins: {} repositories: {} webhook: trust: enabled: false stats: enabled: true endpoint: https://stats.spinnaker.io instanceId: 01FKDR1B3P8PF35RRC93XTE9AS deploymentMethod: {} connectionTimeoutMillis: 3000 readTimeoutMillis: 5000

继续

[root@k8s-master-01 .kube]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/home/spinnaker/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 .kube]# ctr t start -d halyard [root@k8s-master-01 .kube]# ctr t ls TASK PID STATUS halyard 3073271 RUNNING [root@k8s-master-01 .kube]# ctr tasks exec -t --exec-id 3073271 halyard sh bash-5.0$ hal deploy apply --no-validate

​

[root@k8s-master-01 .kube]# ctr t kill --signal 9 halyard [root@k8s-master-01 .kube]# ctr c rm halyard

[root@k8s-master-01 .hal]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/home/spinnaker/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 .hal]# ctr t start -d halyard [root@k8s-master-01 .hal]# ctr t ls TASK PID STATUS halyard 3085723 RUNNING [root@k8s-master-01 .hal]# ctr tasks exec -t --exec-id 3085723 halyard bash bash-5.0$

算了我放弃了......，containerd的安装方式

总结一下失败以及经验：

containerd or docker的运行时中都可以在文件夹 /home/spinnaker/.hal/default/service-settings本地写文件的件方式指定image tag，docker环境下还好，containerd方式下crictl 修改镜像标签自己掌握的不是很好！ containerd命令跟docker还是不一样。启动halyard的方式还是很不好弄，最好的方式还是在一台安装docker的机器上面运行halyard。 halyard执行脚本复制命令的空格格式问题 部署过程中出现数据库地址写错问题...写成了TDSQL-C中的读地址....

​

**

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。