..
pkos) cd flow 구성 (harbor-gitlab-argocd) 2/2
1에서 설치했던 것에 이어 테스트를 진행했다.🔗
설정
harbor 설정
harbor에서 저장할 Project를 생성 후, 이미지를 올리면 자동으로 취약점을 스캔해주는 기능도 활성화해줬다.
projects > new project
projects > project_name > configuration > Vulnerability scanning : ☑ Automatically scan images on push
이런식으로 취약점이 있는지 스캔 후 알려준다.
# login
(nyoung:default) [root@kops-ec2 ~]# echo 'Harbor12345' > harborpw.txt
(nyoung:default) [root@kops-ec2 ~]# cat harborpw.txt | docker login harbor.nyoung.xyz -u admin --password-stdin
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
gitlab 설정
gitlab의 root권한으로 user를 생성해주고, 만들어진 user로 들어가 사용한다.
좌측상단햄버거 > Admin > Users > new user 순으로 새 user를 만들고
좌측상단햄버거 > Admin > Users > user_name > impersonation token에서 생성 후 복사해둔다.
뒤에 gke에서 사용될 user까지 두개 생성했다.
user 생성 후, 만든 user로 재로그인하여 공개 레벨을 internal로 지정한 프로젝트를 생성한다.
(nyoung:default) [root@kops-ec2 ~]# mkdir ~/gitlab-test && cd ~/gitlab-test
# 계정 초기화
(nyoung:default) [root@kops-ec2 gitlab-test]# git config --system --unset credential.helper
(nyoung:default) [root@kops-ec2 gitlab-test]# git config --global --unset credential.helper
(nyoung:default) [root@kops-ec2 gitlab-test]# git config --global user.name "nyoung"
(nyoung:default) [root@kops-ec2 gitlab-test]# git config --global user.email "08eychoi@gmail.com"
(nyoung:default) [root@kops-ec2 gitlab-test]# git clone https://gitlab.nyoung.xyz/nyoung/test-prj.git
Cloning into 'test-prj'...
Username for 'https://gitlab.nyoung.xyz': nyoung
Password for 'https://nyoung@gitlab.nyoung.xyz': $token
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
argocd 설정
# install argocd cli tool
(nyoung:default) [root@kops-ec2 ~]# curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
(nyoung:default) [root@kops-ec2 ~]# install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
(nyoung:default) [root@kops-ec2 ~]# chmod +x /usr/local/bin/argocd
# login argocd
(nyoung:default) [root@kops-ec2 ~]# argocd login argocd.nyoung.xyz --username admin --password ARGOCD_PASSWD
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
'admin:login' logged in successfully
Context 'argocd.nyoung.xyz' updated
# add git repository
(nyoung:default) [root@kops-ec2 ~]# argocd repo add https://gitlab.nyoung.xyz/nyoung/test-prj.git --username nyoung --password GIT_PASSWD
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
Repository 'https://gitlab.nyoung.xyz/nyoung/test-prj.git' added
# repo가 잘 등록된 것을 확인
(nyoung:default) [root@kops-ec2 ~]# argocd repo list --grpc-web
TYPE NAME REPO INSECURE OCI LFS CREDS STATUS MESSAGE PROJECT
git https://gitlab.nyoung.xyz/nyoung/test-prj.git false false false true Successful
# argocd가 설치된 cluster가 추가되어있는 것을 확인
# 현재는 아무런 application 이 없기에 Unknown 상태
(nyoung:default) [root@kops-ec2 ~]# argocd cluster list --grpc-web
SERVER NAME VERSION STATUS MESSAGE PROJECT
https://kubernetes.default.svc in-cluster Unknown Cluster has no applications and is not being monitored.
gke로 cluster를 하나 생성하고 위와 동일하게 cli 설치와 login을 한 뒤, cluster를 등록해줬다.
❯ argocd cluster add gke_PROJECT_ID_asia-northeast3-a_gke-cluster
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `gke_PROJECT_ID_asia-northeast3-a_gke-cluster` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0002] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0003] ClusterRole "argocd-manager-role" created
INFO[0003] ClusterRoleBinding "argocd-manager-role-binding" created
INFO[0008] Created bearer token secret for ServiceAccount "argocd-manager"
WARN[0008] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
Cluster 'https://34.64.58.154' added
❯ argocd cluster list
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
SERVER NAME VERSION STATUS MESSAGE PROJECT
https://34.64.58.154 gke_PROJECT_ID_asia-northeast3-a_gke-cluster Unknown Cluster has no applications and is not being monitored.
https://kubernetes.default.svc in-cluster
테스트
어떤걸 테스트해야할지 생각하다가 argocd에 cluster를 추가한게 무색하지만.. 간략하게 테스트 했다. harbor에 올린 이미지를 사용하는 yaml을 gitlab에 등록 후 해당 repo를 sync하는 app을 생성했다.
harbor에 이미지 올리기
# pull image
(nyoung:default) [root@kops-ec2 ~]# docker pull bsord/tetris
Using default tag: latest
latest: Pulling from bsord/tetris
ae13dd578326: Pull complete
6c0ee9353e13: Pull complete
dca7733b187e: Pull complete
352e5a6cac26: Pull complete
9eaf108767c7: Pull complete
be0c016df0be: Pull complete
802dad68ade4: Pull complete
Digest: sha256:2ffffd2f8834dd13734654eeec142aedbd95e6880e52d692a9e59fd9aa2f9634
Status: Downloaded newer image for bsord/tetris:latest
docker.io/bsord/tetris:latest
# retag image
(nyoung:default)[root@kops-ec2 ~]# docker tag bsord/tetris harbor.nyoung.xyz/pkos/tetris:0.1
(nyoung:default)[root@kops-ec2 ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
bsord/tetris latest 827801013e17 12 months ago 143MB
harbor.nyoung.xyz/pkos/tetris 0.1 827801013e17 12 months ago 143MB
# push image
(nyoung:default)[root@kops-ec2 ~]# docker push harbor.nyoung.xyz/pkos/tetris:0.1
The push refers to repository [harbor.nyoung.xyz/pkos/tetris]
07ebcd9dfb96: Pushed
24037b645d66: Pushed
d00147ef6763: Pushed
2793e885dc34: Pushed
8b8ecda1d12d: Pushed
30c00b5281a1: Pushed
3a626bb08c24: Pushed
0.1: digest: sha256:2ffffd2f8834dd13734654eeec142aedbd95e6880e52d692a9e59fd9aa2f9634 size: 1781
아주..많이 안좋은.. tetris image를 콘솔에서 확인할 수 있다.
해당 이미지를 사용한 yaml파일을 git에 업로드
~/test-prj main* 02:17:53
❯ git add .
~/test-prj main* 02:18:43
❯ git commit -m "add tetris"
[main bb746dd] add tetris
1 file changed, 32 insertions(+)
create mode 100644 tetris.yaml
~/test-prj main ⇡ 02:19:02
❯ git push
Username for 'https://gitlab.nyoung.xyz': gcpuser@nyoung.xyz
Password for 'https://gcpuser@nyoung.xyz@gitlab.nyoung.xyz':
오브젝트 나열하는 중: 4, 완료.
오브젝트 개수 세는 중: 100% (4/4), 완료.
Delta compression using up to 8 threads
오브젝트 압축하는 중: 100% (3/3), 완료.
오브젝트 쓰는 중: 100% (3/3), 487 bytes | 487.00 KiB/s, 완료.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To https://gitlab.nyoung.xyz/nyoung/test-prj.git
68cd98e..bb746dd main -> main
콘솔에서 yaml이 잘 들어간 것을 확인
argocd app 생성
(nyoung:default) [root@kops-ec2 ~]# argocd app create tetris --repo https://gitlab.nyoung.xyz/nyoung/test-prj.git --path tetris --dest-server https://kubernetes.default.svc --dest-namespace default
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
application 'tetris' created
(nyoung:default) [root@kops-ec2 ~]# argocd app list
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
argocd/tetris https://kubernetes.default.svc default default OutOfSync Missing <none> <none> https://gitlab.nyoung.xyz/nyoung/test-prj.git tetris
현재는 sync가 자동이 아니기 때문에 argocd에서 app을 생성하여 파일을 읽었지만, 실제로는 동작하지 않은 상태이다.
argocd sync
(nyoung:default) [root@kops-ec2 ~]# argocd app sync tetris
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
2023-03-25T02:27:24+09:00 Service default tetris OutOfSync Missing
2023-03-25T02:27:24+09:00 apps Deployment default tetris OutOfSync Missing
2023-03-25T02:27:24+09:00 Service default tetris Synced Healthy
2023-03-25T02:27:24+09:00 Service default tetris Synced Healthy service/tetris created
2023-03-25T02:27:24+09:00 apps Deployment default tetris OutOfSync Missing deployment.apps/tetris created
2023-03-25T02:27:24+09:00 apps Deployment default tetris Synced Progressing deployment.apps/tetris created
Name: argocd/tetris
Project: default
Server: https://kubernetes.default.svc
Namespace: default
URL: https://argocd.nyoung.xyz/applications/tetris
Repo: https://gitlab.nyoung.xyz/nyoung/test-prj.git
Target:
Path: tetris
SyncWindow: Sync Allowed
Sync Policy: <none>
Sync Status: Synced to (71db351)
Health Status: Progressing
Operation: Sync
Sync Revision: 71db351f6657ca967a72e175c84b002c2eb737fc
Phase: Succeeded
Start: 2023-03-25 02:27:24 +0900 KST
Finished: 2023-03-25 02:27:24 +0900 KST
Duration: 0s
Message: successfully synced (all tasks run)
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Service default tetris Synced Healthy service/tetris created
apps Deployment default tetris Synced Progressing deployment.apps/tetris created
# 리소스가 생성된 것을 확인 가능
(nyoung:default) [root@kops-ec2 ~]# k get po,svc
NAME READY STATUS RESTARTS AGE
pod/busybox-7846c844fb-lzth4 1/1 Running 0 24h
pod/tetris-7f86b95884-tv28x 1/1 Running 0 26s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 25h
service/tetris NodePort 100.66.131.183 <none> 80:30262/TCP 26s
# synced 상태인 것을 확인 가능
(nyoung:default) [root@kops-ec2 ~]# argocd app list
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
argocd/tetris https://kubernetes.default.svc default default Synced Healthy <none> <none> https://gitlab.nyoung.xyz/nyoung/test-prj.git tetris
automated sync
# automated sync 설정
(nyoung:default) [root@kops-ec2 ~]# argocd app set tetris --sync-policy automated
기존 yaml의 svc type을 NodePort에서 LoadBalancer로 변경하여 git과 cluster에 배포된 상태가 다르게끔 해주었다. argocd는 desired state를 바라보며 계속해서 sync하려하기 때문에, 위 설정을 해두었다면 수동으로 sync하지 않아도 자동으로 하게된다.
(nyoung:default) [root@kops-ec2 ~]# k get svc -w
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 25h
tetris LoadBalancer 100.66.131.183 <pending> 80:30262/TCP 17m
tetris LoadBalancer 100.66.131.183 af0de4e9339c641d4a0401f9f47cc428-697233320.ap-northeast-2.elb.amazonaws.com 80:30262/TCP 19m
아래 이미지를 보면 다른 것을 발견하게 된다면 sync하기 위한 progress상태로 전환된다.
lb로 배포된 서버에도 접근이 잘된다 ~
cluster를 등록해준김에 gke에도 똑같이 배포해봤다.
❯ k create ns test
namespace/test created
❯ argocd app create tetris-gke --repo https://gitlab.nyoung.xyz/nyoung/test-prj.git --path tetris --dest-server https://34.64.58.154 --dest-namespace test --sync-policy automated
application 'tetris-gke' created
# create 직후
❯ argocd app list
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
argocd/tetris https://kubernetes.default.svc default default Synced Healthy Auto <none> https://gitlab.nyoung.xyz/nyoung/test-prj.git tetris
argocd/tetris-gke https://34.64.58.154 test default Synced Progressing Auto <none> https://gitlab.nyoung.xyz/nyoung/test-prj.git tetris
# sync가 완료됨
❯ argocd app list
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
argocd/tetris https://kubernetes.default.svc default default Synced Healthy Auto <none> https://gitlab.nyoung.xyz/nyoung/test-prj.git tetris
argocd/tetris-gke https://34.64.58.154 test default Synced Healthy Auto <none> https://gitlab.nyoung.xyz/nyoung/test-prj.git tetris
❯ k get svc -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tetris LoadBalancer 10.64.8.132 34.64.209.72 80:30498/TCP 15m
콘솔에서도 확인 가능하다.
😎
참고