[DevOps] CD, Routing 컨트롤을 통한 Rollout 방안

• 셋탑 위치(지역) 기반
• 테스트용 vs 일반 사용자
• 요금별(유료 고객 vs 무료 고객)
• Feature Flag
• Ramdom(Weighted)

 

 

Flow Diagram

구성의 예>

 

 

API Path에 따라 stable(blue) 또는 preview(green)으로 라우팅되도록 구성하였습니다.

롤 아웃 진행중에 preview(green) 환경으로 트래픽을 보내기 위해서는 특정 트래픽의 요청 path를 변경해야 합니다.(API GW에서 구성)

 

<Random Routing>

 

Random Routing

 

 

모든 POD를 새버젼(Green Pod)로 보내기 전에 weight에 따라 stable/preview 환경으로 트래픽 전달

특정 시간동안 일부 트래픽을 랜덤으로 Green 환경으로 배포 후 요청 사항을 수집하여 자동/수동으로 Promotion 진행

 

<특정 대상군 별 라우팅>

특정 대상군별 라우팅

인증 POD에서 인증정보(said, macAddr)을 통해 해당 유저를 어디로 라우팅할 지를 전달

Lambda Authorizer에서 API Gateway에 응답시에 해당 헤더를 전달

API G/W에서 route header를 추가하여 Istio에 전달

istio에서 header 규칙에 따라 stable 또는 preview에 전달

 

 

rollout 진행 중에 라우팅 컨트롤 방안 정의로는 랜덤 방식의 경우 추가 개발은 필요 없이 istion 설정만 변경하면 가능합니다. 특정 규칙에 따라 preview 서비스로 보낼 사용자를 정할 경우 어떤 값으로 판단할 지 정의가 필요합니다. 지역, 셋탑 버젼 등에 따라 대상을 점차적으로 확대해 보고 싶은 경우는 Canary 배포가 용이합니다.

 

Random routing 방식을 사용하지 않을 경우는 대상 기준 정의 및 개발이 필요합니다. 목적지 타입와 버젼정보를 조합하여 header 이름, 값들을 통해서 목적지 디바이스 타입이 늘어날 수 있습니다. 버전정보도 바뀔 수 있습니다.

 

메트릭 수집에 따른 Promotion 자동화는 ArgoCD의 AnalysisTemplate 구현을 통해 몇분간의 메트릭을 수집하여 정상 수치 이상일 경우에만 프로모션 가능하도록 설정도 가능합니다.

 

아래는 sevice mesh에서의 Traffic managment에 대한 일부분입니다.

Virtual services Virtual services, along with destination rules, are the key building blocks of Istio’s traffic routing functionality. A virtual service lets you configure how requests are routed to a service within an Istio service mesh, building on the basic connectivity and discovery provided by Istio and your platform. Each virtual service consists of a set of routing rules that are evaluated in order, letting Istio match each given request to the virtual service to a specific real destination within the mesh. Your mesh can require multiple virtual services or none depending on your use case. Why use virtual services? Virtual services play a key role in making Istio’s traffic management flexible and powerful. They do this by strongly decoupling where clients send their requests from the destination workloads that actually implement them. Virtual services also provide a rich way of specifying different traffic routing rules for sending traffic to those workloads. Why is this so useful? Without virtual services, Envoy distributes traffic using round-robin load balancing between all service instances, as described in the introduction. You can improve this behavior with what you know about the workloads. For example, some might represent a different version. This can be useful in A/B testing, where you might want to configure traffic routes based on percentages across different service versions, or to direct traffic from your internal users to a particular set of instances. With a virtual service, you can specify traffic behavior for one or more hostnames. You use routing rules in the virtual service that tell Envoy how to send the virtual service’s traffic to appropriate destinations. Route destinations can be versions of the same service or entirely different services. A typical use case is to send traffic to different versions of a service, specified as service subsets. Clients send requests to the virtual service host as if it was a single entity, and Envoy then routes the traffic to the different versions depending on the virtual service rules: for example, “20% of calls go to the new version” or “calls from these users go to version 2”. This allows you to, for instance, create a canary rollout where you gradually increase the percentage of traffic that’s sent to a new service version. The traffic routing is completely separate from the instance deployment, meaning that the number of instances implementing the new service version can scale up and down based on traffic load without referring to traffic routing at all. By contrast, container orchestration platforms like Kubernetes only support traffic distribution based on instance scaling, which quickly becomes complex. You can read more about how virtual services help with canary deployments in Canary Deployments using Istio. Virtual services also let you: Address multiple application services through a single virtual service. If your mesh uses Kubernetes, for example, you can configure a virtual service to handle all services in a specific namespace. Mapping a single virtual service to multiple “real” services is particularly useful in facilitating turning a monolithic application into a composite service built out of distinct microservices without requiring the consumers of the service to adapt to the transition. Your routing rules can specify “calls to these URIs of monolith.com go to microservice A”, and so on. You can see how this works in one of our examples below. Configure traffic rules in combination with gateways to control ingress and egress traffic. In some cases you also need to configure destination rules to use these features, as these are where you specify your service subsets. Specifying service subsets and other destination-specific policies in a separate object lets you reuse these cleanly between virtual services. You can find out more about destination rules in the next section. Virtual service example The following virtual service routes requests to different versions of a service depending on whether the request comes from a particular user. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata:   name: reviews spec:   hosts:   - reviews   http:   - match:     - headers:         end-user:           exact: jason     route:     - destination:         host: reviews         subset: v2   - route:     - destination:         host: reviews         subset: v3 The hosts field The hosts field lists the virtual service’s hosts - in other words, the user-addressable destination or destinations that these routing rules apply to. This is the address or addresses the client uses when sending requests to the service. hosts: - reviews   The virtual service hostname can be an IP address, a DNS name, or, depending on the platform, a short name (such as a Kubernetes service short name) that resolves, implicitly or explicitly, to a fully qualified domain name (FQDN). You can also use wildcard ("*") prefixes, letting you create a single set of routing rules for all matching services. Virtual service hosts don’t actually have to be part of the Istio service registry, they are simply virtual destinations. This lets you model traffic for virtual hosts that don’t have routable entries inside the mesh. Routing rules The http section contains the virtual service’s routing rules, describing match conditions and actions for routing HTTP/1.1, HTTP2, and gRPC traffic sent to the destination(s) specified in the hosts field (you can also use tcp and tls sections to configure routing rules for TCP and unterminated TLS traffic). A routing rule consists of the destination where you want the traffic to go and zero or more match conditions, depending on your use case. Match condition The first routing rule in the example has a condition and so begins with the match field. In this case you want this routing to apply to all requests from the user “jason”, so you use the headers, end-user, and exact fields to select the appropriate requests. - match:    - headers:        end-user:          exact: jason   Destination The route section’s destination field specifies the actual destination for traffic that matches this condition. Unlike the virtual service’s host(s), the destination’s host must be a real destination that exists in Istio’s service registry or Envoy won’t know where to send traffic to it. This can be a mesh service with proxies or a non-mesh service added using a service entry. In this case we’re running on Kubernetes and the host name is a Kubernetes service name: route: - destination:     host: reviews     subset: v2 Note in this and the other examples on this page, we use a Kubernetes short name for the destination hosts for simplicity. When this rule is evaluated, Istio adds a domain suffix based on the namespace of the virtual service that contains the routing rule to get the fully qualified name for the host. Using short names in our examples also means that you can copy and try them in any namespace you like.   Using short names like this only works if the destination hosts and the virtual service are actually in the same Kubernetes namespace. Because using the Kubernetes short name can result in misconfigurations, we recommend that you specify fully qualified host names in production environments. The destination section also specifies which subset of this Kubernetes service you want requests that match this rule’s conditions to go to, in this case the subset named v2. You’ll see how you define a service subset in the section on destination rules below. Routing rule precedence Routing rules are evaluated in sequential order from top to bottom, with the first rule in the virtual service definition being given highest priority. In this case you want anything that doesn’t match the first routing rule to go to a default destination, specified in the second rule. Because of this, the second rule has no match conditions and just directs traffic to the v3 subset.   We recommend providing a default “no condition” or weight-based rule (described below) like this as the last rule in each virtual service to ensure that traffic to the virtual service always has at least one matching route.