SlideShare a Scribd company logo
1 of 29
Download to read offline
Milking the most
out of thousands of
Kubernetes clusters
What to expect from the session
• Intro
• How is CFA using K8s?
• What does our
architecture look like?
• How are we
engineering around
K8s for our business?
• Q&A
Internet of Things: Why?
AT PEAK HOUR
1 sandwich every 16 seconds
1 box of nuggets every 25 seconds
1 order of waffle fries every 14 seconds
1 car through the drive thru every 22 seconds
267 total transactions
Chick-fil-A Architecture (2017)
MSGing
Web
Server
Local
AuthEdge
Cloud
Event
Fwding
Apps
…
Local Persistence/Storage
Connectivity
Analytics Management
Things
OAuth Server MQTT
Edge Tools
Chick-fil-A Architecture (Today)
MSGing
Local
Auth
Edge
Cloud
Event
Fwd
Apps
…
Local Persistence/Storage
Connectivity
Analytics Management
Things
OAuth Server MQTT Fleet
Why Containers? Why Kubernetes?
Idea Code
Production
Code
Value
Impact
Optimize for
Accelerate
North American Data Centers
Google
Cloud
AWSAzure
North American Data Centers
Google
Cloud
AWSAzure
Cloud-fil-A
Restaurant “Data Centers”
Intel: Quadcore processor, 8 GB RAM, SSD
Engineering Around K8s
• How we build and repair bare
metal clusters
• SRE Lessons Learned
• How we deploy applications to
thousands of clusters
Challenges of Bare Metal K8s clustering at scale
• Goal: #code2prod
• Simple enough for a non-
technologist to install
• Manageable remotely
• Automated device discovery
and self-clustering
• Self healing & HA
How we Bare Metal Cluster K8s at scale
Highlander Hooves Up
TOOLS
Sherlock FleetRKEImage
PROCESS
Bootstrapping Clusters
• Highlander
– Node coordination and
clustering leader election
using UDP
– Execute clustering (RKE)
– Swap KubeDNS for CoreDNS
– Base OAuth identity
negotiation
– Controller Pods (control
plane activity/Istio)
Initializing Clusters
What we considered
• Kops = love it, no bare metal
• Kubespray = slow + brittle
• kubeadmin = maybe in the future
• RKE = fairly simple, works for us
Future State?
• Stick w/ RKE, Kubeadmin, or roll our own to meet our needs
Resetting Cluster State
• Requirement: Need to be
able to re-image remotely
• Solution: Overlay FS + HAMS
– Manages wiping clusters
and restoring to base
Hooves Up
• Self-healing AWS SSM
Registration
• Free even for non-AWS
deployments
• Able to do remote
commands and patch
reporting/management
Lessons learned
• Use K8s feature set and don’t reinvent the wheel
• MVP. MVP. MVP.
• Ensure aggregated and searchable logging
• Deep health checks are a must --> Use /healthz
• Every service needs “/metrics”
endpoint
How do we deploy to our restaurants?
• Large number of
deployment targets
• Complex success/fail
criteria
• Array of application types
• What approaches did we
consider?
kubectl
/
Introducing Fleet
• Design Goals
– Simple to use / reason about
– Use declarative approach
– Support for variety of deployment
models (canary, blue/green)
– Rollout over flexible time period
– Sane rollback behaviors
– Leverage standard k8s API
– Full visibility
Fleet Ecosystem Components
• Fleet Client
– Git webhook, REST call, CLI
• Fleet Server API
– Code generation for
deployment, service,
ingress files
– Git management for cluster
repositories
– Deployment status tracking
• Atlas
– Repository of deploy-ready,
k8s compliant application
files
• Vessel
– Deployed on cluster, git
pull, kubectl apply, report
status
• Dashboards
Sample Templates
Fleet Walk Thru/Demo PLACEHOLDER
Application Configuration
HTTP POST Request
K8s config example
Atlas
Fleet Walk Thru/Demo PLACEHOLDER
Where you can find us
www.linkedin.com/in/brian-chambers
www.linkedin.com/in/calebrhurd
@brianchambers21
@calebrhurd
https://medium.com/@cfatechblog
https://github.com/chick-fil-a

More Related Content

What's hot

IT4IT / DevOps Tooling Landscape 2022
IT4IT / DevOps Tooling Landscape 2022 IT4IT / DevOps Tooling Landscape 2022
IT4IT / DevOps Tooling Landscape 2022 Rob Akershoek
 
Application Modernization using the Strangler Pattern
Application Modernization using the Strangler PatternApplication Modernization using the Strangler Pattern
Application Modernization using the Strangler PatternTom Laszewski
 
Identity and Access Management Introduction
Identity and Access Management IntroductionIdentity and Access Management Introduction
Identity and Access Management IntroductionAidy Tificate
 
IT Infrastructure Managed Services and RIMS
IT Infrastructure Managed Services and RIMSIT Infrastructure Managed Services and RIMS
IT Infrastructure Managed Services and RIMSRazak Mohammed Ali
 
Microservices for Application Modernisation
Microservices for Application ModernisationMicroservices for Application Modernisation
Microservices for Application ModernisationAjay Kumar Uppal
 
Reengineering The IT Operating Model to Embrace The Power Of The Cloud
Reengineering The IT Operating Model to Embrace The Power Of The CloudReengineering The IT Operating Model to Embrace The Power Of The Cloud
Reengineering The IT Operating Model to Embrace The Power Of The Cloudaccenture
 
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...Amazon Web Services
 
IdM Reference Architecture
IdM Reference ArchitectureIdM Reference Architecture
IdM Reference ArchitectureHannu Kasanen
 
Marlabs Capabilities Overview: IT Services
Marlabs Capabilities Overview: IT ServicesMarlabs Capabilities Overview: IT Services
Marlabs Capabilities Overview: IT ServicesMarlabs
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...DATAVERSITY
 
On-premise to Microsoft Azure Cloud Migration.
 On-premise to Microsoft Azure Cloud Migration. On-premise to Microsoft Azure Cloud Migration.
On-premise to Microsoft Azure Cloud Migration.Emtec Inc.
 
Designing an IT Solution
Designing an IT SolutionDesigning an IT Solution
Designing an IT SolutionPhilippe Julio
 
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...DATAVERSITY
 
Using Big Data to Drive Customer 360
Using Big Data to Drive Customer 360Using Big Data to Drive Customer 360
Using Big Data to Drive Customer 360Cloudera, Inc.
 
Identity and Access Management
Identity and Access ManagementIdentity and Access Management
Identity and Access ManagementPrashanth BS
 
Multi-Cloud Strategy for Unrestricted Possibilities
Multi-Cloud Strategy for Unrestricted PossibilitiesMulti-Cloud Strategy for Unrestricted Possibilities
Multi-Cloud Strategy for Unrestricted PossibilitiesHarsh V Sehgal
 

What's hot (20)

IT4IT / DevOps Tooling Landscape 2022
IT4IT / DevOps Tooling Landscape 2022 IT4IT / DevOps Tooling Landscape 2022
IT4IT / DevOps Tooling Landscape 2022
 
Application Modernization using the Strangler Pattern
Application Modernization using the Strangler PatternApplication Modernization using the Strangler Pattern
Application Modernization using the Strangler Pattern
 
Identity and Access Management Introduction
Identity and Access Management IntroductionIdentity and Access Management Introduction
Identity and Access Management Introduction
 
Cloud Migration
Cloud MigrationCloud Migration
Cloud Migration
 
IT Infrastructure Managed Services and RIMS
IT Infrastructure Managed Services and RIMSIT Infrastructure Managed Services and RIMS
IT Infrastructure Managed Services and RIMS
 
Cloud Migration: A How-To Guide
Cloud Migration: A How-To GuideCloud Migration: A How-To Guide
Cloud Migration: A How-To Guide
 
Microservices for Application Modernisation
Microservices for Application ModernisationMicroservices for Application Modernisation
Microservices for Application Modernisation
 
Reengineering The IT Operating Model to Embrace The Power Of The Cloud
Reengineering The IT Operating Model to Embrace The Power Of The CloudReengineering The IT Operating Model to Embrace The Power Of The Cloud
Reengineering The IT Operating Model to Embrace The Power Of The Cloud
 
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
Migrating Financial and Accounting Systems from Oracle to Amazon DynamoDB (DA...
 
IdM Reference Architecture
IdM Reference ArchitectureIdM Reference Architecture
IdM Reference Architecture
 
Cloud Computing Strategy and Architecture
Cloud Computing Strategy and ArchitectureCloud Computing Strategy and Architecture
Cloud Computing Strategy and Architecture
 
Marlabs Capabilities Overview: IT Services
Marlabs Capabilities Overview: IT ServicesMarlabs Capabilities Overview: IT Services
Marlabs Capabilities Overview: IT Services
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
 
On-premise to Microsoft Azure Cloud Migration.
 On-premise to Microsoft Azure Cloud Migration. On-premise to Microsoft Azure Cloud Migration.
On-premise to Microsoft Azure Cloud Migration.
 
Cloud Migration: Moving to the Cloud
Cloud Migration: Moving to the CloudCloud Migration: Moving to the Cloud
Cloud Migration: Moving to the Cloud
 
Designing an IT Solution
Designing an IT SolutionDesigning an IT Solution
Designing an IT Solution
 
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
 
Using Big Data to Drive Customer 360
Using Big Data to Drive Customer 360Using Big Data to Drive Customer 360
Using Big Data to Drive Customer 360
 
Identity and Access Management
Identity and Access ManagementIdentity and Access Management
Identity and Access Management
 
Multi-Cloud Strategy for Unrestricted Possibilities
Multi-Cloud Strategy for Unrestricted PossibilitiesMulti-Cloud Strategy for Unrestricted Possibilities
Multi-Cloud Strategy for Unrestricted Possibilities
 

Similar to Chick-fil-A: Milking the most out of thousands of kubernetes clusteres

Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKSMigrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKSWeaveworks
 
OneAPI Series 2 Webinar - 9th, Dec-20
OneAPI Series 2 Webinar - 9th, Dec-20OneAPI Series 2 Webinar - 9th, Dec-20
OneAPI Series 2 Webinar - 9th, Dec-20Tyrone Systems
 
One Kubernetes to rule them all (ZEUS 2019 Keynote)
One Kubernetes to rule them all (ZEUS 2019 Keynote)One Kubernetes to rule them all (ZEUS 2019 Keynote)
One Kubernetes to rule them all (ZEUS 2019 Keynote)Simon Harrer
 
Lc3 beijing-june262018-sahdev zala-guangya
Lc3 beijing-june262018-sahdev zala-guangyaLc3 beijing-june262018-sahdev zala-guangya
Lc3 beijing-june262018-sahdev zala-guangyaSahdev Zala
 
Service-Level Objective for Serverless Applications
Service-Level Objective for Serverless ApplicationsService-Level Objective for Serverless Applications
Service-Level Objective for Serverless Applicationsalekn
 
Data(?)Ops with CircleCI
Data(?)Ops with CircleCIData(?)Ops with CircleCI
Data(?)Ops with CircleCIJinwoong Kim
 
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMwareVMUG IT
 
Couchbase Connect 2016
Couchbase Connect 2016Couchbase Connect 2016
Couchbase Connect 2016Michael Kehoe
 
Kubernetes Community Growth and Use Case
Kubernetes Community Growth and Use CaseKubernetes Community Growth and Use Case
Kubernetes Community Growth and Use CaseChris Gaun
 
Customer Sharing: HTC - What is in AWS Cloud for me?
Customer Sharing: HTC - What is in AWS Cloud for me?Customer Sharing: HTC - What is in AWS Cloud for me?
Customer Sharing: HTC - What is in AWS Cloud for me?Amazon Web Services
 
Simplify Your Way To Expert Kubernetes Management
Simplify Your Way To Expert Kubernetes ManagementSimplify Your Way To Expert Kubernetes Management
Simplify Your Way To Expert Kubernetes ManagementDevOps.com
 
BayInfotech (BIT) ACI Portfolio
BayInfotech (BIT) ACI PortfolioBayInfotech (BIT) ACI Portfolio
BayInfotech (BIT) ACI PortfolioMaulik Shyani
 
Driving Digital Transformation With Containers And Kubernetes Complete Deck
Driving Digital Transformation With Containers And Kubernetes Complete DeckDriving Digital Transformation With Containers And Kubernetes Complete Deck
Driving Digital Transformation With Containers And Kubernetes Complete DeckSlideTeam
 
Database as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on KubernetesDatabase as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on KubernetesObjectRocket
 
Introduction of Kubernetes - Trang Nguyen
Introduction of Kubernetes - Trang NguyenIntroduction of Kubernetes - Trang Nguyen
Introduction of Kubernetes - Trang NguyenTrang Nguyen
 
AWS Sydney Summit 2013 - Big Data Analytics
AWS Sydney Summit 2013 - Big Data AnalyticsAWS Sydney Summit 2013 - Big Data Analytics
AWS Sydney Summit 2013 - Big Data AnalyticsAmazon Web Services
 
oci-container-engine-oke-100.pdf
oci-container-engine-oke-100.pdfoci-container-engine-oke-100.pdf
oci-container-engine-oke-100.pdfNandiniSinghal16
 
DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...
DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...
DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...Docker, Inc.
 

Similar to Chick-fil-A: Milking the most out of thousands of kubernetes clusteres (20)

Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKSMigrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
 
OneAPI Series 2 Webinar - 9th, Dec-20
OneAPI Series 2 Webinar - 9th, Dec-20OneAPI Series 2 Webinar - 9th, Dec-20
OneAPI Series 2 Webinar - 9th, Dec-20
 
One Kubernetes to rule them all (ZEUS 2019 Keynote)
One Kubernetes to rule them all (ZEUS 2019 Keynote)One Kubernetes to rule them all (ZEUS 2019 Keynote)
One Kubernetes to rule them all (ZEUS 2019 Keynote)
 
Lc3 beijing-june262018-sahdev zala-guangya
Lc3 beijing-june262018-sahdev zala-guangyaLc3 beijing-june262018-sahdev zala-guangya
Lc3 beijing-june262018-sahdev zala-guangya
 
Service-Level Objective for Serverless Applications
Service-Level Objective for Serverless ApplicationsService-Level Objective for Serverless Applications
Service-Level Objective for Serverless Applications
 
Data(?)Ops with CircleCI
Data(?)Ops with CircleCIData(?)Ops with CircleCI
Data(?)Ops with CircleCI
 
Managing Your Cloud Assets
Managing Your Cloud AssetsManaging Your Cloud Assets
Managing Your Cloud Assets
 
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware
 
Couchbase Connect 2016
Couchbase Connect 2016Couchbase Connect 2016
Couchbase Connect 2016
 
Kubernetes Community Growth and Use Case
Kubernetes Community Growth and Use CaseKubernetes Community Growth and Use Case
Kubernetes Community Growth and Use Case
 
Customer Sharing: HTC - What is in AWS Cloud for me?
Customer Sharing: HTC - What is in AWS Cloud for me?Customer Sharing: HTC - What is in AWS Cloud for me?
Customer Sharing: HTC - What is in AWS Cloud for me?
 
Simplify Your Way To Expert Kubernetes Management
Simplify Your Way To Expert Kubernetes ManagementSimplify Your Way To Expert Kubernetes Management
Simplify Your Way To Expert Kubernetes Management
 
BayInfotech (BIT) ACI Portfolio
BayInfotech (BIT) ACI PortfolioBayInfotech (BIT) ACI Portfolio
BayInfotech (BIT) ACI Portfolio
 
Driving Digital Transformation With Containers And Kubernetes Complete Deck
Driving Digital Transformation With Containers And Kubernetes Complete DeckDriving Digital Transformation With Containers And Kubernetes Complete Deck
Driving Digital Transformation With Containers And Kubernetes Complete Deck
 
Database as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on KubernetesDatabase as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on Kubernetes
 
Introduction to Amazon EC2
Introduction to Amazon EC2Introduction to Amazon EC2
Introduction to Amazon EC2
 
Introduction of Kubernetes - Trang Nguyen
Introduction of Kubernetes - Trang NguyenIntroduction of Kubernetes - Trang Nguyen
Introduction of Kubernetes - Trang Nguyen
 
AWS Sydney Summit 2013 - Big Data Analytics
AWS Sydney Summit 2013 - Big Data AnalyticsAWS Sydney Summit 2013 - Big Data Analytics
AWS Sydney Summit 2013 - Big Data Analytics
 
oci-container-engine-oke-100.pdf
oci-container-engine-oke-100.pdfoci-container-engine-oke-100.pdf
oci-container-engine-oke-100.pdf
 
DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...
DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...
DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...
 

Recently uploaded

Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 

Recently uploaded (20)

Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 

Chick-fil-A: Milking the most out of thousands of kubernetes clusteres

  • 1. Milking the most out of thousands of Kubernetes clusters
  • 2. What to expect from the session • Intro • How is CFA using K8s? • What does our architecture look like? • How are we engineering around K8s for our business? • Q&A
  • 4. AT PEAK HOUR 1 sandwich every 16 seconds 1 box of nuggets every 25 seconds 1 order of waffle fries every 14 seconds 1 car through the drive thru every 22 seconds 267 total transactions
  • 5. Chick-fil-A Architecture (2017) MSGing Web Server Local AuthEdge Cloud Event Fwding Apps … Local Persistence/Storage Connectivity Analytics Management Things OAuth Server MQTT Edge Tools
  • 6. Chick-fil-A Architecture (Today) MSGing Local Auth Edge Cloud Event Fwd Apps … Local Persistence/Storage Connectivity Analytics Management Things OAuth Server MQTT Fleet
  • 7. Why Containers? Why Kubernetes? Idea Code Production Code Value Impact Optimize for
  • 9. North American Data Centers Google Cloud AWSAzure
  • 10. North American Data Centers Google Cloud AWSAzure Cloud-fil-A
  • 11. Restaurant “Data Centers” Intel: Quadcore processor, 8 GB RAM, SSD
  • 12. Engineering Around K8s • How we build and repair bare metal clusters • SRE Lessons Learned • How we deploy applications to thousands of clusters
  • 13. Challenges of Bare Metal K8s clustering at scale • Goal: #code2prod • Simple enough for a non- technologist to install • Manageable remotely • Automated device discovery and self-clustering • Self healing & HA
  • 14. How we Bare Metal Cluster K8s at scale Highlander Hooves Up TOOLS Sherlock FleetRKEImage PROCESS
  • 15. Bootstrapping Clusters • Highlander – Node coordination and clustering leader election using UDP – Execute clustering (RKE) – Swap KubeDNS for CoreDNS – Base OAuth identity negotiation – Controller Pods (control plane activity/Istio)
  • 16. Initializing Clusters What we considered • Kops = love it, no bare metal • Kubespray = slow + brittle • kubeadmin = maybe in the future • RKE = fairly simple, works for us Future State? • Stick w/ RKE, Kubeadmin, or roll our own to meet our needs
  • 17. Resetting Cluster State • Requirement: Need to be able to re-image remotely • Solution: Overlay FS + HAMS – Manages wiping clusters and restoring to base
  • 18. Hooves Up • Self-healing AWS SSM Registration • Free even for non-AWS deployments • Able to do remote commands and patch reporting/management
  • 19. Lessons learned • Use K8s feature set and don’t reinvent the wheel • MVP. MVP. MVP. • Ensure aggregated and searchable logging • Deep health checks are a must --> Use /healthz • Every service needs “/metrics” endpoint
  • 20. How do we deploy to our restaurants? • Large number of deployment targets • Complex success/fail criteria • Array of application types • What approaches did we consider? kubectl /
  • 21. Introducing Fleet • Design Goals – Simple to use / reason about – Use declarative approach – Support for variety of deployment models (canary, blue/green) – Rollout over flexible time period – Sane rollback behaviors – Leverage standard k8s API – Full visibility
  • 22. Fleet Ecosystem Components • Fleet Client – Git webhook, REST call, CLI • Fleet Server API – Code generation for deployment, service, ingress files – Git management for cluster repositories – Deployment status tracking • Atlas – Repository of deploy-ready, k8s compliant application files • Vessel – Deployed on cluster, git pull, kubectl apply, report status • Dashboards
  • 24. Fleet Walk Thru/Demo PLACEHOLDER
  • 27. Atlas
  • 28. Fleet Walk Thru/Demo PLACEHOLDER
  • 29. Where you can find us www.linkedin.com/in/brian-chambers www.linkedin.com/in/calebrhurd @brianchambers21 @calebrhurd https://medium.com/@cfatechblog https://github.com/chick-fil-a