What is K8s?
- Software-defined Datacenter using Container orchestration
- Bin packing: what machine to use?
- Monitors state of cluster
- Auto-scaling via service discovery load balancing
- Manages secrets/configurations
- Manages storage versioning
- Automated roll-out/roll-backs
- A/B testing of new technologies
- Everything is ephemeral
Terminology
- Pods: Containers and Volumes
- Smallest unit of compute
- Examples
- Container
- Container + Storage
- Multiple containers + Storage: if they have to share storage
- Multiple containers + Multiple Storage: if they both have to communicate
- Deployment: Creates/Maintains pods
- Uses specific images
- Maintains scalability based on usage
- Nodes: Hosts within the cluster
- They can run: Kubelet: Container runtime (docker, etc), Kube-proxy
- Worker nodes: run everything (handles failures)
- Contol plane node: always at least 1 but can be multiple
- Kubelet has an API, ties node into the cluster
- Kube-proxy manages IP tables rules for virtual IP’s
- Services: Load balancers
- Creates DNS name, virtual IP, incoming/outgoing port pair
- Redirect traffic to pods whose labels match service of interest
- Predictable name for DNS
app.default.svc.cluster.local
- Services create SVC records for named ports
_80-80._tcp.app.default.svc.cluster.local
- Service has labels which identify version of API, matches by label
- Deployments: Method for creating pods/handling scaling
- Namespaces: Used as a logical grouping for k8s objects
- Usually by project
- default: resources deployed when namespace isnt specified
- kube-system: default k8s control plane (kube-)
How does K8s work
- Control loops: declarative system telling it what needs to be run
- Watches for nodes to fail, then brings them back up
- Does the state of the cluster match what is declared?
Attacking K8s clusters
- Control plane Node-Only components
- K8s API Server
- Accepts declarative object configs, generated from kubectl and API requests
- First point of contact for cluster
- etcd Server
- Retains the state of every object in cluster
- Allow “is the answer different from last time” queries
- Stores in database
- Run “I am a node, what pods do I run?” queries
- Responds with version answer
- Controller manager
- Runs control loops to bring clusters state to parity w/ etcd
- Multiple controllers, all compiled into a binary w/ exception of Cloud controller, Scheduler
- Attacking
- From INSIDE inside cluster, inside container, inside pod
- Once inside, lots of attack surface, usually starts in pod
- Actor compromised the app within pod
- Actor phished/compromised a person w/ access to pod
- Actor was auth’d and wants to escalate
- Good to be able to handle compromised pod first then be able to handle compromised node
- Staged pod w/ priv container or container breakouts
- Abuse access in pod to access other services
- Attack other containers in pod
- Make requests to API or Kubelet
- Run commands in different pod
- Start a new pod w/ priv and node filesystem/resources access (by default can add crazy permissions)
- Gather secrets that k8s provides to pods
- Connect to k8s dashboard to perform actions
- Interact w/ etcd server to change cluster state
- Interact w/ cloud service provider via cluster account
- kopsbucket : stores admin creds, read bucket
- From INSIDE node but outside of pod
- From OUTSIDE, compromising cluster
- Left open dashboard exposed
RBAC and Auth Modules
- Place restrictions on API server
- Defines what PRINCIPALS can perform ACTIONS
- Principals are users or services accounts
- jay in group system:authenticated
- Actions are VERBS combined with OBJECTS
- Role and Role Binding
- Roles specify a list of actions
- roleRef: binds role to name and subject
- Delete service account defaults
YAML Review
- Each YAML represent kind, apiversion, metadata, rules
- roleRef: binds role to name and subject
Defense
- Year old clusters are out of date, support 9 months of releases
- Network Policy: Containers should be able to reach out to external services (LDAP, etc)
- CIS Benchmark:
- Image Safety:
History of Microservices