prometheus"Prometheus in Cloud-Native Engineering - The Prometheus monitoring system" and time series database.
Independent GitHub snapshot · Automated classification · Original documentation governs
name: prometheus compatibility: opencode completeness: 95 content-types:
related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd
Category: alerting
Status: Active
Stars: 63,688
Last Updated: 2026-04-22
Primary Language: Go
Documentation: https://prometheus.io/docs/
Prometheus is a core component of the cloud-native ecosystem, serving as a monitoring and alerting toolkit designed for reliability and scalability in dynamic cloud-native environments.
Complex monitoring requirements in dynamic cloud environments with frequent scale changes, microservices, and ephemeral containers. It provides reliable metrics collection, storage, and alerting.
Use Prometheus when you need to monitor microservices, require metrics-based alerting, want to integrate with Kubernetes via kube-state-metrics and cAdvisor, or need a scalable, pull-based monitoring system. Consider alternatives if you need native distributed tracing or complex log correlation.
Deployment Failures
Performance Issues
Configuration Errors
Integration Problems
This tutorial will guide you through installing, configuring, and using Prometheus for monitoring Kubernetes applications.
Before beginning, ensure you have:
kubectl configured with cluster-admin permissionshelm CLI (for Helm installation method)Verify your setup:
# Check cluster connectivity
kubectl cluster-info
# Verify kubectl configuration
kubectl get nodes
# Verify access to kube-system namespace
kubectl get pods -n kube-system
Prometheus can be installed using various methods including kubectl, Helm, or using the Prometheus Operator.
# Create monitoring namespace
kubectl create namespace monitoring
# Download and apply Prometheus Operator manifests
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml
# Verify installation
kubectl get pods -n monitoring
# Or use kube-prometheus-stack for a complete installation
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/kube-prometheus-crd.yaml
# Wait for CRDs to be created
kubectl wait --for=condition=Established --all=CustomResourceDefinition --timeout=30s
# Apply kube-prometheus-stack
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/
# Add the Prometheus Community Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add prometheus-stack https://prometheus-community.github.io/helm-charts
helm repo update
# Install kube-prometheus-stack (includes Prometheus, Grafana, Alertmanager)
helm install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
--wait
# Verify installation
kubectl get pods -n monitoring
helm list -n monitoring
# Create custom values file
cat > prometheus-values.yaml << EOF
prometheus:
prometheusSpec:
retention: 30d
replicas: 2
resources:
requests:
memory: 400Mi
cpu: 100m
limits:
memory: 2Gi
cpu: 1000m
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: standard
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
alertmanager:
alertmanagerSpec:
replicas: 3
resources:
requests:
memory: 100Mi
cpu: 50m
grafana:
enabled: true
adminPassword: admin
EOF
# Install with custom configuration
helm install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
-f prometheus-values.yaml \
--wait
# Create namespace
kubectl create namespace monitoring
# Deploy Prometheus without Operator
kubectl apply -n monitoring -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: monitoring
---
related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
verbs: ["get"]
---
related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: monitoring
---
related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitoring
data:
prometheus.yml: |
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'kubernetes-nodes'
kubernetes_sd_configs:
- role: node
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
---
related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd
apiVersion: v1
kind: Service
metadata:
name: prometheus
namespace: monitoring
labels:
app: prometheus
spec:
selector:
app: prometheus
ports:
- port: 9090
targetPort: 9090
---
related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
serviceAccountName: prometheus
containers:
- name: prometheus
image: prom/prometheus:v2.45.0
args:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
ports:
- containerPort: 9090
volumeMounts:
- name: config
mountPath: /etc/prometheus
- name: storage
mountPath: /prometheus
volumes:
- name: config
configMap:
name: prometheus-config
- name: storage
emptyDir: {}
EOF
related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd
# Port-forward to access Prometheus UI
kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090
# Or access via LoadBalancer
kubectl -n monitoring patch svc prometheus-kube-prometheus-prometheus -p '{"spec":{"type":"LoadBalancer"}}'
# Or access via Ingress
kubectl -n monitoring create ingress prometheus \
--class=nginx \
--rule="prometheus.example.com/*=prometheus-kube-prometheus-prometheus:9090"
# Add custom scrape configuration via ConfigMap
cat > prometheus-scrape-config.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-additional-configs
namespace: monitoring
data:
additional-scrape-configs.yaml: |
- job_name: 'myapp'
static_configs:
- targets: ['myapp-service:8080']
metrics_path: /metrics
EOF
kubectl apply -f prometheus-scrape-config.yaml
# Create alerting rules
cat > prometheus-alert-rules.yaml << EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: application-alerts
namespace: monitoring
labels:
prometheus: kube-prometheus
role: alert-rules
spec:
groups:
- name: application.rules
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate detected"
description: "Error rate is {{ $value | humanizePercentage }}"
- alert: ServiceDown
expr: up == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Service {{ $labels.instance }} is down"
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."
EOF
kubectl apply -f prometheus-alert-rules.yaml
# ServiceMonitor to automatically discover and scrape services
cat > service-monitor.yaml << EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: myapp-monitor
namespace: production
labels:
release: prometheus
spec:
selector:
matchLabels:
app: myapp
namespaceSelector:
matchNames:
- production
endpoints:
- port: metrics
interval: 15s
path: /metrics
scheme: http
EOF
kubectl apply -f service-monitor.yaml
related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd
# View all metrics
curl http://localhost:9090/api/v1/label/__name__/values
# Query CPU usage
curl -s 'http://localhost:9090/api/v1/query?query=node_cpu_seconds_total' | jq .
# Query memory usage
curl -s 'http://localhost:9090/api/v1/query?query=node_memory_MemAvailable_bytes' | jq .
# Query pod CPU usage
curl -s 'http://localhost:9090/api/v1/query?query=sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))' | jq .
# CPU usage percentage
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Memory usage percentage
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100
# HTTP error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
# Request latency percentiles
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
# Up time percentage
avg(rate(up[5m])) by (instance) * 100
# Access Grafana
kubectl -n monitoring port-forward svc/prometheus-grafana 3000:80
# Or get admin password
kubectl -n monitoring get secret prometheus-grafana \
-o jsonpath="{.data.admin-password}" | base64 -d; echo
# Import dashboard via API
curl -X POST http://localhost:3000/api/dashboards/db \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d @dashboard.json
# Check Prometheus status
kubectl get pods -n monitoring -l app=kube-prometheus-prometheus
# View Prometheus logs
kubectl logs -n monitoring prometheus-kube-prometheus-prometheus-0 -c prometheus
# Check Prometheus configuration
kubectl -n monitoring get configmap prometheus-kube-prometheus-prometheus -o yaml
# View Prometheus targets
curl http://localhost:9090/api/v1/targets | jq .
# View active alerts
curl http://localhost:9090/api/v1/alerts | jq .
# View rule groups
curl http://localhost:9090/api/v1/rules | jq .
# Scale storage (for StatefulSet deployments)
kubectl -n monitoring scale statefulset prometheus-kube-prometheus-prometheus --replicas=3
# Check storage usage
kubectl -n monitoring exec prometheus-kube-prometheus-prometheus-0 -- df -h /prometheus
# Extend retention period
kubectl -n monitoring patch prometheus prometheus-kube-prometheus-prometheus \
--type='merge' -p '{"spec":{"retention":"90d"}}'
# Create backup of Prometheus data
kubectl -n monitoring exec prometheus-kube-prometheus-prometheus-0 \
-- mkdir -p /backup
kubectl -n monitoring exec prometheus-kube-prometheus-prometheus-0 \
-- cp -r /prometheus/* /backup/
# Export backup
kubectl -n monitoring cp monitoring/prometheus-kube-prometheus-prometheus-0:/backup \
./prometheus-backup --container=prometheus
# Using Helm
helm upgrade prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
-f prometheus-values.yaml
# Using kubectl (Operator)
kubectl -n monitoring apply -f manifests/
# Rollback if needed
helm rollback prometheus -n monitoring
Content generated automatically. Verify against official documentation before production use.
# prometheus.yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: production
environment: prod
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
# Kubernetes node metrics
- job_name: 'kubernetes-nodes'
kubernetes_sd_configs:
- role: node
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
# Kubernetes pods
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
# Kubernetes services
- job_name: 'kubernetes-services'
kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
# rules/alerting.rules.yml
groups:
- name: application-alerts
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate detected"
description: "Error rate is {{ $value | humanizePercentage }} for the last 5 minutes."
- alert: ServiceDown
expr: up{job="kubernetes-pods"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Service is down"
description: "Service {{ $labels.pod }} in namespace {{ $labels.namespace }} has been down for more than 2 minutes."
- alert: HighLatency
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m]))
by (le, service)
) > 1
for: 10m
labels:
severity: warning
annotations:
summary: "High latency detected"
description: "99th percentile latency is above 1 second for service {{ $labels.service }}."
- name: infrastructure-alerts
rules:
- alert: HighMemoryUsage
expr: |
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
/ node_memory_MemTotal_bytes > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "High memory usage on node"
description: "Node {{ $labels.instance }} is using {{ $value | humanizePercentage }} of memory."
- alert: HighCPUUsage
expr: |
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage on node"
description: "Node {{ $labels.instance }} is using {{ $value | humanizePercentage }} of CPU."
# ServiceMonitor for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: myapp-monitor
namespace: monitoring
labels:
release: prometheus
spec:
selector:
matchLabels:
app: myapp
namespaceSelector:
matchNames:
- production
endpoints:
- port: http
interval: 15s
path: /metrics
scheme: http
scrapeTimeout: 10s
honorLabels: true
metricRelabelings:
- sourceLabels: [__name__]
regex: 'go_.*'
action: drop
Use this skill when:
Assess Requirements — Understand the use case, scale, integration needs, and existing infrastructure. Checkpoint: Document requirements, constraints, and success criteria.
Design Architecture — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. Checkpoint: Verify the architecture addresses all requirements and follows CNCF conventions.
Implement & Configure — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. Checkpoint: Validate all YAML against schema and test in a staging environment.
Deploy & Monitor — Apply manifests to the cluster, verify component health, and confirm observability is working. Checkpoint: Confirm all pods/services are running, probes passing, and metrics/alerts configured.
These links are resolved at load time — content is fetched and inlined to provide the latest documentation.