Observability & CI/CD

Metrics · Logs · Traces · Alerting · SLOs · Pipelines · SAST/DAST · GitOps

Three Pillars of Observability

Metrics · Structured Logs · Distributed Traces

MetricsRED method per service: Rate (req/s), Errors (error %), Duration (latency histogram). USE method per node: Utilisation, Saturation, Errors. Metrics Collection Platform scrape interval: configurable. Retention: configurable hot/cold retention tiers.

Structured LogsJSON-structured logs only — no free-text log lines. Mandatory fields: timestamp, traceId, spanId, serviceId, level, message. PII must never appear in logs — masked at emission point.

Distributed TracingW3C TraceContext propagation across all services. Every request carries a traceId. configurable sampling rate in production; 100% for errors and slow requests and slow requests (>P99). Span retention: configurable.

SLOsEach tier-1 service defines: Availability SLO (99.9%), Latency SLO (P99 < 500ms), Error Rate SLO (<0.1%). Error budget burn rate alerts at 2× and 5× burn rate. SLO dashboards public to engineering.

AlertingAlert on symptoms (SLO breach, error budget burn), not causes (CPU%). Multi-window multi-burn-rate alerting (1h/6h windows). Runbook URL mandatory in every alert. On-call rotation with defined escalation path.

RED Method SLO/SLA W3C TraceContext Error Budget No PII in Logs

CI/CD Pipeline — Security & Quality Gates

SAST · DAST · SCA · Container Scan · GitOps

Pipeline Stages — Every Service

Build → Unit Tests → SAST Scan → SCA → Image Scan

Contract Tests → Integration Tests → DAST Scan → Deploy (Canary)

SASTStatic analysis on every commit. Blocks merge on critical/high severity findings. OWASP Top 10, injection flaws, insecure deserialization, hardcoded secrets.

SCASoftware Composition Analysis: all third-party dependencies scanned for CVEs. CVSS ≥7.0 blocks pipeline. Licence compliance checked — GPL dependencies flagged.

Container ScanningBase image and application layer scanned for CVEs before push to registry. Only images from approved base registries accepted. Distroless images preferred.

DASTDynamic Application Security Testing runs against deployed service in integration environment. DAST Scanner / DAST Scanner. Automated attack scenarios including SQLi, XSS, IDOR.

GitOpsKubernetes manifests in Git are the single source of truth for cluster state. GitOps Controller/GitOps Controller continuously reconciles. No manual kubectl apply in production. All changes via PR with review.

SAST DAST SCA GitOps Distroless Canary Deploy

Non-Functional Requirements — Summary Reference

NFR Category	Requirement	Target / Standard	Status
Availability	Platform availability SLO	99.99% (≈ minutes per year)	Mandatory
Latency	API P99 latency (tier-1 services)	< Sub-second P99	Mandatory
Throughput	CDR processing rate	Very High — stream processing sustained	Mandatory
RTO	Revenue-critical service recovery	<Minutes (Tier 0 — near-zero)	Mandatory
RPO	Maximum data loss window	Near-zero (Tier 0 — synchronous replication)	Mandatory
Security	Encryption in transit	TLS 1.3+ on external traffic, mTLS on all inter-service communication	Mandatory
Security	Encryption at rest	AES-256-GCM all data stores	Mandatory
Identity	Authentication protocol	OAuth 2.0 / OIDC + Workload Identity Standard workload identity	Mandatory
PII Compliance	GDPR — Right to Erasure SLA	Confirmed deletion within the regulatory SLA window	Mandatory
Resilience	Circuit breaker on all external calls	Error rate >50% in 10 s window triggers open	Mandatory
Idempotency	All financial mutation endpoints	Idempotency key + 24 h dedup store	Mandatory
Scalability	HPA on all stateless services	CPU target 60%, scale within a short cool-down window	Mandatory
Anti-Affinity	Pod distribution	Required: no two replicas on same node; Preferred: spread across AZs	Mandatory
Technical Debt	Sprint debt remediation allocation	Minimum 20% per sprint	Recommended
Observability	Distributed tracing coverage	100% of requests carry traceId; 10% sampled	Mandatory
CI/CD Security	SAST + SCA in every pipeline	Critical/High CVEs block merge	Mandatory

Observability & CI/CD

Telecom

Apps

Company