Cilium

Introduction

To collect metrics from Cilium, its observability layer, Hubble, must be enabled with the following Helm flags:

    --set prometheus.enabled=true \
    --set operator.prometheus.enabled=true \
    --set hubble.enabled=true \
    --set hubble.metrics.enabled="{dns,drop,tcp:destinationContext=pod|ip;sourceContext=pod|ip,flow:destinationContext=pod|ip;sourceContext=pod|ip,port-distribution,icmp,http:destinationContext=pod|ip;sourceContext=pod|ip}"

For more information see the official documentation on:

• Setting up Hubble Observability
• Hubble Metrics

Cilium in default configuration doesn't expose the source and target labels. They need to be turned on in the configuration:

dns,drop,tcp:destinationContext=pod|ip;sourceContext=pod|ip,flow:destinationContext=pod|ip;sourceContext=pod|ip,port-distribution,icmp,http:destinationContext=pod|ip;sourceContext=pod|ip

When specifying the source and/or destination context, multiple contexts can be specified by separating them via the | (pipe) symbol. When multiple are specified, the first non-empty value is added to the metric as a label.

L7 metrics

Out of the box, Hubble and its metrics provide L3/L4 visibility (IP address identities, TCP port, and flags) for all Cilium-managed traffic (e.g. via the TCP, drop, and port-distribution metrics). For L7 visibility (HTTP headers for latency, DNS response codes, etc.) you need to explicitly opt-in (e.g. this applies to the HTTP and DNS metrics).

Cilium allows enabling L7 visibility via annotations when running in the Kubernetes cluster (manual configuration). The official documentation suggests using the kubectl command to enable the required configuration for the deployed Kubernetes object.

Format of the annotations

<{Traffic Direction}/{L4 Port}/{L4 Protocol}/{L7 Protocol}>

for example:

Sample annotations

<Egress/53/UDP/DNS>,<Egress/80/TCP/HTTP>

To annotate a pod, the following command can be issued:

 kubectl annotate pod foo -n bar io.cilium.proxy-visibility="<Egress/53/UDP/DNS>,<Egress/80/TCP/HTTP>"

How to automate annotating

warning

It is NOT RECOMMENDED to use this script in production. The script should be treated as an example and we do not take any responsibility for the potential damage caused by its application.

It's very important to use the script with caution (especially with the -x flag).

To automate the process of annotation, we have prepared a sample script that can be used for quick testing of the configuration. To see its usage syntax and available options run it with -h option. Keep in mind that the script has important limitations:

• it looks for the ports by analyzing the definitions of the pods, so if a pod doesn't specify any ports and relies solely on a service, it won't get annotated,
• it ignores pods that are already annotated with the io.cilium.proxy-visibility annotation, even if the value of the annotation is incorrect or empty,
• it generates only ingress direction annotations,
• the annotations are set on pod level, so they will be removed when the pod is redeployed (for whatever reason),
• it assumes all the ports are HTTP (which correctly matches HTTP/1.1, HTTP/2 and gRPC endpoints), so it can potentially generate incorrect annotations - this can potentially lead to broken communication, e.g. in case of databases or DNS pods.

Installation of the aiops-agent with Cilium support

The following script can be used to install aiops-agent with Cilium support:

helm upgrade --install --create-namespace aiops-agent cobrick/aiops-agent --version {VERSION} -n aiops \
--set global.clientId={CLIENT_ID} \
--set global.clientSecret={CLIENT_SECRET} \
--set global.tenant={TENANT} \
--set global.site={SITE} \
--set tags.linkerd=false
--set tags.cilium=true

See the aiops-agent documentation for more details.