Prerequisites

• Find the number of cores on the instance type

  To find the number of cores, run the following command:

  aws ec2 describe-instance-types \
    --instance-type m5.24xlarge \
    --query "InstanceTypes[0].VCpuInfo.DefaultCores" \
    --output text

  This value must be substituted for <cpu-cores> in the EC2 run-instances command.

• Create a user data file

  Create the user data file by copying the following contents into a file named worker-user-data.bash:

  #!/bin/bash
  /etc/eks/bootstrap.sh xrd-cluster

  For XRd Control Plane, add the following two sysctl settings to the user data file:

  echo "fs.inotify.max_user_instances=64000" >> /etc/sysctl.conf
  echo "fs.inotify.max_user_watches=64000" >> /etc/sysctl.conf

Bring Up the Worker Node

Bring up the worker node by running the following command:

aws ec2 run-instances \
  --image-id <xrd-ami-id> \
  --count 1 \
  --instance-type m5.24xlarge \
  --key-name <key-pair-name> \
  --block-device-mappings "DeviceName=/dev/xvda,Ebs={VolumeSize=56}" \
  --iam-instance-profile "Arn=<node-profile-arn>" \
  --network-interfaces "DeleteOnTermination=true,DeviceIndex=0,Groups=<sg-id>, SubnetId=<private-subnet-1>,PrivateIpAddress=10.0.0.10" \
  --cpu-options CoreCount=<cpu-cores>,ThreadsPerCore=1 \
  --tag-specifications "ResourceType=instance,Tags=[{Key=kubernetes.io/cluster/xrd-cluster,Value=owned}]" \
  --user-data file://worker-user-data.bash

Make a note of the instance id, <worker-instance-id>.

This command brings up an EC2 instance with the following settings:

• A 56-GB primary partition - required to store any process cores that the XRd generates.

• A single interface in the first private subnet with permissions to communicate with the EKS control plane. This interface is used for cluster control plane communications. The assigned IP address is 10.0.0.10.

• One thread per core (SMT or Hyper-Threading turned off). This is to prevent the "noisy neighbor effect" (where processes scheduled on a different logical but same physical core hampers the performance of high priority processes) for the high-performance packet processing threads.

• A tag that is required by EKS to display the node should be allowed to join the cluster.

• A user data file that runs the EKS bootstrap script with the cluster name.

The requirements for XRd Control Plane are as follows:

• You can use a smaller (and cheaper) instance type, for example, m5.2xlarge.

• The --cpu-options line is not required.

Turn off the source or destination check for the instance, by running the following command:

aws ec2 modify-instance-attribute \
  --instance-id <worker-instance-id> \
  --no-source-dest-check

When the worker node is up, check if the worker node is added to the cluster.

# kubectl get nodes
NAME                        STATUS   ROLES    AGE    VERSION
ip-10-0-0-10.ec2.internal   Ready    <none>   1m     v1.22.17-eks-48e63af

Note	If you do not see the worker node, check the EKS configuration steps.

Add Subnets for Data Traffic

The setup for data subnets and security groups vary with each deployment. In this example two subnets are created, and isolated from the internet. One security group is created, which allows all communication inside the security group, but rejects any traffic from outside the security group.

These subnets and the security group can be used for interfaces on multiple worker nodes.

Note	The data traffic subnets must be created in the same Availability Zone (AZ) as the worker node.

Create two subnets for data traffic using the following command:

aws ec2 create-subnet \
  --vpc-id "<vpc-id>" \
  --cidr-block "10.0.100.0/24" \
  --availability-zone "<region>a"
aws ec2 create-subnet \
  --vpc-id <vpc-id> \
  --cidr-block "10.0.101.0/24" \
  --availability-zone "<region>a"

Note	The a after the <region> in the availability zone specification indicates the first subnet in the region.

Make a note of the subnet IDs, <data-subnet-1> and <data-subnet-2>.

Create the Security Group

Create a security group for the data network traffic using the following command:

aws ec2 create-security-group --group-name "xrd-data" \
  --description "Data traffic for XRd EKS Cluster" \
  --vpc-id <vpc-id>

Make a note of the output group ID, <data-sg-id>.

A single egress rule that allows all traffic egress is added when a security group is created, but no ingress rules are added. So, all incoming traffic is dropped. You must add an ingress rule for all traffic originating from the same security group, so that all traffic is allowed within the security group.

aws ec2 authorize-security-group-ingress \
  --group-id <data-sg-id> \
  --source-group <data-sg-id> \
  --protocol all

Create and Attach Interfaces

The following example creates two interfaces on the worker node to be used by XRd, one in each of the two subnets created. Run the following commands to create these interfaces:

aws ec2 create-network-interface \
  --description "XRd worker 1 data 1" \
  --groups <data-sg-id> \
  --private-ip-address "10.0.100.10" \
  --subnet-id <data-subnet-1> \
  --tag-specifications "ResourceType=network-interface,Tags=[{Key=node.k8s.amazonaws.com/no_manage,Value=true}]"
aws ec2 create-network-interface \
  --description "XRd worker 1 data 2" \
  --groups <data-sg-id> \
  --private-ip-address "10.0.101.10" \
  --subnet-id <data-subnet-2> \
  --tag-specifications "ResourceType=network-interface,Tags=[{Key=node.k8s.amazonaws.com/no_manage,Value=true}]"

Make a note of both the network interface IDs, <data-interface-1> and <data-interface-2>.

To attach the interfaces to the worker node, run the following commands:

aws ec2 attach-network-interface \
  --device-index 1 \
  --instance-id <worker-instance-id> \
  --network-interface-id <data-interface-1>
aws ec2 attach-network-interface \
  --device-index 2 \
  --instance-id <worker-instance-id> \
  --network-interface-id <data-interface-2>