There are lots of good reasons to be running a Selenium grid in cloud or Kubernetes containers: It’s less maintenance, it’s faster with more reliable network hopping, it’s self healing and it can auto scale. So what do we need to get this off the ground?
So let’s get started, first off we need to login to the portal and create a Kubernetes service/ cluster resource.
Pick subscription & allocate a resource group, my resource group here is called ‘Selenium‘
Cluster name: I’m calling mine here ‘k8s-grid‘
Region: (Europe) West Europe (this is one of the regions that allows for virtual nodes which are required for scaling), unlike (Europe) UK South, for example.
Node Size: B4ms (4 cores to allow for some scaling)
Click next (Node pools) and enable virtual nodes.
Click next (Authentication)
You can choose either a Service principal if you have one, or a slightly easier wrapper around it in System-assigned managed identity. Keep RBAC Enabled with default encryption type.
Click next (Networking)
Just leave this as it comes.
Click next (Integrations)
I turned off Container monitoring for my MSDN account.
Click next (Review and create)
Validation should pass, click create and wait for it to deploy.
Open powershell or whatever terminal you prefer, login to Azure:
Now connect to the cluster specifying the Resource Group and Cluster name we gave in a previous step:
az aks get-credentials --resource-group Selenium --name k8s-grid
You should see something like
|
1 |
Merged "k8s-grid" as current context in C:\Users\jholsgrove\.kube\config |
That’s it, you are connected!
First create a selenium-hub-deployment.yaml locally with the deployment details (Kubernetes has some good examples here, but I will list them out anyway making the odd tweak)
|
1 |
kubectl create -f selenium-hub-deployment.yaml |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
apiVersion: apps/v1 kind: Deployment metadata: name: selenium-hub labels: app: selenium-hub spec: replicas: 1 selector: matchLabels: app: selenium-hub template: metadata: labels: app: selenium-hub spec: containers: - name: selenium-hub image: selenium/hub:3.141 ports: - containerPort: 4444 resources: limits: memory: "1000Mi" cpu: ".5" livenessProbe: httpGet: path: /wd/hub/status port: 4444 initialDelaySeconds: 30 timeoutSeconds: 5 readinessProbe: httpGet: path: /wd/hub/status port: 4444 initialDelaySeconds: 30 timeoutSeconds: 5 |
We can now see the pod initialising/ running:
|
1 2 3 4 5 6 |
PS C:\temp\k8s> kubectl create -f selenium-hub-deployment.yaml deployment.apps/selenium-hub created PS C:\temp\k8s> kubectl get pods NAME READY STATUS RESTARTS AGE selenium-hub-7ff45ff687-swvjg 0/1 ContainerCreating 0 11s PS C:\temp\k8s> |
Next we are going to create a service to allow us to get to the Hub web interface, note we need to make sure the type is set to ‘LoadBalancer‘ and create selenium-hub-svc.yaml:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
apiVersion: v1 kind: Service metadata: name: selenium-hub labels: app: selenium-hub spec: ports: - port: 4444 targetPort: 4444 name: port0 selector: app: selenium-hub type: LoadBalancer sessionAffinity: None |
To deploy: kubectl create -f selenium-hub-svc.yaml
Now run kubectl get services and note the external IP address that was assigned (this might take a few moments to be assigned)
|
1 2 3 4 |
PS C:\temp\k8s> kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 27m selenium-hub LoadBalancer 10.0.81.187 20.50.14.88 4444:32455/TCP 19s |
We can now navigate to the Hub Console at: http://20.50.14.88:4444/grid/console
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
apiVersion: apps/v1 kind: Deployment metadata: name: selenium-node-chrome labels: app: selenium-node-chrome spec: replicas: 1 selector: matchLabels: app: selenium-node-chrome template: metadata: labels: app: selenium-node-chrome spec: volumes: - name: dshm emptyDir: medium: Memory containers: - name: selenium-node-chrome image: selenium/node-chrome-debug:3.141 ports: - containerPort: 5555 volumeMounts: - mountPath: /dev/shm name: dshm env: - name: HUB_HOST value: "selenium-hub" - name: HUB_PORT value: "4444" resources: limits: memory: "1000Mi" |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
apiVersion: apps/v1 kind: Deployment metadata: name: selenium-node-firefox labels: app: selenium-node-firefox spec: replicas: 1 selector: matchLabels: app: selenium-node-firefox template: metadata: labels: app: selenium-node-firefox spec: volumes: - name: dshm emptyDir: medium: Memory containers: - name: selenium-node-firefox image: selenium/node-firefox-debug:3.141 ports: - containerPort: 5900 volumeMounts: - mountPath: /dev/shm name: dshm env: - name: HUB_HOST value: "selenium-hub" - name: HUB_PORT value: "4444" resources: limits: memory: "1000Mi" cpu: ".5" |
And lets see the nodes being added:
|
1 2 3 4 5 |
PS C:\temp\k8s> kubectl get pods NAME READY STATUS RESTARTS AGE selenium-hub-7ff45ff687-swvjg 1/1 Running 0 5m34s selenium-node-chrome-778787c6f-x2bdd 0/1 ContainerCreating 0 19s selenium-node-firefox-5c9444c5db-2mlwh 0/1 ContainerCreating 0 9s |
Refresh the grid console and see the nodes: http://20.50.14.88:4444/grid/console
We can scale by replicating nodes and have them sit idle:
|
1 2 |
PS C:\temp\k8s> kubectl scale deployment selenium-node-firefox --replicas=10 deployment.extensions/selenium-node-firefox scaled |
This adds 10 firefox replicas.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
PS C:\temp\k8s> kubectl get pods NAME READY STATUS RESTARTS AGE selenium-hub-7ff45ff687-swvjg 1/1 Running 0 15m selenium-node-chrome-778787c6f-x2bdd 1/1 Running 0 9m58s selenium-node-firefox-5c9444c5db-2mlwh 1/1 Running 0 9m48s selenium-node-firefox-5c9444c5db-66thg 0/1 Pending 0 47s selenium-node-firefox-5c9444c5db-b54l9 0/1 Pending 0 47s selenium-node-firefox-5c9444c5db-hdqmk 1/1 Running 0 9m48s selenium-node-firefox-5c9444c5db-hhh88 1/1 Running 0 47s selenium-node-firefox-5c9444c5db-j448l 0/1 Pending 0 47s selenium-node-firefox-5c9444c5db-ltd2w 0/1 Pending 0 47s selenium-node-firefox-5c9444c5db-q9pd2 0/1 Pending 0 47s selenium-node-firefox-5c9444c5db-xfgdt 0/1 Pending 0 47s selenium-node-firefox-5c9444c5db-xxbr6 0/1 Pending 0 47s` |
We can also autoscale horizontally when all of the pods cpu-percentage is more than x% – if this threshold is passed a new pod will be created to run:
|
1 2 |
PS C:\temp\k8s> kubectl autoscale deployment selenium-node-chrome --cpu-percent=50 --min=1 --max=10 horizontalpodautoscaler.autoscaling/selenium-node-chrome autoscaled |
If a pod crashes we can delete it using the command: kubectl delete pod <podname>
After doing this, the pod will automatically be recreated in an effort to self heal.Edit
|
1 |
kubectl get deployments |
|
1 2 3 4 5 |
PS C:\temp\k8s> kubectl get deployments NAME READY UP-TO-DATE AVAILABLE AGE selenium-hub 1/1 1 1 16m selenium-node-chrome 1/1 1 1 11m selenium-node-firefox 3/10 10 3 11m |
kubectl get hpa -w (see replicas increase and decrease as a test run is being executed)
kubectl exec selenium-hub-7ff45ff687-swvjg – printenv (See config for a pod)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
PS C:\temp\k8s> kubectl exec selenium-hub-7ff45ff687-swvjg -- printenv PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=selenium-hub-7ff45ff687-swvjg KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1 KUBERNETES_SERVICE_HOST=10.0.0.1 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT=tcp://10.0.0.1:443 KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443 KUBERNETES_PORT_443_TCP_PROTO=tcp DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true TZ=UTC HOME=/home/seluser GRID_MAX_SESSION=5 GRID_NEW_SESSION_WAIT_TIMEOUT=-1 GRID_THROW_ON_CAPABILITY_NOT_PRESENT=true GRID_JETTY_MAX_THREADS=-1 GRID_CLEAN_UP_CYCLE=5000 GRID_BROWSER_TIMEOUT=0 GRID_TIMEOUT=1800 GRID_DEBUG=false GRID_HUB_PORT=4444 GRID_HUB_HOST=0.0.0.0 |
kubectl logs <podname> (See logs from a node, for example)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
PS C:\temp\k8s> kubectl logs selenium-node-chrome-778787c6f-x2bdd 2020-06-19 08:47:14,852 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing 2020-06-19 08:47:14,854 INFO supervisord started with pid 7 2020-06-19 08:47:15,856 INFO spawned: 'xvfb' with pid 10 2020-06-19 08:47:15,857 INFO spawned: 'selenium-node' with pid 11 2020-06-19 08:47:16,858 INFO success: xvfb entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) 2020-06-19 08:47:16,858 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) Connecting to the Hub using the host selenium-hub and port 4444 08:47:17.534 INFO [GridLauncherV3.parse] - Selenium server version: 3.141.59, revision: e82be7d358 08:47:17.937 INFO [GridLauncherV3.lambda$buildLaunchers$7] - Launching a Selenium Grid node on port 5555 2020-06-19 08:47:18.161:INFO::main: Logging initialized @1275ms to org.seleniumhq.jetty9.util.log.StdErrLog 08:47:18.944 INFO [WebDriverServlet.<init>] - Initialising WebDriverServlet 08:47:19.244 INFO [SeleniumServer.boot] - Selenium Server is up and running on port 5555 08:47:19.244 INFO [GridLauncherV3.lambda$buildLaunchers$7] - Selenium Grid node is up and ready to register to the hub 08:47:19.435 INFO [SelfRegisteringRemote$1.run] - Starting auto registration thread. Will try to register every 5000 ms. 08:47:20.276 INFO [SelfRegisteringRemote.registerToHub] - Registering the node to the hub: http://selenium-hub:4444/grid/register 08:47:20.435 INFO [SelfRegisteringRemote.registerToHub] - The node is registered to the hub and ready to use |
Simple, just point them at your new grid! A very crude test example is below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
[TestMethod] public void TestMethod1() { var options = new ChromeOptions(); options.AddArgument("--headless"); options.AddArgument("--no-sandbox"); options.AcceptInsecureCertificates = true; var driver = new RemoteWebDriver(new Uri("http://20.50.172.134:4444/wd/hub/"), options.ToCapabilities()); driver.Navigate().GoToUrl("https://www.google.com"); driver.FindElementById("hplogo"); driver.Quit(); driver.Dispose(); driver.Close(); } |
Written By
A slight departure from the usual testing related stuff, awhile ago I made a chat bot service in Azure in a fleeting attempt to dip my toes into AI based tech. The below is a hopefully faithful recreation of how I did it, I really wish I documented it at the time. Make a QNA Service in Azure To get started, log into Azure Portal and make a Resource Group. Add the resource QnA Maker, s...