My 1st house!

Odd as it sounds, as of May 2022, I’d never been a property owner (house/condo). A trip to Arizona inspired me to become a home owner. Why? Long-term, I’d like to do as my parents do, and become a snowbird. They travel to Florida each winter from Winnipeg, MB, Canada where I grow up. Montreal, Quebec, where I reside is not as cold as Winnipeg, but really, everywhere in Canada has the “winter tax”, and I’d like to stop paying it ❄️🌨️

My criteria for the house was simple, but was at odds with the condo type dwelling common for most folks from Montreal. However, as I grew up in Winnipeg, MB, I’m used to BUNGALOWS. So, I wanted an un-attached house, small, close to 1200 square feet, front and back yard w patio, and room for one car (which as of this posting, I still don’t own, but soon will)


On my return from Arizona, I mentioned my snow-bird plan to my then landlord in Verdun, Montreal, Quebec. She got me in contact with her BF’s friend who’s a realtor, thank you Emilie!

I chatted with my new realtor Mathieu on the phone, gave my criteria, and away we went! To contradict myself, I did entertain the idea of a condo for one a single day of visits, just to keep my mind open to calling an apartment a home (aka, condo), after 3 visits, I backed out of the remaining visit, and advised Mathieu we should focus only on small houses going forward. I visited 2 houses on my own, and one final house with Mathieu, and the 3rd one was the winner! Here were the original realtor pix that sold me on the modern updated interior

I bid / won in mid May , completed my home inspection, and moved in Sat May 27, 2022

I’m a huge fan of Microsoft OneNote, and I’ve used it to organized previous moves. For this move, the list of to-do items became expansive very fast. I set tasks into “week 1” “month 1” , “first six months”. This included minor fixes for electrical issues, getting gutters installed, painting scratches made during moving, etc

Some highlights:

  • I got a chance to consolidate my home lab gear into a single server rack, finally! This is something I wanted to do for years
  • I’m KINDA color-blind, so, complex shades of blue/green/etc I don’t see so well. The house is mostly black/white/dark brown, so buying items for it have been EZ! I mostly stick to black/white items
  • I’ve enjoyed simple fix-er-up tasks: replacing a tub spout, caulking my tub, adding a missing step to my patio, leveling out my washer / dryer with anti-vibration feet. Like anything, if you don’t know how to do it, someone has posted it to YouTube! Review & measure twice, and you’ll be good to go
  • I’m really enjoying the gardening / landscaping / lawncare side of things. I’ve bought a bunch of ‘exotic’ plants that inspired the idea to buy the house when I was in Arizona in April, and keep adding more! I started with 3 provided by my landlord, and now 7 new plants: ZZ, Palm, Yucca
  • OMG, it’s SO quiet. the interior of the house has all new windows, with excellent sound-proofing. For whatever reason, my hearing appears to be improving with AGE!? I can’t explain it, it’s a bit of a burden, and certainly explains why I’ve moved into a house from an apartment rental where you get sounds from above/below. As well, I can now crank my SONOS wireless system up beyond 15%

Getting ‘administrative events’ from a janky system

Recently, I was helping a co-worker with an issue noted where Win 2019 MCS clones going into a janky state intermittently. I’m a big fan of the windows event viewer for immediate troubleshooting, however, the system was so degraded, we were unable to use compmgmt.msc to remote to the system to open eventvwr.msc

However, we were able to remote in via PowerShell

enter-PSsession -jankyVDA

I’m familiar with PS cmdlets get-winevent, but in this case, we didn’t want to filter through tons of errors, instead, we just wanted the ‘admin events view’, which is a filtered view of only warnings/errors:

How do I get only the admin events via PS? Thanks to this REDDIT post, I now know

https://www.reddit.com/r/PowerShell/comments/bitgnc/script_to_pull_all_administrative_events_in_event/

The code (below PS code was converted to HTML via this tool )

to HTML )
$xmlFilter = "$($env:TEMP)\adminFilter.xml"
$header = "<QueryList>`r`n  <Query Id=`"0`" Path=`"Application`">"
$footer = "  </Query>`r`n</QueryList>"
$loglist = @()
$EventLogs = Get-WinEvent -Force -ListLog * -ErrorAction SilentlyContinue
foreach ($Log in $EventLogs) {
  if ($Log.LogType -eq "Administrative") {
    $loglist += $log.logName
  }
}
set-content $xmlFilter $header
foreach ($logName in $loglist) { Add-Content $xmlFilter "    <Select Path=`"$($logName)`">*[System[(Level=1 or Level=2 or Level=3)]]</Select>" } 
add-content $xmlFilter $footer
#start notepad $xmlFilter 

$aa = Get-WinEvent -FilterXml ([xml](Get-Content $xmlFilter))

With the value of $aa created, we can easily export it via the below one-liner

export-csv -NoTypeInformation -Path c:\admin\RecentEvents.csv

Then you can grab the .csv , open it in excel on a working machine and review the events as required!

Arizona April 2022

My trip to Arizona April 7 to 14th, 2022 was a long-time coming. I’d originally wanted to go in Dec 2019. However, I decided to stay in Montreal for the holidays. 2 months later, covid-19 came in put in a wrench in any travel plans I might have had for the states for all of 2020

2021 came around, and Steve Greenberg’s EUC Master’s retreat event came up! However, it had to be cancelled due to Covid-19 concerns. ⏩ to April 2022, and the event is back on, and my AZ plans are back on track!

I LOVE travelling to the United States for tech related events, for the past 5 years I’ve attended the below events, some paid for by my employer, others I foot the bill. ALL were worth it for the experience of the respective city / meeting people / professional development

  • Expert to expert, NYC 2017
  • Citrix Synergy Atlanta, Georgia 2019
  • IGEL Disrupt Feb 2020, Nashville, Tennessee
  • EUC Master’s retreat, April 2022, Scottsdale, Arizona

For the Arizona trip, I decided to extend my trip beyond the Friday to Sunday scheduled dates of the conference. I’ve been in dire need of a break from Canada. The covid-19 response here in Canada is about 6-12 months behind the rest of the world. Matched only by our common-wealth brethren in Australia & New Zealand. That is, endless lockdowns, curfews here in Quebec, and hygiene theatre that all provide little to no benefit to dealing with the pandemic C19 endemic

/end covid rant

I was unable to bring my beloved beast Coffee Bean for this trip. So, it was the first time in 2 years, I’ve been without her. About half way through my trip, the fancy Montreal doggy hotel, sent me a pic of her sitting on a chair looking sad AF. I felt like she was staring into my SOUL, compelling me to come home. I immediately thought of the angry cricket fan MEME dude

Did I give in to her guilt-tripping tricks? NO!

The conference was amazing. Even the hotel was top of the pops! DAMN. Here’s some pix from the Saguaro in Scottsdale

Conference-wise, what I really enjoyed, was it wasn’t all NERD STUFF! I’d say about 60% of the content was technical, 40 non. I’ve done quite a few on-line tech talks, but NEVER in person. You may wonder “Owen, how can this be, whenever I talk to you in person/zoom/phone, I can’t get you to STFU, and you’ve got a hot take on EVERYTHING!”. Well , you’d be right, I am very comfortable with speaking in person to one or more people at a time, but NOT WHEN IT’S SCHEDULED. The organizers of the event setup a planning session Friday evening to decide on break-out sessions that would be done on Sat/Sun. We passed a mic around, and voted on the topics we wanted to hear about / speak about. I raised two topics close to my heart: Sleep / IT standards via automation. I was able to speak on both topics over the next 2 days, awesome! It wasn’t so nerve wracking, as I mostly just spoke from memory, I feel sticking to a script / PowerPoint slide would make it more stressful. Kind of like trying to re-play a song on guitar note for note / chord for chord, ya miss something, your audience will know!

The conference ended on Sunday April 10, 2022. I said my good-byes to everyone I could find, and changed over to another hotel about 15 mins away

Now begins the second part of my trip, which had much looser schedule

Step one, acquire BLUE BOY! That is, this super awesome futuristic car , the Hyundai Ionic 5

TBH, I’m not sure why I was so hell bent on renting this particular car. I’m a bit of a contrarian I guess? The more obvious / cheaper choice would have been a Tesla Model 3. I ended up getting the only Ionic 5 in the Phoenix area, and it cost me about $30 more per day vs the equivalent from Tesla. That being said, OMG, what a ride. The interior is STRAIGHT UP STAR TREK TNG. Electric cars essentially have instant torque, so, in sport mode, you put your foot down, and the car FLIES. Navigation, seats, interior, road noise were like nothing I had experienced when doing my exhaustive test drives last year

I used boy boy to get around Scottsdale / Phoenix for sight-seeing and to visit locals I knew in the area. Here in Quebec, I don’t do much hiking, as I find Canadian terrain boring. Prob from living here for so long (lakes/trees/bears/blah). The grass is always greener, and the desert is thrilling! Cactus / scorpions / palm trees!

Here’s some pix from a visit to the desert botanical garden on Sunday April 10, 2022. I want to touch them ALL, but I know I shouldn’t

Monday April 11, 2022!

Time to take blue boy out for some shopping. 2017 onwards, I collect shot glasses wherever I go. Here are some day / night time shots of blue boy

Tuesday April 12, 2022

Went for a brief hike on gateway trailhead. During the hike I thought, if I lived here, how long would it take me before I would take this view for granted like my man Larry David says in this GQ article. CYE 4 life

My beloved dog coffee

Wed April 13, 202

My last day! I hooked up with the event organizers Steve/Beth to check out cave creek, where Steve lives. Such a beautiful area. Enjoy my last set of pix from AZ!

Thursday April 14, 2022

The road home! I prefer afternoon flights, as I struggle to wake up early. That being said, I love all aspects of flying. I recently bought an iPad Pro 12.9 for reading in bed, I brought it along with me for plane TV/movie stuff and copied over some recent tv/movie content. On the way back, I noted that the dimensions of my iPad stand fit perfectly into the sleeve of seat in front of me where the barf bag is! My plane had janky wifi, and no screens in coach, so, I had the ultimate in-flight infotainment system to watch the new HALO tv series (which is great) and “Lock stock and two smoking barrels”, which is not as a good as I remember it!

My flight from Phoenix had a brief lay-over in Toronto, there was some drama with a shortage of TSA / custom agents to process incoming travelers, which created an epic line. I was about half way through said line of about 300 people going through security checks when a beautiful tall black Gal Gadot / Wonder woman-looking lady appeared asking if anyone was connecting to Montreal. She allowed me to skip the line. As a polite Canadian, of course I apologized to everyone I cut.

Thankfully, I was able to board and get home to my beloved dog , who my gracious land lord Emilie had taken care of a few hours after coffee bean was dropped off by the doggy hotel pet taxi.

I will tell you, it had only been a week, but I teared 😭😢😿up when I got home and saw her waiting at the top of the stairs for me. However, I immediately stopped crying when I got close to her, she STANK of other dogs at the doggy hotel! 😆🤣😂😹

BATH TIME FOR BEAN RAY!

My checked luggage filled with Arizona snakes, rocks, and cactus got stuck in Toronto for one day, but who unpacks their bag as soon as they get home from vacation? Not me!

In summary ; this trip was extremely satisfying both professionally and personally. I’m so glad I got to go, and am very excited to return next year for the next EUC Master’s retreat!

Thanks for reading 🙂

Owen

Custom offline ISO Windows deployment method as a packer alternative

Background

For 2022, it was time to start using the packer automation I learned / wrote about in 2022, however, when I first tried to use it with my first 2 clients, it failed, in each case, for a different reason. On a third attempt to use it, I didn’t have access to the client’s environment, so had to talk the client’s on-site staff through filling out the required entries in the related packer XML/JSON/HCL files. By the time we were done filling out verifying the details, 30 mins had elapsed, the avg time to build a new Win 1x, Win Server 20xx image with windows updates is about 25 mins. At this point, I was 0/3 on using packer for my client work. As per this post, I’m often only with a client for 5-10 business days, burning an hour or more troubleshooting / prepping packer where it won’t be used again is not a good use of time. As such, the offline/custom ISO method was born!

This blog post will describe how I made it work, as of April 5, 2022, i’m 3/3 on new deployments with the new method. Sorry, Packer bros

Comparison of packer vs custom offline ISO method

For many years, windows installations have been driven by esoteric config file called ‘autounattend.xml’. For my packer based methods, I included samples, my new offline method also uses these files. however, the key difference is in the work flow:

Packer method

  • You download the latest Windows ISO to your desktop: 10 mins
  • You upload the ISO to the Hypervisor datastore: 10 mins
  • Packer > Creates VM on Hypervisor ( I only got it working on VMware env): 30 seconds
  • Packer copies over autounattend.xml / ps1 scripts to newly created shell: 10 seconds
  • Windows install starts: 10 seconds
  • Packer waits for WinRM to be enabled on new windows install: 15 mins
  • Packer shuts down VM: 10 seconds
  • You boot the VM again to apply any run once / scheduled tasks you set in previous steps: 15 minutes

All of the above is 100% dependent on properly formatted Packer JSON/HCL files, one mistake, even a lowercase/UPPERCASE character and your build is hooped

Let’s look at the method i’m now using for all my builds, which is a custom ISO injected with all the same scripts I was using on my packer-based builds

  • You download the latest Windows ISO to your desktop: 10 mins
  • You create a shell on the hypervisor ( tested as of today on VMware/Nutanix): 2 mins
  • You inject the latest window ISO with the required custom files (hypervisor drivers/XML/PS scripts): 5 mins
  • You upload the custom ISO to the Hypervisor datastore: 10 mins
  • You start the shell and boot it to custom ISO: 10 seconds
  • The rest is completely automated, as well, you don’t have to edit JSON/HCL configs, deal with network / firewall issues / or type-os where you or the client chose the wrong datastore / host / network / etc: 15 mins
  • Time to completion: Roughly the same, the difference, no 🙉🙊🐵business to deal with

The process / steps to get it done

  • Download ISO for Win 10 / Server 2019 / Server 2022 on your jump server / desktop, ensure you’ve got a fast link to your Nutanix Prism/VMware vCenter, don’t do this over VPN, son!
  • Mount ISO on your desktop
  • Run DISM to capture IMAGE INDEX as per Take Inventory of an Image or Component Using DISM | Microsoft Docs, this value will be required to amend the autounattend.xml that we will inject into the custom ISO
  • Dism /Get-ImageInfo /imagefile:D:\sources\install.wim (PATH TO YOUR WIM)
  • Download a sample Win 10/server autounattend.xml from my git hub here:

    https://github.com/getvpro/Standard-WinBuilds/tree/master/Offline_Builds/Autounattend_xml
  • Open it with a proper text editor: notepad3, VS code, NotePad++ , etc
  • CTRL+H to search/replace through all entries that state “CHANGE ME”, password / organization, save it
  • Search for IMAGE/INDEX, amend as required based on above screenshot, each index represents a different type of OS install, choose the one you want so you don’t have to do it twice
  • Install AnyBurn: https://www.anyburn.com/download.php
  • Open AnyBurn, brose to the ISO you downloaded, choose EDIT ISO
  • Open Powershell as admin and run the following

new-item -ItemType Directory -Path c:\admin
new-item -ItemType Directory -Path C:\admin\Offline_ISO
new-item -ItemType Directory -Path C:\admin\Offline_ISO\hypervisor_drivers
new-item -ItemType Directory -Path C:\admin\Offline_ISO\Langpack
new-item -ItemType Directory -Path C:\admin\Offline_ISO\Scripts

  • Upload ISO to hypervisor datastore
  • Start VM and ensure it’s set to boot to the ISO you uploaded

The rest of the install will proceed in a similarly automated fashion to the packer business. HyperVisor drivers for VMware / Nutanix / Citrix Xen Hypervisor will be installed, some basic post-windows install tasks will be run, including installing any language .cab files you might have populated into the Lang_Pack folder. When this part is done, scheduled tasks to cover starting / monitoring the windows update process will be run

That’s it!

As I said, my last 3 projects I’ve used this, in each case, I was able to finish my work early/on-time

Owen



Monkey biznuzz time mgmt startegy

🐵

If you’ve read my bio/current job/recent posts, you’ll know I do It consulting for a company in Quebec, Canada. 90% of my projects are 2 to 3 weeks long in scope, if we run out of time, two things will happen:

-The project will be paused
-The client will need to buy more hours

Neither of which is great

I’ve been in my current role for 2.5 years, more than half of my projects finish on-time, this is typical for everyone in our team, why? 🐵🐒 business

Defining Monkey business / examples

‘THINGS YOU DIDN’T ACCOUNT FOR / PLAN FOR THAT MESS UP YOUR SCHEDULE”

Here are some recent examples from my work over the past few years:

  • Client not having DHCP setup anywhere in their environment, which meant my beloved packer automation would not work
  • Worse, Extra DHCP servers authorized in AD that have bad scope options set that provide incorrect DNS info
  • Citrix Hypervisor (XenCenter) any $ you saved on using Citrix HV instead of a proper hypervisor like Nutanix / VMware , will certainly be lost to troubleshooting efforts / downtime down the line
  • Messy AD environments, clients still using FRS , clients who’ve not updated ADMX templates in 10 or more years
  • In-house apps that require manual efforts to get working on newer OS versions: Win 1x, Win Server 20xx
  • Lost private keys to TLS certificates
  • Internal windows update servers running against new OU’s , and no-one has access to manage the servers to stop it from happening
  • Incorrect licenses purchased: ESXi, Nvidia, RDS, etc

The above is just a sample from what I’ve seen. Bottom line, 🐵monkey business🐵will happen, it’s just a matter of when and how much time you’ll lose trying to stop said monkey’s from jumping on the bed

Mitigating monkey business

For the first year of my job, I mostly followed a ‘ronin / cowboy’ method of delivering projects, I did what I thought was best and called my co-workers for help when stuff went wrong. My work was more chaotic and stressful as a result

I like Chuck Berry, but this song was playing in my head far too often

I would estimate a lot of my projects ended up looking like this from a pie-chart perspective:

Coming into year 2, me and the other dudes on my team started to do weekly ‘best practices meetings each Friday afternoon. We didn’t just talk, we wrote stuff down into our WIKI, and made it LAW! As such, I started to follow a more pedantic method

  1. Following the OSI layer of troubleshooting, this has been drilled into since taking a Cisco CCNA course in 2004. I never did sit the exam to get the accreditation, but the troubleshooting steps remained with me. Here’s my related blog post on the topic

    TLDR; start with the basics when troubleshooting an issue, start at the physical layer and work your way up, troubleshooting layer 7 ‘app issues’ is not where you want to ever be, or to start
  2. Follow an SOP for your project deployment. In my job, we use Perfect WIKI, which is a teams add-in. Within your document(s), define how you deploy images, GPOS, install apps, etc
  3. A tie-in to the above, automate your standard operating procedures (SOP) as much as possible. Hand-installing an OS/apps/windows updates isn’t the way to go anymore, it’s error prone and slow
  4. Review the client’s environment during initial meetings, ID any potential sources for monkey biz, and if possible, have the client to agreed to resolve these issues before you start

Following these 5 points for my last 2 Citrix implementations, I’ve ended up finishing each project EARLY. I’d estimate my time spent on 🐵🐒biz is down from 50% to 30%.

As a result, I’ve got more time for the new items listed in the following pie chart:

1) Reviewing / improving the implementation
2) Providing hand-off docs to the client

Following an SOP that uses automation makes it easier for your co-workers to take over your work if you’re away on vacation/sick leave/etc. They can refer to internal docs / github to ID how you did your stuff, and if your automation uses logging (I hope it does) then ID when you did your work to the hhmmss 🙂

Now! I am just one person. In the comments, let me know how you do your own project work and/or any interesting examples of ‘monkey business’

Owen

Measuring the impact of ‘Direct workload connect’ on Citrix Cloud session brokering time with ControlUP / CQI

In this blog post, I’m going to talk about a Citrix cloud feature you might have heard of, but not implemented. It’s called Direct workload connect:
https://docs.citrix.com/en-us/citrix-workspace/optimize-cvad/workspace-network-location.html

In this post, I will review how to get it working, and most important of all , how to measure the impact of implementing the feature using numbers. I will use simple / free tools to show the impact: Citrix Connection Quality Indicator and a single PowerShell script from ControlUP contributor Guy Leech

The story. I had a client in March of 2022 who went ‘all-in’ with Citrix Cloud for their control/access layer. As part of the work, they had requested to review the best means to route traffic intelligently based on internal / external users using Citrix Cloud workspace/gateway. To be honest, I slept on both V1 / V2 of ‘Citrix Rendezvous‘, as well as ‘Citrix workload connect’ when they were respectively announced over the past few years. Rendezvous I thought had something to do with Apple ‘bonjour’, so, in my mind, I though it was related to optimizing audio/video steams that ride on HDX? Kind of correct! However, what if you want to reduce the time your internal users spend waiting on the below screen? Will rendezvous help reduce your normal Citrix Cloud session brokering of 5-13 seconds? After much testing and one slack post , I confirmed it will not:

Enter ‘Citrix direct workload connection’

While we work from home a lot more these days, many of us are back in the office, and want to full speed access to the virtual resources we were using from home

The steps to set up the ‘Citrix direct workload connection’ are easy enough. In fact, my man Eric Trond had an enthusiastic post from last year on the topic with a related script which I used for my first crack @ it. In the above post, he described a huge ICA RTT reduction following implementing it. Eric is in Brazil, his Cloud Instance was in the states. He was able to reduce his ICA RTT from 769 MS to 17 MS. MAN! That’s great. His results are provided by the free tool ‘Citrix Connection Quality Indicator’ should be on all your Citrix VDAs

To start, let’s capture some BEFORE examples, in this way, we can measure the before/after impact of enabling the direct workload connection feature.

We will do this in 3 steps:

1st: Follow this ControlUP article to set all the required GPO settings to enable audit logging so that you can run their powerful ‘analyze logon duration’ script. YES, there’s a lot of settings that you might not have set before, NO, don’t skip any of them

2nd: download the Control up analyze logon script to the VDA you’ll be using for testing, for me, i’m keeping this script in a local folder on any golden images I deploy. If you don’t want to do a PVS/MCS update just to have the script on the c:\ drive, you can just copy it over for the duration of your non-persistent session, you are using non-persistent machines for your clients, right? :p

3rd: install Citrix Connection quality indicator on your target VM, as above, you should have this in your golden image already, it’s free / very useful. If not, you can RDP to your target VM with an admin account and install it, you’d then need to configure the related GPO to enable it.

Once CQI is installed, copy over the related ADML / ADMX to your AD from here: C:\Program Files (x86)\Citrix\Connection Quality Indicator\Configuration

….to your PolicyDefinitionsFolder on your AD. Don’t have a central policy store? Create one via the following MS KB:

With the above 3 steps completed, logon to your VDA via Citrix cloud as you normally would. Open CQI to collect the ICA RTT, take a screenshot and save your desktop / Onenote / wiki / etc

I’ve gone though this exercise 3 times now, one on my lab, twice for clients, here’s one of the results BEFORE enabling the ‘Citrix direct workload connection’. It’s not terrible, but considering the connection is being made from a PC from within the same internal network where the datacenter VDAs reside, it’s not great

Next, you will want to capture additional metrics via the ControlUP analyze logon duration script you downloaded in step 2. Once logged on fully to your VDA session , open Powershell as admin, and CD to the directory where you downloaded the analyze_logonduration.ps1 script

Run the script against the session you logged on to VDA via ICA in the following format:

Record your results! Here, we have 14.9 seconds of brokering time, yikes, that’s almost enough time to call your significant other and tell them you’ll be late coming home, because your VDA logon was slow! 😂🤣

With your results saved (don’t forget!), you can follow the Citrix guide for the required steps to enable workspace direct connect on your Citrix Cloud instance

You will need to setup network locations for all the public IP (egress) points for your office(s), for the client I setup last week, there were just two, you could have lots. You’d only want to add the entries for locations where internal office workers are connecting to Citrix resources via Citrix cloud. Once the locations are set, it’s time to test!

For troubleshooting, it’s a good idea to Follow this KB from Citrix to enable workspace app ICA logging for the currently logged in user on the internal network PC you want to test on

With the above logging in place, let’s repeat your VDA logon tests

Logon to Citrix Cloud again, open the ICA logging file you set via the above Citrix KB for enabling workspace app ICA logging. Within the ICA log file, CTRL-F to ‘address=’, you should see the address appear as the name of the VM you’re connecting to. Here’s a snap of a working direct workload ICA connection:

If it’s NOT working, and you took the long route to your internal VDA via Citrix Cloud, you’ll see the familiar STA ticket reference in the ‘address’ field. This is a clear indication of 🐒🐵biznuzz. It happened to me when I was first testing this on my lab, I had set an incorrect public IP / subnet re-check your network location and re-test to resolve monkey business

Now, back to benchmarking. If it works, you should see MUCH less time on this screen

But what is ‘less’? it’s relative, let’s re-run our tests and capture some AFTER metrics

Start with CQI. Oooh boy, what do we have here? ICA RTT @ 2 MS / latency of 1 ms! We are off to a great start!

Next, time to fire up the ControlUP analyze logon duration script again!

What do we have here? From 14.9 seconds to 2.4 seconds from the time you clicked launch to actual session launch. An 83% reduction! SNAP. Pat your self on the back, you’ve made a measurable impact to your internal Citrix Cloud users

That’s more like it. You ever see the bills for the new hosting hardware that runs your VMs? I sometimes do, they aren’t cheap. Giving your internal Citrix Cloud users the fastest possible logon times will make it easier for mgmt to justify ‘going to the cloud’ when the annual renewal comes up

Thank you for reading and have a great day 😁

Owen

Packer Part 3 – Windows 11 build

Intro

Hot on the heels of the CUGC presentation I did 2 weeks back with my co-worker / friend Jonathan Pitre, I took some time this week to update my Packer config files for Windows 11. I have a client who wants to do a Citrix POC with Windows 11, so, let’s brain dump all the stuff while it’s still fresh

Btw, the related blog post on Win 2022 packer is here , and the YouTube video of the actual CUGC presentation is here

Goals and pre-reqs

The goals are the same as before, deliver a fully built windows golden (reference) image with as few clicks as possible to a VMware vCenter cluster. In my case, my entire build process is started / managed / logged by PowerShell. The Packer.exe is called via a base script, that’s used to power on the newly created Packer VM after the initial packer provisioning is done. I will cover the setup steps in the next section.

The pre-reqs for this build are as follows:

  • Download a non-trial version of Windows 11 from your Microsoft portal, I use my.visualstudio.com . Once it’s downloaded, upload it to your VMware datastore and give it a human-readable name, packer config files are case-sensitive. I’d suggest win_11_month_year_business.iso
  • Upload a recent VMware tools ISO to your datastore, again, keep the name simple. I included a sample name in my packer HCL config file. I’d suggest vmt_11_3_x.iso
  • Run through the following to setup your vCenter as a key provider, this is a pre-req for the virtual TPM requirement of Windows 11

Setup

On your windows machine create a directory structure as shown below, starting with c:\Program Files\Packer

Within the config sub-folder create two folders, one called HCL, one called autounattend

Files will be downloaded to each in further steps

Download / extract packer from packer.io to the c:\Program Files\Packer folder you created

Set a system environment variable to c:\Program Files\Packer

The config files

Download the required Win 11 HCL / XML templates from my github HERE:

Start with the HCL file, open it using a text editor (like Notepad3)

Edit line 31 to cover the path to where you store the Windows 11.ISO downloaded in the previous section

Edit line 122 to point to the datastore path where you uploaded a recent VMwareTools.iso

CTRL + H to do a search/replace for anything labelled as “CHANGEME”, amend as required for your environment

Download the Win 11 Autounattend.xml sample file from my GITHUB HERE

Open the file in NotePad3 or a similar editor, use CTRL + H to search and replace through the file for any references to CHANGEME. These are mostly references to local admin name/password, ensure you set the username / password the same as the HCL you just edited

I converted my JSON legacy template to HCL this week. The newly updated HCL has a call to download plug-ins via the following command which you need to run to download the vSphere plug-ins to your %appdata% directory

open CMD as admin
CD to c:\Program Files\Packer
packer init “C:\Program Files\Packer\config\HCL\Win11_EFI_Enterprise.json.pkr.hcl”

This is required in order for the vTPM variable on line 116 of the Packer HCL to work

You’re just about done with the download/setup of your config files, one last set of files into a new directory: c:\Program Files\Packer\Scripts\pvscsi

Within the new directory, you will need 4 files from the windows VMware tools ISO. They are buried deeeeep!

With the ISO mounted on your windows machine, find them here, changing the “D:” drive accordingly:

D:\Program Files\VMware\VMware Tools\Drivers\pvscsi\Win8\amd64

The requirement for these files, is to ensure that your Win 11 EFI install detects the VMware paravirtual SCSI controller during the install. Note: I did not have to follow this process on my Win 10 / Win 2022 EFI based installs. TBH, I’m not 100% why this is now a requirement, but will update my blog post If i find out, else, if you know, let me know in the comments

Download the final script, to c:\Program Files\Packer this script is used to start the packer build

If you don’t already have it installed, install VMware PowerCLI , it’s required for the above script

Starting the build

open Powershell as admin, then CD to c:\Program Files\Packer

Start the script: .\Start-PackerBuild

You will be asked to choose the OS , 1 for Win 11, 2 for Windows 2022

You’ll be asked for a VM name, this should be the same as is set in the related JSON/HCL file

You will be asked for your vCenter instance name

You will be asked for vCenter credentials

Assuming the VM doesn’t exist already, the Packer build will start, if you choose a VM name that exists in your environment already, the script will exit. This is done for safety reasons. In my lab, I run packer with a -force switch to delete the existing VM, I wouldn’t recommend this in prod, you might have a co-worker also creating / testing VMs on your vCenter environment, and will delete their VM if you use ‘-force’ when launching Packer

Watching the build

The base windows install will proceed in En-US. All reboots / logins / prompts / selecting the disk, putting in the volume license key etc are automated

Note: the VLK can be amended later via the normal method:
Windows > Activation > Change key

The first script to run on the first reboot is the Install-VMwareTools.ps1 script, this needs to be done to enable Packer to pickup the IP of the VM shell it’s created on your vCenter infra

Once VMware tools is installed, the Start-FirstSteps.ps1 script will run. It contains the most important steps, as it downloads 2 more scripts / scheduled tasks from my github

The actual order of execution of all these scripts is shown in the following screenshot:

I live in Montreal, Quebec. A lot of business need to have both of Canada’s official languages installed on their systems to maintain compliance with these folks https://www.oqlf.gouv.qc.ca/accueil.aspx. These lads can levy fines of up to $7000 Cdn for not having Fr-Ca support on a computer system. This includes Fr-CA physical keyboards. With this build automation, my end of it is the Fr-Ca language pack.

My French speaking/listening skills aren’t great, but I can read it well enough, and can certainly navigate windows when it’s running in Fr-CA, enough to automate and configure!

The Start-FirstSteps.ps1 script automates the entire process of download the language cabs for Fr-CA from my github, installing it via Add-WindowsPackage , as well as the last part, to actually add Fr-CA as a display language you can see from the notification area of the task bar. Of the new code I added to the Start-FirstSteps.ps1 script from last year to now, this part took quite a while. Lang packs are per OS, so you need to DL the correct one for your exact OS build: Win 11 / Win 10 1909 / 20H1 / 21H1 / Server 2016 / Server 2019 etc.

I’m probably breaking some kind of law by hosting the Fr-CA .cab on my github, but I don’t care, come @ my , MS bros. You owe me for the Windows updates nightmares over the years. Apparently, from your non-existing QA !

/end rant

The script downloads the multi-part zip and extracts it

If you want to be MORE complaint and have access via your employer to my.visualstudio.com, you should download the entire pack and adjust the lines in the script that download the Fr-Ca.cab

if you don’t need Fr-Ca support in your image, open the related example Autounattend.xml from my github, and search for the following:

CMD /c reg.exe ADD “HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment” /v FrenchCaLangPack /t REG_SZ /d 1 /f

Change the /D 1 to /D 0

Once the Fr-Ca language pack is installed, the machine will reboot, and start processing two custom scheduled tasks. The first scheduled task will start windows updates, the second will monitor it’s progress. Of all the new code I’ve this packer v2 project, this piece was the most challenging and frustrating. For about 3 years, i’ve been happy using the Powershell module PSWindowsUpdate to automate installation of windows updates on my lab, and for some clients. However, during regression testing of my packer v2 build , I was NOT able to get PSWindowsUpdate to successfully apply windows updates 100% of the time. It worked well, but would get stuck during the download process about 50% of the time, which is not usable for an automated build. I tried all kinds of work-arounds , but could never do better than 50%. As such, I decided to seek out an alternative method, and found one that’s actually built into windows! It’s an exe I’d never heard of before called UsoClient.exe
The exact line I’m using is UsoClient.exe StartInteractiveScan

This opens the familiar windows update UI we use to patch windows interactively, however, with the “StartInteractiveScan” option selected a scan is done, and patches start applying right away

The imported scheduled tasks run in the system context, the default behavior for running a PS script via a scheduled task in the system context means the logged in user won’t see the output of the script or even the windows update UI, which isn’t ideal.

So, to present this info to the logged in user, my co-worker Jon Pitre recommended I have a look at an SCCM component called ServiceUI.exe. Launching Powershell via this exe will show the output to the current user, neat! In this way, the various stages of the packer build can be shown to the user. There are several auto logons set for the build to cover the reboots after windows updates have applied

Once it’s determined by the Monitor-WinUpdates.ps1 script running as a scheduled task that there are no more updates to apply, the related scheduled tasks will be disabled. A final window to the user will be shown that indicates the total build time, and that you’re ready to join the machine to your AD domain or run whatever post base build actions you want to for your client environment

With the build done, it’s time to install some apps! For that, you’ll want to switch over my good friend/fellow Canadian CTA/co-worker Jonathan Pitre’s git hub, HERE

The packer build installs the pre-reqs for most of the Jon’s app install scripts: nuget, Winget, Powershell application deploy toolkit, evergreen and more, so, you would just need to choose the apps you need for your golden image and let it rip!

I hope this blog post was useful. I banged it out on a VERY cold Sunday here in Montreal: -18 degrees centigrade 🧊🥶

Have a nice day 😀

Owen

Windows build automation w Packer  / Powershell 2022 redux

Last year, I wrote a long post about using Packer.IO to automate basic VMware shell creation and Win 10 / Win 2019 installation. At that time, I only ended up using the solution for re-builds on own home lab. This year, I’ve had the need to build golden images for multiple clients, each time, the process was manual and error prone as no automation was used.

In the last week of Nov 2021, I decided to sit down last week and re-visit my packer / Powershell windows build templates.

I’m very happy to share that I’ve got automation in place to deliver a fully built / base optimized / bi-lingual (En/Fr) / windows patched Windows EFI image in approx 25 minutes. Last year, the builds were about 10 mins, but didn’t do HAVE of what I have now. Let’s get into it!

For reference, here was the blog post from last year on packer / windows build automation for VMware environments

If you’ve not read it, give it a read, as I won’t be re-reviewing most of the stuff in did in the above post (which was LOOOONG). This new blog post is about the new PowerShell code I wrote to achieve a better level of automation

The Goal

Building golden images for windows is a bit of a mug’s game. Like anything in the mostly unregulated world of IT, there’s not really an agreed upon standard

The scripts / config files are on my GitHub here

To start, you will deploy windows with an autounattend.xml. Autounattend.xml files have been around for a while, and you can use them with packer or MDT or SCCM, or other. The idea is to deliver a windows build with no prompts. The full structure of the autounattend.xml file is described in my blog post from last year

Setup

On your windows machine create a directory structure as shown below, starting with c:\Program Files\Packer

Within the config sub-folder create two folders, one called JSON, one called autounattend

Files will be downloaded to each in further steps

Download / extract packer from packer.io to the c:\Program Files\Packer folder you created

Set a system environment variable to c:\Program Files\Packer

The config files

As above, you can review my blog post from last year if you want a full primer on packer / JSON / autounattend.xml usage

For this blog post, download the required Win 2022 JSON / XML templates from my github here:

https://github.com/getvpro/Build-Packer/tree/master/Config/JSON

https://github.com/getvpro/Build-Packer/tree/master/Config/Autounattend

Start with the JSON file, open it using a text editor (like Notepad3)

Edit line 30 to cover the path to where you’ve got an up-to-date VMwareTools.iso

Edit line 50 to and choose a unique local admin user that will be used for the build process, you can delete it when done, or rename it, but it needs to be the same as is set in the autounattend.xml file you will be editing next

Line line 66-79 for your environment

next, open the Autounattend.xml file

CTRL + H to do a search/replace for anything labelled as “CHANGEME”, amend as required for your environment, ensuring you’ve set the username / password the same as the JSON you just edited

Line 102 can be edited for your local time zone as well

Line 103 can be edited to amend your preferred computer name

The scripts

https://github.com/getvpro/Build-Packer/tree/master/Scripts

There are 4 scripts that are called as first logon activities from the autounattend.xml file

You will need to download them each to the c:\Program Files\Packer\Scripts folder created earlier

Lastly, download the packer start build script, essentially a wrapper that starts packer/then uses PowerCLI to connect to vCenter / start the VM once Packer has done it’s initial provisioning

https://github.com/getvpro/Build-Packer/blob/master/Scripts/Start-PackerBuild-Win2022.ps1

You will need to install VMWare PowerCLI to use the script

Open Powershell as an admin, and run

Install-Module -Name VMware.PowerCLI -AllowClobber -force

..if you get a Nuget package manager can’t be found BS error, run the following:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Then the above again

With the files downloaded, the json/autounattend.xml edited for your environment, you’re ready to start the build

Build process

Launch the Start-PackerBuild-Win2022.ps1, it will ask for the VM name you set in the JSON file for your new packer build VM, your vCenter name and credentials to connect to it. record relevant vCenter info to be used to power on the VM once packer has done it’s first tasks

The Start-FirstSteps.ps1 script contains the most important steps, as it downloads 2 more scripts / scheduled tasks from my github

The actual order of execution of all these scripts is shown in the following screenshot

I live in Montreal, Quebec. A lot of business need to have both of Canada’s official languages installed on their systems to maintain compliance with these folks https://www.oqlf.gouv.qc.ca/accueil.aspx. These lads can levy fines of up to $7000 Cdn for not having Fr-Ca support on a computer system. This includes Fr-CA physical keyboards. With this build automation, my end of it is the Fr-Ca language pack.

My French speaking/listening skills aren’t great, but I can read it well enough, and can certainly navigate windows when it’s running in Fr-CA, enough to automate and configure!

The Start-FirstSteps.ps1 script automates the entire process of download the language cabs for Fr-CA from my github, installing it via Add-WindowsPackage , as well as the last part, to actually add Fr-CA as a display language you can see from the notification area of the task bar. Of the new code I added to the Start-FirstSteps.ps1 script from last year to now, this part took quite a while. Lang packs are per OS, so you need to DL the correct one for your exact OS build: Win 10 1909 / 20H1 / 21H1 / Server 2016 / Server 2019 etc.

As I’m using Server 2022, Microsoft doesn’t include the various language packs on the ISO, it’s a separate download that isn’t titled as Microsoft says it should be. I got it via my.visualstudio.com

I’m probably breaking some kind of law by hosting the Fr-CA .cab on my github, but I don’t care, come at my , MS bros 🙂

The script downloads the multi-part zip and extracts it, but if you want it , the link to the Fr-CA cab is here

If you want to be MORE complaint and have access via your employer to my.visualstudio.com, you should download the entire pack and adjust the lines in the script that download the Fr-Ca.cab

if you don’t need Fr-Ca support in your image, open the related example Autounattend.xml from my github, and search for the following:

CMD /c reg.exe ADD “HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment” /v FrenchCaLangPack /t REG_SZ /d 1 /f

Change the /D 1 to /D 0

Once the Fr-Ca language pack is installed, the machine will reboot, and start processing two custom scheduled tasks. The first scheduled task will start windows update, the second will monitor it’s progress. Of all the new code I’ve this packer v2 project, this piece was the most challenging and frustrating. For about 3 years, i’ve been happy using the Powershell module PSWindowsUpdate to automate installation of windows updates on my lab, and for some clients. However, during regression testing of my packer v2 build , I was NOT able to get PSWindowsUpdate to successfully apply windows updates 100% of the time. It worked well, but would get stuck during the download process about 50% of the time, which is not usable for an automated build. I tried all kinds of work-arounds , but could never do better than 50%. As such, I decided to seek out an alternative method, and found one that’s actually built into windows! It’s an exe I’d never heard of before called UsoClient.exe
The exact line I’m using is UsoClient.exe StartInteractiveScan

This opens the familiar windows update UI we use to patch windows interactively, however, with the “StartInteractiveScan” option selected a scan is done, and patches start applying right away

The imported scheduled tasks run in the system context, the default behavior for running a PS script via a scheduled task in the system context means the logged in user won’t the output of the script or even the windows update UI, which isn’t ideal. So, to present this info to the logged in user, my co-worker Jon Pitre recommended I have a look at an SCCM component called ServiceUI.exe. Launching Powershell via this exe will show the output to the current user, neat! In this way, the various stages of the packer build can be shown to the user. There are several auto logons set for the build to cover the reboots after windows updates have applied. Once it’s determined by the Monitor-WinUpdates.ps1 script that there are no more updates to apply, the related scheduled tasks will be disabled, and final window to the user will be shown that indicates the total build time, and that you’re ready to join the machine to your AD domain or run whatever post base build actions you want to for your client environment

With the build done, it’s time to install some apps! For that, you’ll want to switch over my good friend/fellow Canadian CTA/co-worker Jonathan Pitre’s git hub, HERE

The packer build installs the pre-reqs for most of the Jon’s app install scripts: nuget, Winget, Powershell application deploy toolkit, evergreen and more, so, you would just need to choose the apps you need and let it rip!

I will update this blog post once more when I’ve got my Packer JSON config files updated to HCL

have a great day and happy automating 🙂

Owen






My 5 year quest for a good night’s sleep

UPDATED FOR MONDAY MARCH 28, 2022! I finally received my CPAP machine prescription, it’s now just a matter of waiting to get a machine, but, the hard part is over!

This is my sleep story. It’s a work in progress

Anyone who’s seen my eyes in-person or a video call from about Dec 2021 to now so can tell I’m goddamn tired. I don’t try to hide it, I’m not even sure how I would, maybe a Zoro mask?

The Mask of Zorro Rebooted a Classic Hero - Paste

How did I get here? Let’s review , year-by-year from 2017

2017

I’ve had sleep issues going back to about 2017, at the time, I wasn’t too worried about it. I had left my comfortable staff job at Royal Bank of Canada to try my hand at independent IT consulting. I had a lot to learn, and being a naturally anxious person, day time anxiety eventually started to impact my night-time sleep. I was able to keep my sleep *mostly* stable with heavy gym activities 3-4 times a week with a personal trainer. At this time, I was getting approx 6 hours sleep per night, but rarely waking up, so, it was a solid six hours

2019

I moved to another consulting position that was more demanding, but also more fulfilling. I understood this new job would increase my stress/anxiety and further prey on my sleep like a thief in the night. Around this time, I started to take more proactive action to help maintain proper sleep.

Along with regular trips to the gym, I made the following changes in 2019:

  • Started to practice mindful meditation
  • Reduced temperature in the room to be cooler than usual in the winter
  • Added ear plugs to remove outside noise disturbances
  • Added a weighted blanket

The above sleep hygiene changes worked well enough, I would get 6 – 7 hours of sleep, and wake up feeling energized and ready for the day, at this time, I was working from an office 3-4 days a week, so would wake up around 730 am to get to the office for 830 am

Feb 2020

At this point, Canada was in full pandemic mania mode, and we’ve been ever since with really no end in sight. The constant fear-driven messaging from our Canadian federal / provincial / media agencies has certainly played it’s part in leading to the situation we’ve got as of 2022 , where 30% of Canadians are reporting “burn out” .

I’m not burnt out, but let me tell you, I’m a bit CRISPY around the edges, boy 🤣. My sleep has never been worse as of Jan 2022, but it started to go to 💩 around Feb 2020

2021

2021 was essentially 2020 v2. Essentially, everything I was doing at this point to improve my sleep, was actually just to “break even”. This is the year where I really started to notice the feeling of waking up, and feeling like I’d hardly slept @ all

Feeling *mostly* undeterred and highly organized with my beloved MS OneNote tracking system, I made the following changes in 2021:

  • Ensuring any bathroom prep is done 1 hour before bed: Brushing/flossing/washing my face/etc
  • No alcohol / caffeine/ sugary drinks at night – M-Thursday
  • Prescription sleeping pills from the doc
  • Changes to my reading / TV / movie consumption before bed. For example, comedies or cartoons only for TV, and nothing “heavy’ in terms of news content that might get me fired up
  • Strict M-Thursday bedtime of 12 am. In reality, this meant it felt natural to go to bed around 12 am on the weekend days of Fri-Sat-Sun as well. “Late” for me now is about 1 am
  • Stopped reading the daily Covid-19 case counts for Montreal/Canada/The World/Mars/etc

I’ve listed the various changes over the past 3 years , as I believe them all worthwhile habits to follow. Certainly other less “anxiety prone” people might adopt even just a few that I’ve listed, and note an improvement in the quality of their sleep

Dec 2021

I was on a 10 day break from work, and noted that my sleep was WORSE than ever? But why? I wasn’t working, just seeing the odd friend, talking on the phone, relaxing and reading comics. I was going to bed at 12 am, waking up at 8 am, but felt like I’d only slept a few hours. It was time to seek out professional help.

I’m from Manitoba, but currently reside in Quebec, which has a famously terrible health care system. It runs in a semi-broken state at the best of times, C19 has made things worse. My doctor resigned in 2021, and I’ve been on a waiting list to get a new doc ever since, I have no idea when I’ll get a new doctor, so have been using walk-in clinics in the Montreal suburb of Verdun where I reside. Walk-in clinics bring their own frustration, as each time I’ve visited the clinic down the street from me, I get a new doc, who asks me the same questions.

Regardless, I visited my assigned random doc on Dec 31, 2021 (happy new year!) and got a referral to a private sleep clinic. I got an “at home sleep test” to be evaluated for sleep apnea ; which is where you stop breathing for very short periods of time while asleep, and wake up.

The at home sleep test involves hooking up a monitor to your index finger and a set of small probes into your nostrils. I found the first test super uncomfortable, the second wasn’t as bad. It’s not much fun having all this shit hooked up, it is a means to an end.

I’ve got a remote video consultation with a sleep specialist at the clinic in February. I would then hope to be given a recommended solution. Generally, this is a CPAP / BPAP machine such as the below

BiPAP vs CPAP Machine: What Is The Difference Between CPAP and BiPAP  Machines? | CPAP.com

Other more comfortable options include custom mouth pieces that move the position of the jaw foreword slightly to increase airflow, or custom nose pieces, such as the below:

Introducing the world's first hoseless, cordless, maskless, battery-powered  CPAP device. — Airing

Now, I breath through my nose during sleep, so the CPAP/BPAP option or custom nose piece would probably be more ideal. It will be up to the sleep specialist at the clinic to guide me in early Feb 2022. Another option, the most extreme , is sinus surgery 👃 😱

March 2022

I finally received my CPAP machine prescription, it’s now just a matter of waiting to get a machine, but, the hard part is over! I will update this blog when I get my CPAP machine in the coming weeks / months, due to C19 bullshit, there are shortages

And to all, a good night

I will update the blog post as I progress through treatment and recovery. If your own sleep sucks, I hope you can take something away from my experience and steps to improve my sleep

To close, here are some pix of my little beast, who has ZzZzZero issues sleeping 😄

Sleep well, my friends!

Owen and coffee bean ♥

Old school troubleshooting based on OSI model

Intro

In college, the OSI model was drilled into me, as one of the IT admin courses I was taking included the full Cisco Certified Network Admin track. While I never did take up a job doing pure network stuff, what I learned about the OSI model is still useful to this day

We have paid tools such as the excellent ControlUP / eG Innovations to aid us with troubleshooting virtualized environmental issues , it’s always good to triage your performance / outage with some good ‘ol fashioned troubleshooting that doesn’t need to make use of paid tools. If you’re a consultant like me, you aren’t always going to be able to install/configure paid tools for your clients, or perhaps the client is VERY small, 1 or 2 virtualization hosts, so , it doesn’t make sense to setup these tools either.

This blog post will review the OSI model, and provided relevant examples for each layer go over the basics, with examples that are applicable to virtualized environments

What is the OSI model?

Let’s start with the below image from imperva

As above, I learned of the OSI model in college, so, I had test/exam/quiz questions on it, to remember the 7 layers of the cake, there are a few acronyms you can use. I chose the following

ALL
PEOPLE
SEEM
TO
NEED
DATA
PROCESSING

I’ve been working in IT professionally (whatever that means to you) for 21 years. I believe that’s a pretty good chunk of time to have formed an efficient method for troubleshooting networked computers. From experience, the most complicated (and therefore lengthy ) issues you DON’T want to start your troubleshooting steps with are layers 4-7. The worst, is application troubleshooting. Why? Way too many variables at play: settings, interoperability with the underlying operating system / drivers/ runtimes. Trust me, you don’t want to start you path here. Instead, I always start with the ground floor of your network device /computer, the physical layer

1 – Physical layer

Here, you can review the basics at the physical layer that might be the culprit for your performance issues. Our first bullet list will mostly cover those physical / hardware settings that would be relevant to check on your virtualization hypervisor

  • What is the expected link speed of the network card? I had an HP desktop unit I’d re-purposed for use with ESXi last year, I was noting issues with the mgmt. interface going down, the built-in NIC was supposed to run at 1 Gbe, but refused to go beyond 10mbit/sec, yikes! I ended up disabling it, and cutting over to a dual-head 1GB intel NIC
  • Modern SSDs using the NVM protocol and connect using M.2 or U.2 on the motherboard. However, there are still plenty of regular SATA-based drives that connect using traditional SATA cables, of which there are TWO cable types. 3 Gbit/ 6Gbit /sec, check that you’ve got the right cable connected to ensure full bandwidth
  • Not all PCI express slots are created equally. In some systems, you’ll only get one of your x16 PCI express lane running at full speed. So, if you have a fancy 10 Gbe NIC in your virtualization hosts like me, but also have dual or quad head 1 Gbit NICs, place the slower NICs in the down-shifted X4 PCI slot, and the 10 Gbe nic in the x16
  • RAM: Ensure your virtualization host isn’t mixing / matching RAM speeds, else, you get the slowest speeds
  • ESXi by default will install and set itself to “Balanced” power mode, this means your CPU will max out at the 80% mark, unless you’re working in a “green” or LEED certified datacenter, you’ll want to set it “MAXIMUM PERFORMANCE” as such:

2 – Data Link

A few simple items to check at the data link layer include the following:

  • VLANs set on your network
  • A manually coded MAC address set on your network card, it’s possible you or someone else set one as part of a PVS implementation
  • The data link layer is also where error control is handled, so, if you’re using PVS, you’ll want to inspect your network switches for any corresponding error messages

3 – Network

As we move up the layers , our troubleshooting steps will be become more complex and involved, this is where you’ll more than likely need to start using built-in CLI tools for your OS: *nix or windows, here’s a good starting list, I’ve ordered them based on what to start with:

Ping from a regular command line or test-netconnection via powershell, you can also test individual ports via test-netconnection -computername XXYY -port 123

-Netstat -ano: to show a mass list of port bindings to IP addresses , very handy if your troubleshooting issues with service ports on on-prem CVAD controllers where you suspect a binding isn’t working as expected

-Tracert: I’ve used this countless times to ID issues with default gateways, that is, where a virtual device isn’t getting the correct default gateway. You’ll see this in a tracert, where the routing process will fail after hop 1 or 2, in a large enterprise, you could have 5-6 hops to get all the way from source to destination

-Nslookup: useful for testing basic name resolution , Citrix / VMware / Nutanix require static DNS entries for some functionality , if you suspect an entry is incorrect / missing, and need to update the DNS record in MS active directory DNS or unix BIND, start with an nslookup first

-WireShark: This tool is extremely powerful, it’s actually useful for troubleshooting problems from layers 2 all the way to 7! However, it can certainly be used for the NETWORK LAYER to start

4 – Transport

Here is where TCP/IP based communication lives. There are some quick wins to be had here in terms of troubleshooting performance . The first relates to a cursed feature set that Microsoft developed for use with Windows Server 2003, and it’s been a thorn in many an IT person’s side ever since! It’s called Scalable Networking Pack – Wikipedia, the Microsoft article is here

For years, blogs/tech/articles/or even MS support engineers would recommend disabling RSS, receive side scaling. When modern PC’s only had a single vCPU or pCPU assigned, this made sense to have it disabled, but when was the last time you worked on a system with a single core? However, the RSS setting you might have put in place as best practice years ago via script/GPO/GPP/BAT/3rd party tool doesn’t actually get reset when you update VMware tools, as such, you should evaluate ALL of the following settings on the network card properties of your physical/virtual computer. Each should be tested and enabled where possible to ensure optimal performance

4k jumbo frames setting: Here you will want to use a proper method to ID the maximum MTU window size you can set on your physical network cards, switches and virtual network cards. You can do this via VMKPING on ESXi, and ping -L on windows

TCP chimney offload

Receive side scaling

5 – Session

To be honest, you will probably never do any pure troubleshooting at the session layer. Instead, you will troubleshoot layer 7 application issues that can cause Citrix “session” disconnects due to presentation layer (TLS security) or network issues that can occur along the way through layers 1-4. The session layer would be considered a “legacy” layer as we transition to connectionless based protocols such as UDP. However, TCP is a valid fall back for UDP sessions, so, it’s important to know the basics of session based troubleshooting. For troubleshooting Citrix-based environments using receiver / workspace app, refer to the previous layer troubleshooting notes and read on for the final two layers!

6 – Presentation layer

This is where encryption related issues can occur. All the tech vendors will remove support for older versions of TLS as time goes on, ensure that you’ve got the right TLS versions configured on your networking devices and end-points to align with industry best practices. Protip: A certificate that’s expired on your ADC might not initially appear as a “your cert has expired” on workspace app , use a daily health check, or set a calendar reminder when your ADC / StoreFront certs are coming up on renewal

7 – Application

As I mentioned in the first part of this blog post, this is the least fun / most complicated layer to troubleshoot, hopefully, the troubleshooting you’ve done with layers 1-6 have removed the need to keep the party going @ layer 7. If not, let’s get into it!

Anti-virus exclusions:

A classic way to lose CPU cycles is scanning things you don’t want scanned! 

For Citrix, check the following KB:
https://docs.citrix.com/en-us/tech-zone/build/tech-papers/antivirus-best-practices.html#virtual-apps-and-desktops

For VMware Horizon environments, check the followng:
https://kb.vmware.com/s/article/2082045

Windows updates:
As per Microsoft’s insane release schedule, windows is never really the same for long. Monthly patches will fix some issues, close security holes, and open new security holes and create new issues. Identifying if the most recent windows update patch you applied comes down to good testing. If you’re running a virtualized environment , silo off a section of VMs to deploy the latest windows updates, and provide them to dedicate test users, as well, ensure you’ve got a formal testing process that can be filled out and tracked each month

Runtimes:
Vendors such as Citrix/VMware/Nutanix/etc will include updates to pre-req programing language runtimes required with their apps, but they often don’t install the latest versions. To determine if you’re most recently installed vendor binary hasn’t created a new issue with pre-existing installed runtimes, reboot the machine, and open eventviewer and check for any new administrative events related to missing/replaced/corrupted runtimes.

Vendor updates:
There are a few schools of thought on how best to deploy vendor updates: One is evergreen, where you are always on the latest version of the application. The other is follow a semi-annual , LTSR or quarterly approach. I’ve found the best approach, is a hybrid based on what makes sense, and will reduce the impact should something go wrong, or you have an environment where proper UAT is not possible, and new environment changes can only be validated in production

Here’s what I use for most of my clients that are on a windows 201x / Win 10 based virtual desktop:

To cover the need to patch for critical security vulnerabilities, the following elements can be patched monthly

  • The windows OS
  • Stick with an patch the “current release” channel of MS office

Also to cover the need to patch for security issues, you’ll want to review the release notes on Google Chrome / Edge Chromium and patch monthly. That being said, both vendors are tightening the noose with legacy browser settings, you may find your clients internal web apps stop working as expected if you update to the latest version of either of the above browsers, always test, or if this is not possible, send out a comm to the IT/business contact with a link to the release note

Middle-ware apps such as WEM/FSLogix/CVAD/Citrix Optimizer should be updated as required when there are features you want, I’ve had to back out of plenty of changes for multiple clients where I blindly installed the latest version of each . FSLogix QA has certainly suffered since the MS takeover a few years back, the last thing you want is to introduce a prod issue with your profile management solution “just to be on the latest” version. Again, unless there’s a critical feature / fix you want in your middleware app, and you’re still supported should you need to escalate to the vendor, leave it alone

General tips / Wrap-up

The meat of the above blog post advocates for troubleshooting based on a basic understanding of the OSI model, that being said, it’s not always practical to through all the layers, from experience, a huge chunk of the time we spend on “Why is Citrix slow” comes down to the application level. The ace in the hole for me in troubleshooting application / system issues in virtualized Citrix environment, is basic Powershell loops, you can make these more advanced, and I certainly recommend my own github, or Sasha Tomet’s for daily environment health checks if you want to go more advanced, but let’s start with a basic example:

User’s are intermittently complaining they can’t connect to a non-persistent Citrix VDI desktop

Obviously, your first steps should be review Citrix studio / director for anything obvious, however, this route may not provide you with root cause on your issue

One of my favorite uses for Powershell, is to mass collect event IDs, the below code reads all VDs from a delivery group, filters out offline assets, and checks for specific win event IDs, you can edit the event ID to suit your own needs

$ReportTime = (Get-Date).ToString('MM-dd-yyyy-hhmm-tt')
$ReportsPath = "\\UNC\PATH\TOFOLDERTOSAVEXLS"

Add-PSSnapin Citrix*

IF (Get-PSsnapin -name  Citrix.Sdk.Proxy.V1) {

    Get-XDAuthentication -CustomerId CtxCloudID

}

Else {

    Write-warning "Please install Citrix Powershell SDK to access Citrix Cloud, path below"
    write-warning "The script will now exit"
    EXIT
}

If (-not(Get-Module -ListAvailable ImportExcel)) {

    write-host "Installing ImportCSV module and pre-reqs"
    Install-Module ImportExcel

}


### CTX Cloud Win 10 VDI Prod Leo/Molson
write-host "Collecting asset info from Citrix cloud" -ForegroundColor Cyan

$Assets = Get-BrokerMachine -DesktopGroupName "Your DG NAME HERE" -MaxRecordCount 2000 | Where {$_.PowerState -eq "On"} | Select MachineName, @{E={$_.AssociatedUserFullNames};Label='AssociatedUserFullNames'}, @{E={$_.AssociatedUserNames};Label='AssociatedUserNames'} , SessionClientName, SessionClientVersion, SessionStartTime, DesktopGroupName

## #Start!

$AssetsTotal = $Assets | Measure | Select-Object -ExpandProperty Count
$AssetsLeft = $AssetsTotal

$OutArray = @()

ForEach ($i in $Assets) {

    $VM = $i.MachineName.Split("\")[1]
    $User = $i.AssociatedUserFullNames
    $AssociatedUserFullNames = $i.AssociatedUserFullNames
    $AssociatedUserNames = $i.AssociatedUserNames
    $SessionClientName = $i.SessionClientName
    $SessionClientVersion = $i.SessionClientVersion
    $SessionStartTime = $i.SessionStartTime
    $DeliveryGroup = $i.DesktopGroupName
    
    write-host "Checking $VM"

    Write-Host "Checking $VM now" -ForegroundColor green
    write-host  "$AssetsLeft remaining to process.." -ForegroundColor cyan

    if (Test-Connection -ComputerName $VM -Count 1 -ErrorAction SilentlyContinue) {

        $Ping = "Online"
        
        write-host "$VM is online" -ForegroundColor Green
        
        $ID = Get-WinEvent -ComputerName $VM -FilterHashtable @{LogName='System' ; 'ID'='1058'} -ErrorAction SilentlyContinue | Select -last 1
        
        IF ($ID) {
            
            write-warning "Sys event 1058 found on $VM"
            
            $IDXML = [xml]$ID.ToXML()
            $DC = $IDXML.event.EventData.Data[6].'#text'

            $MSG = $ID  | Select -first 1 | Select-object -ExpandProperty Message
            $Time = $ID | Select -first 1 | Select-object -ExpandProperty TimeCreated
        }
        
         Else  {

            $MSG = "No event ID 1007 found"
            $Time = "N/A"
            $DC = "N/A"
        }       

    }
    
    Else {
        
        write-host "$VM is offline" -ForegroundColor yellow
        $Ping = "Offline"
        $ID = "N/A"
        $MSG = "N/A"
        $DC = "N/A"
        $Time = "N/A"

    }

    $OutArray += New-Object PSObject -property @{

    VM = $VM
    Ping = $Ping
    MSG = $MSG
    Time = $Time
    DC = $DC
    AssociatedUserFullNames = $AssociatedUserFullNames
    AssociatedUserNames = $AssociatedUserNames
    SessionClientName = $SessionClientName
    SessionClientVersion = $SessionClientVersion
    SessionStartTime = $SessionStartTime
    DeliveryGroup = $DeliveryGroup
    }

    $AssetsLeft --

} #ForEach asset

$OutArray | Select VM, Ping, Time, DC, MSG,  @{E={$_.AssociatedUserFullNames};Label='AssociatedUserFullName'}, @{E={$_.AssociatedUserNames};Label='AssociatedUserName'},`
SessionClientName,SessionClientVersion,SessionStartTime, DeliveryGroup | Export-Excel -Path "$ReportsPath\EventID-1058-$ReportTime.xlsx" -AutoFilter -Autosize -FreezeTopRow -BoldTopRow

$OutArray | Where {$_.MSG -like "The Processing*"} | Select VM, Ping, Time, DC, MSG, DeliveryGroup | ogv 
Create your website with WordPress.com
Get started