However, PSADT is a third party module, to install it, you need public internet access. Bad luck, maybe? it’s come up for me many times over the years, i’ve been working in an environment that blocks such modules, as such, there’s another means, built right into most versions of windows
From July 2021 to August 2022, I went on an epic # of test drives , 30 in total. I ended up purchasing a lovely Subaru WRX 2011 with a manual transmission. You can read more about my criteria / experience in my original blog post HERE
Some pix of the Subaru WRX . I added a nice Android touchscreen head-unit, and some ambient lights, else it was left stock
After driving the Subaru around for the summer into the winter of 2022, I realized that manual transmission was no longer for me. From a teenager into adult hood, my entire driving experience has been manual. I learned on manual, sat my driving test on manual, and owned 2 manual cars (two Honda Civics). At the time, manuals were more reliable and more fuel efficient, that’s just not the case in 2022. In fact, we probably crossed that threshold years ago, I just wasn’t paying attention. Automatic cars with paddle shifters / sport shifters are a lot of fun, and to be honest, driving around Montreal one way streets and manually shifting through 3 gears to get from one stop sign to the next feels dated. It’s like vinyl records, I ain’t dropping the needle or flipping the record to listen to my favorite Steely Dan albums. Spotify / Sonos all the way, son
Towards the end of my test drive process, I discovered something interesting. The cars I really liked driving had a “magic ratio”. As per my previous blog posts, I’m a big fan of MS Excel – even using one to vote in our ridiculously unnecessary stupid Canadian election in Sept 2021. Of the 20 individual models of cars I drove, I found the ones I liked the most had were light weight, with a horse power of 250 to 300. When I updated my spreadsheet to divide the weight of the car by the HP, I came up with a magic “thrust to weight ratio”. My final test drive in May of 2022 was a BMW 328i, which I found very disappointing. Why? THURST TO WEIGHT. The BWM 328i is 230 HP but 3600 LBs in weight, which isn’t very sexy. After finishing the test drive, and went home and updated my spread sheet with the new “thrust to weight ratio” , then reviewed against the cars I liked vs the ones I didn’t. The results were interesting, and certainly aligned with more subjective notes I made after each test drive. Prior to adding column “C”, I had just highlighted in green / blue the car specs I found important, as well as a note on the test drive. As you can, 5 of the 14 items I tagged had “thrust to weight” value less than 15. The ones I didn’t like so much had a value of 15 or more.
Over-all, the Lexus IS 350 F was my favorite car of the ones I drive, but I couldn’t justify the cost at the time, it occupies position 2 in the sheet. I bought the car in position 3, the Subaru WRX 2011, light weight with 265 HP, lots of fun!
Which brings me to a definitive week in Gulfport, Florida Dec 2022. I flew with my beloved beast to visit my parents. I don’t like Canadian winters, so it was nice to get away for Christmas at their Florida home
Whenever I travel to the states, I use Turo.com to book nice cars. Turo is like Airbnb for cars. You can rent all kinds of neat stuff. April 2022, in Arizona it was a Tesla-killer Hyundai Ionic 5 electric. October 2022 in Kentucky, it was a nice Honda Accord Hybrid. Dec 2022 in Gulfport, I went all out, booking a BMW X1 2018 and a Lexus IS 350 F. Both cars I had driven in 2021 as part of my test drives
2018 BWM X1 328i
2014 Lexus IS 350 F Sport, why 3 pix against the single pic of the BMW X1? The Lexus was 3 times sexier, thus the pic of me looking sexy against the murder red interior. I’ve since shaved the winter beard, as it was making me look oldddd
I chose these two cars, as they were two of my favorite test drives from 2021 to 2022, and I’m looking to sell / replace the Subaru WRX 2011 hatchback this winter. And so, I’ve started to scour the local buy/sell via FB MarketPlace / Autotrader/ Kijiji Auto for a 2015 BMW X3 (similar to the X1, a bit bigger, but with a 300 HP engine), or a 2014-2015 Lexus IS 350 F. I’ve chosen these years based on costs, each can be had on the used CDN market for in/around $23k at the time of this posting, based on selling my Subaru WRX 2011 for about $9-10k, the price difference is reasonable. Each would come without warranty, so the BMW is a bit riskier for yearly repairs costs
I’ll update this blog post with the end result, it should be one of the above , as the test drives were basically completed in Florida last month, but, but it’s TBD
UPDATE FOR JAN 18, 2023
A winner has been declared, and that winner is the mighty 3rd gen Lexus IS 350 F Sport
On Wed Jan 18, 2023, I traded in my aging Subaru WRX 2011 (at a CDN $3k loss against what I paid for in August of 2022) for a fully-loaded 2017 Lexus IS 350 F Sport
To be honest, the Lexus IS is the car I should have bought in 2021 when I was doing all those god-damn test drives. I had test driven 2 that year. I was stubborn, having only owned manual transmission cars prior, I thought going with manual was the right choice again, WRONG. All the of advantages / fun factors for manual transmission are gone. Hindsight is always 20/20. Here we are in 2023, and I’ve got my dream car. My second car was an Acura 1.7 EL which I nick-named ‘Silver bird’, as such, the Lexus IS will be ‘silver bird v2
Here is me with the beard trimmed as promised against the back drop of the murder-red leather interior which I 💓
Let’s go buy some booze to celebrate
Side-by-side against the Subaru WRX 2011 which I used as trade-in for silver bird v2
Hello friends, has this ever happened to you, you spill red wine on your white couch and don’t know what to do?
Kidding aside, surely you’ve run into this issue before? You note the C drive on your windows VM is running low, extend the disk on your hypervisor, on opening diskmgmt.msc, you see the following:
As a result, you cannot extend your C drive to include the extra space you added in the previous step. This is due to VM’s obeying the rules of “contiguous blocks” that regular physical drives do. DANG
Ten seconds later on Google, you find something like THIS I’ve used similar methods in the past, but it’s a long / manual process with room for error with commands you’ve probably not used since Windows XP. Facing the same issue on 3 of my home lab VMs today, I decided to try out a tool that I found a few years back for re-formatting SSDS to install windows, where I had previously installed ESXi. It’s called GPARTED it’s free / fast / ez to use
It’s time for my favorite topic, home planning and upgrading 😁
This article will cover changes I’ve made or have planned based on the move to my house. You can read through in it’s entirety or skip to the respective sections for compute hosts, case/rack, networking, storage, hypervisor choices, desk workspace area, UPS, and monitoring
I recently bought a house, see my post on that topic here
With the new house, I decided to finally buy a server rack to contain all my home lab / internet gear. After some careful measuring, I chose a 15U from SysRacks (amazon.ca link). Little did I know, the company is from Montreal! It was delivered first week of June 2022 in a truck with the Sysracks logo! NICE
If you do go with SysRacks, be warned, what others have said in the Amazon reviews section is true, the instructions aren’t great, especially the the installation of the 19 rack mount ears. I struggled with this piece, having not done a rack mount server install in many years. once done, I placed the unit into my new home office and it fits perfectly. With the rack installed, I started to look into what other 19″ profile items I could install into the new rack
Within the new rack will be a Tripp Lite SMART1500LCD 8 port rack mount UPS (Amazon link)
I’ve had UPS units connected to my home computer equipment for almost 10 years, i chose the above unit to consolidate to just one unit, replacing my existing pair of older APC brand UPS units
Pic 1 – This was the temp setup I had from June to Sept 5, all the my lab gear was placed on top of the new server rack while I wait to finalize my purchases for new rack mount UPS / Storage units
Pic 2 is after I placed all my gear inside the case, except for my cable modem (for the rare time I need to power cycle it)
Pic 3 is with the locking front door installed
Update for Sept 13, 2022, I removed the Tripp Lite Smart1400 LCD 8 port UPS, too loud/hot! I wasn’t able to get the rack cooler than 29 degrees Celsius with it installed. I’ve gone back to my original 4 port APC unit, and will more than likely sell the Trip Lite unit or try to return to the manufacturer
Workspace / Monitor setup
I’ve had a two monitor setup since about 2015, it’s worked really well. However, as time has gone on, and slack/teams have mostly replaced outlook, I feel a need a dedicated third monitor just for communications. I’m 100% WFH in my job, so am constantly monitoring for alerts/emails/etc, there are no taps on the shoulders in my work day to advise me that I’m needed for something urgent. Years ago, I had a boss that was all about “inbox zero” and 100% replies on client requests, it’s been years since I’ve worked in an environment with such expectations, but the habit has stuck, I don’t think I’ve missed a replying to an important email in about 10 years…
So, my original plan was to try out a 3 monitor via desk attached arm setup as such
Here’s what it looked like after assembly
However, after trying the setup of the Samsung 27 ” monitor on top, with the two Dell 23″ monitors below, I decided it was NOT for me
So, a few Sundays ago in Sept 2022, and one Redbull later, I switched to the following:
The 2x Dell 23″ monitors are on top, the larger Samsung 27″ is on the bottom. I re-used an iPad stand I wasn’t using to mount my webcam. on an unrelated note, since becoming a home owner, I LOVE PLANTS, OMG. SO MANY. The Yuka on the left was outside till Sept 26, 2022, but it’s now getting cold in Montreal, Canada where I live, so it was time to bring it inside
It should be stated, via Reddit/friends, I’ve researched ultra-wide monitors off/on for the past year, but none are the right fit for my workflow at this time. If you’ve got a great working Ultrawide setup and are doing EUC engineering / design / architect work like me, post a pic in the comments, and include what gear you used! I’m not fully sold on the 3 monitor setup, but will keep using it for the rest of 2022
I’m professionally certified on Citrix / VMware, and do some Nutanix integration work with Citrix. I regularly do VMware project work to stand-up new vSan implementations and help customers migrate from vSphere 6.7 to 7 as the Oct 15, 2022 EOL dates approach. I don’t currently run Nutanix on any of my home hardware. My choice to use vSphere is based on job requirements, and my love of their VMUG Advantage program. For $200 USD per year, you can get full access to the entire VMware suite. Nutanix only provides older versions of Prism/AHV via their community edition program. The CE version is often quite behind the GA versions available to customers, so I’ve had the scenario where I wasn’t able to get newer windows builds to boot. Until they rectify this, I’ll stick with VMware
The on-going debate ; AMD vs Intel
I ran my personal desktop on an HP AMD 5600G based system for about a month in from Aug to Sept 2022. Worked fine for two monitors and Windows 11. However, with the exact model I chose from HP , I wasn’t able to drive 3 monitors. So, I switched to an HP EliteDesk G4 Core i7 8700. Before selling the HP AMD unit, I did test ESXI on it, the results weren’t good. I had to disable “secure boot” to get around the “ESXi pink screen of death” many others have reported trying to use AMD home hardware with ESXi. As well, the built-in NIC wasn’t detected, as the HP AMD desktop I bought used from AMD only had one full speed PCI express port, my upgrade path was limited
I’m not alone. There are posts from 2017 all the way to 2020 from home lab fans attempting to use commodity AMD mobo/Ryzen CPUs notingm ESXi 6.7 / 7.x “pink screen of death”. Some report running months without incident, however, to date, I’ve not had either of my HP EliteDesk 800 G3 SFF (Core i5 6500) units running ESXi 6.7 / 7.0.x crash in about 3 years of 24/7 use. As the years have gone by since I finished college in 2005, my “home lab” is no longer used to practice implementations for clients / learn / research, I host plex for me / friends, run active directory, have file servers for archiving and more. If/when any of these servers / services go down, I treat it like prod, and get it fixed as soon as possible. As such, having any of my ESXi hosts go down randomly due to AMD / ESXi issues isn’t going to work for me. I can’t explain why AMD EPYC processors aren’t impacted by the same issues as the Ryzen 3/5/7 counter-parts, maybe it comes down to lack of QA from VMWare on AMD desktop parts? If you have any theories, or have a working AMD mobo/CPU combo, let me know! Also, post your working hardware config to this EPIC thread on William Lab’s blog, I submitted my experience with the HP AMD Pavilion 5600G
The replacement for my 6th gen Intel based HP EliteDesk G3s will be the HP EliteDesk 800 G4 model, which has an Intel Core i7-8700 (6 cores / 12 threads) chip. To date, I’ve not read of similar PSOD issues on this particular model. This model is easily found on eBay for about $400 CDN per box
I’ll re-use my existing Samsung 970 EVO NVM / trad SSD for storage
In 2019, I bought the Mikrotik CRS309-1G-8S+IN Cloud Router Switch 8xSFP+ switch. Mikrotik is a small Latvian-based networking manufacturer who make robust / reliable well priced gear. The unit has been rock solid, I see no reason to replace it at this point, however, assisting a co-worker with some home lab choices recently, he found the a suitable unit QNAP QSW-M408S 10GbE It’s well reviewed/priced on Amazon
For 10 GBe network cards in your hosts, I like older Intel X520-DA2 model cards. When I was still buying them in 2019, they could be found on eBay for about $75-100, but YMMV as of 2020. These cards aren’t fancy, they don’t support RDMA, for instance, however, I’ve found them reliable and fast. Synthetic benchmarks showed close to the expected line speed , around 9000 Mbit/sec. Real world usage was about 7200 Mbit /sec. The nice thing about this card, you can actually find it on the VMware HCL, good luck finding your other components on there 😜
For the longest time, here’s been how I’ve provided large-file / long-term storage @ home
Step 1: Buy/install a large 3.5 traditional hard drive into a single physical server, for the past 5 years, an ESXI host
Step 2: 3-4 years later, notice I’m running out of space
Step 4: But new 3.5 HDD that’s at least 25% larger in size than the one it’s replacing
Step 5: Migrate data from old to new drive, and yes, it takes longer to copy over all my data each time
Step 6: Think about a better way, look at current available NAS units from Synology/QNAP, curse at the price and lack of 10 GBe + M.2 NVM support
Step 7: Evaluate TrueNAS (previously FreeNAS) get annoyed with administrative over-head and stop using it after a few days
Repeat steps 1-7 till 💀
However, it’s 2022, it’s time to break the cycle
As I’ve got a 19 inch server rack now, I’m looking into a 19″ rack-mountable QNAP TS-432PXU-2G-US NAS unit. It’s got 3.5 drive support only, but 4 bays, and has built-in 10 GBe support. With a 4-bay unit, I can install one 3.5 drive today, and grow my storage needs as time goes on via RAID 5 or similar via this process. I can look at adding M.2 support for NVM drives via a PCI express add-in card later. However, my plan is re-enable vSan on my home lab, which would use the SSD/NVM drives already in my HP ESXi hosts. I’ve used vSan on/off for years, but as of Aug 30, 2022, I’ve got it disabled as I had re-purposed my third ESXi host for use with Nutanix CE, and didn’t want to use have vSan running as a 2-node cluster with an external vSan witness appliance
Monitoring / cooling
I monitor my physical / virtual assets by a script I maintain on GitHub, here
I don’t do kW power monitoring for now, but might do now that I’m settled into my house. If you have any suggestions for software/hardware to do so, let me know in the comments
I’ve installed a basic LCD screen that shows temperature / humidity inside my Sysracks server cage. I’m averaging about 23 degrees Celsius / 73.4 Farenheit with two low CFM 120 MM fans. The fan that came from Sysracks sounded like a jet engine, and could not be throttled down via a speed control swtich, so I replaced two 120 MM adjustable speed fans from Amazon
As with any purchase, do your research as much as possible, finding someone who’s got the exact same unit you want to buy, who’s written a formal review on their blog / YouTube Video / Reddit etc is always a good idea
Share what you have in the comments and happy hunting 😀
Odd as it sounds, as of May 2022, I’d never been a property owner (house/condo). A trip to Arizona inspired me to become a home owner. Why? Long-term, I’d like to do as my parents do, and become a snowbird. They travel to Florida each winter from Winnipeg, MB, Canada where I grow up. Montreal, Quebec, where I reside is not as cold as Winnipeg, but really, everywhere in Canada has the “winter tax”, and I’d like to stop paying it ❄️🌨️
My criteria for the house was simple, but was at odds with the condo type dwelling common for most folks from Montreal. However, as I grew up in Winnipeg, MB, I’m used to BUNGALOWS. So, I wanted an un-attached house, small, close to 1200 square feet, front and back yard w patio, and room for one car (which as of this posting, I still don’t own, but soon will)
On my return from Arizona, I mentioned my snow-bird plan to my then landlord in Verdun, Montreal, Quebec. She got me in contact with her BF’s friend who’s a realtor, thank you Emilie!
I chatted with my new realtor Mathieu on the phone, gave my criteria, and away we went! To contradict myself, I did entertain the idea of a condo for one a single day of visits, just to keep my mind open to calling an apartment a home (aka, condo), after 3 visits, I backed out of the remaining visit, and advised Mathieu we should focus only on small houses going forward. I visited 2 houses on my own, and one final house with Mathieu, and the 3rd one was the winner! Here were the original realtor pix that sold me on the modern updated interior
I bid / won in mid May , completed my home inspection, and moved in Sat May 27, 2022
I’m a huge fan of Microsoft OneNote, and I’ve used it to organized previous moves. For this move, the list of to-do items became expansive very fast. I set tasks into “week 1” “month 1” , “first six months”. This included minor fixes for electrical issues, getting gutters installed, painting scratches made during moving, etc
I got a chance to consolidate my home lab gear into a single server rack, finally! This is something I wanted to do for years
I’m KINDA color-blind, so, complex shades of blue/green/etc I don’t see so well. The house is mostly black/white/dark brown, so buying items for it have been EZ! I mostly stick to black/white items
I’ve enjoyed simple fix-er-up tasks: replacing a tub spout, caulking my tub, adding a missing step to my patio, leveling out my washer / dryer with anti-vibration feet. Like anything, if you don’t know how to do it, someone has posted it to YouTube! Review & measure twice, and you’ll be good to go
I’m really enjoying the gardening / landscaping / lawncare side of things. I’ve bought a bunch of ‘exotic’ plants that inspired the idea to buy the house when I was in Arizona in April, and keep adding more! I started with 3 provided by my landlord, and now 7 new plants: ZZ, Palm, Yucca
OMG, it’s SO quiet. the interior of the house has all new windows, with excellent sound-proofing. For whatever reason, my hearing appears to be improving with AGE!? I can’t explain it, it’s a bit of a burden, and certainly explains why I’ve moved into a house from an apartment rental where you get sounds from above/below. As well, I can now crank my SONOS wireless system up beyond 15%
Recently, I was helping a co-worker with an issue noted where Win 2019 MCS clones going into a janky state intermittently. I’m a big fan of the windows event viewer for immediate troubleshooting, however, the system was so degraded, we were unable to use compmgmt.msc to remote to the system to open eventvwr.msc
However, we were able to remote in via PowerShell
I’m familiar with PS cmdlets get-winevent, but in this case, we didn’t want to filter through tons of errors, instead, we just wanted the ‘admin events view’, which is a filtered view of only warnings/errors:
How do I get only the admin events via PS? Thanks to this REDDIT post, I now know
My trip to Arizona April 7 to 14th, 2022 was a long-time coming. I’d originally wanted to go in Dec 2019. However, I decided to stay in Montreal for the holidays. 2 months later, covid-19 came in put in a wrench in any travel plans I might have had for the states for all of 2020
2021 came around, and Steve Greenberg’s EUC Master’s retreat event came up! However, it had to be cancelled due to Covid-19 concerns. ⏩ to April 2022, and the event is back on, and my AZ plans are back on track!
I LOVE travelling to the United States for tech related events, for the past 5 years I’ve attended the below events, some paid for by my employer, others I foot the bill. ALL were worth it for the experience of the respective city / meeting people / professional development
Expert to expert, NYC 2017
Citrix Synergy Atlanta, Georgia 2019
IGEL Disrupt Feb 2020, Nashville, Tennessee
EUC Master’s retreat, April 2022, Scottsdale, Arizona
For the Arizona trip, I decided to extend my trip beyond the Friday to Sunday scheduled dates of the conference. I’ve been in dire need of a break from Canada. The covid-19 response here in Canada is about 6-12 months behind the rest of the world. Matched only by our common-wealth brethren in Australia & New Zealand. That is, endless lockdowns, curfews here in Quebec, and hygiene theatre that all provide little to no benefit to dealing with the pandemic C19 endemic
/end covid rant
I was unable to bring my beloved beast Coffee Bean for this trip. So, it was the first time in 2 years, I’ve been without her. About half way through my trip, the fancy Montreal doggy hotel, sent me a pic of her sitting on a chair looking sad AF. I felt like she was staring into my SOUL, compelling me to come home. I immediately thought of the angry cricket fan MEME dude
Did I give in to her guilt-tripping tricks? NO!
The conference was amazing. Even the hotel was top of the pops! DAMN. Here’s some pix from the Saguaro in Scottsdale
Conference-wise, what I really enjoyed, was it wasn’t all NERD STUFF! I’d say about 60% of the content was technical, 40 non. I’ve done quite a few on-line tech talks, but NEVER in person. You may wonder “Owen, how can this be, whenever I talk to you in person/zoom/phone, I can’t get you to STFU, and you’ve got a hot take on EVERYTHING!”. Well , you’d be right, I am very comfortable with speaking in person to one or more people at a time, but NOT WHEN IT’S SCHEDULED. The organizers of the event setup a planning session Friday evening to decide on break-out sessions that would be done on Sat/Sun. We passed a mic around, and voted on the topics we wanted to hear about / speak about. I raised two topics close to my heart: Sleep / IT standards via automation. I was able to speak on both topics over the next 2 days, awesome! It wasn’t so nerve wracking, as I mostly just spoke from memory, I feel sticking to a script / PowerPoint slide would make it more stressful. Kind of like trying to re-play a song on guitar note for note / chord for chord, ya miss something, your audience will know!
The conference ended on Sunday April 10, 2022. I said my good-byes to everyone I could find, and changed over to another hotel about 15 mins away
Now begins the second part of my trip, which had much looser schedule
TBH, I’m not sure why I was so hell bent on renting this particular car. I’m a bit of a contrarian I guess? The more obvious / cheaper choice would have been a Tesla Model 3. I ended up getting the only Ionic 5 in the Phoenix area, and it cost me about $30 more per day vs the equivalent from Tesla. That being said, OMG, what a ride. The interior is STRAIGHT UP STAR TREK TNG. Electric cars essentially have instant torque, so, in sport mode, you put your foot down, and the car FLIES. Navigation, seats, interior, road noise were like nothing I had experienced when doing my exhaustive test drives last year
I used boy boy to get around Scottsdale / Phoenix for sight-seeing and to visit locals I knew in the area. Here in Quebec, I don’t do much hiking, as I find Canadian terrain boring. Prob from living here for so long (lakes/trees/bears/blah). The grass is always greener, and the desert is thrilling! Cactus / scorpions / palm trees!
Here’s some pix from a visit to the desert botanical garden on Sunday April 10, 2022. I want to touch them ALL, but I know I shouldn’t
Monday April 11, 2022!
Time to take blue boy out for some shopping. 2017 onwards, I collect shot glasses wherever I go. Here are some day / night time shots of blue boy
My last day! I hooked up with the event organizers Steve/Beth to check out cave creek, where Steve lives. Such a beautiful area. Enjoy my last set of pix from AZ!
Thursday April 14, 2022
The road home! I prefer afternoon flights, as I struggle to wake up early. That being said, I love all aspects of flying. I recently bought an iPad Pro 12.9 for reading in bed, I brought it along with me for plane TV/movie stuff and copied over some recent tv/movie content. On the way back, I noted that the dimensions of my iPad stand fit perfectly into the sleeve of seat in front of me where the barf bag is! My plane had janky wifi, and no screens in coach, so, I had the ultimate in-flight infotainment system to watch the new HALO tv series (which is great) and “Lock stock and two smoking barrels”, which is not as a good as I remember it!
My flight from Phoenix had a brief lay-over in Toronto, there was some drama with a shortage of TSA / custom agents to process incoming travelers, which created an epic line. I was about half way through said line of about 300 people going through security checks when a beautiful tall black Gal Gadot / Wonder woman-looking lady appeared asking if anyone was connecting to Montreal. She allowed me to skip the line. As a polite Canadian, of course I apologized to everyone I cut.
Thankfully, I was able to board and get home to my beloved dog , who my gracious land lord Emilie had taken care of a few hours after coffee bean was dropped off by the doggy hotel pet taxi.
I will tell you, it had only been a week, but I teared 😭😢😿up when I got home and saw her waiting at the top of the stairs for me. However, I immediately stopped crying when I got close to her, she STANK of other dogs at the doggy hotel! 😆🤣😂😹
BATH TIME FOR BEAN RAY!
My checked luggage filled with Arizona snakes, rocks, and cactus got stuck in Toronto for one day, but who unpacks their bag as soon as they get home from vacation? Not me!
In summary ; this trip was extremely satisfying both professionally and personally. I’m so glad I got to go, and am very excited to return next year for the next EUC Master’s retreat!
For 2022, it was time to start using the packer automation I learned / wrote about in 2022, however, when I first tried to use it with my first 2 clients, it failed, in each case, for a different reason. On a third attempt to use it, I didn’t have access to the client’s environment, so had to talk the client’s on-site staff through filling out the required entries in the related packer XML/JSON/HCL files. By the time we were done filling out verifying the details, 30 mins had elapsed, the avg time to build a new Win 1x, Win Server 20xx image with windows updates is about 25 mins. At this point, I was 0/3 on using packer for my client work. As per this post, I’m often only with a client for 5-10 business days, burning an hour or more troubleshooting / prepping packer where it won’t be used again is not a good use of time. As such, the offline/custom ISO method was born!
This blog post will describe how I made it work, as of April 5, 2022, i’m 3/3 on new deployments with the new method. Sorry, Packer bros
Comparison of packer vs custom offline ISO method
For many years, windows installations have been driven by esoteric config file called ‘autounattend.xml’. For my packer based methods, I included samples, my new offline method also uses these files. however, the key difference is in the work flow:
You download the latest Windows ISO to your desktop: 10 mins
You upload the ISO to the Hypervisor datastore: 10 mins
Packer > Creates VM on Hypervisor ( I only got it working on VMware env): 30 seconds
Packer copies over autounattend.xml / ps1 scripts to newly created shell: 10 seconds
Windows install starts: 10 seconds
Packer waits for WinRM to be enabled on new windows install: 15 mins
Packer shuts down VM: 10 seconds
You boot the VM again to apply any run once / scheduled tasks you set in previous steps: 15 minutes
All of the above is 100% dependent on properly formatted Packer JSON/HCL files, one mistake, even a lowercase/UPPERCASE character and your build is hooped
Let’s look at the method i’m now using for all my builds, which is a custom ISO injected with all the same scripts I was using on my packer-based builds
You download the latest Windows ISO to your desktop: 10 mins
You create a shell on the hypervisor ( tested as of today on VMware/Nutanix): 2 mins
You inject the latest window ISO with the required custom files (hypervisor drivers/XML/PS scripts): 5 mins
You upload the custom ISO to the Hypervisor datastore: 10 mins
You start the shell and boot it to custom ISO: 10 seconds
The rest is completely automated, as well, you don’t have to edit JSON/HCL configs, deal with network / firewall issues / or type-os where you or the client chose the wrong datastore / host / network / etc: 15 mins
Time to completion: Roughly the same, the difference, no 🙉🙊🐵business to deal with
The process / steps to get it done
Download ISO for Win 10 / Server 2019 / Server 2022 on your jump server / desktop, ensure you’ve got a fast link to your Nutanix Prism/VMware vCenter, don’t do this over VPN, son!
Extract the hypervisor_drivers.zip to c:\admin\Offline_ISO\hypervisor_drivers
The structure should look like this when done:
Open AnyBurn, browse to the ISO you downloaded, choose EDIT IMAGE FILE
Upload ISO to hypervisor datastore
Browse to the location where you downloaded the windows ISO to in step 1
Drag / drop over the c:\admin\offline_ISO\scripts & c:\admin\offline_ISO\hypervisor_drivers folders to the root of the ISO in the AnyBurn interface as follows:
Attach the uploaded ISO to your new VM and ensure it’s set to boot to the ISO you uploaded (follow the specific process for ESXi/Nutanix/HyperV/CTX HV)
The rest of the install will proceed in a similarly automated fashion to the packer business. Hypervisor drivers for VMware / Nutanix / Citrix Xen Hypervisor will be installed, some basic post-windows install tasks will be run, including installing any language .cab files you might have populated into the Lang_Pack folder. When this part is done, scheduled tasks to cover starting / monitoring the windows update process will be run
As I said, for most of my projects in 2022, I’ve used this method. In each case, I was able to finish my work early/on-time to allow for additional work (writing docs, troubleshooting environmental issues/etc)
If you’ve read my bio/current job/recent posts, you’ll know I do It consulting for a company in Quebec, Canada. 90% of my projects are 2 to 3 weeks long in scope, if we run out of time, two things will happen:
-The project will be paused -The client will need to buy more hours
Neither of which is great
I’ve been in my current role for 2.5 years, more than half of my projects finish on-time, this is typical for everyone in our team, why? 🐵🐒 business
Defining Monkey business / examples
‘THINGS YOU DIDN’T ACCOUNT FOR / PLAN FORTHAT MESS UP YOUR SCHEDULE”
Here are some recent examples from my work over the past few years:
Client not having DHCP setup anywhere in their environment, which meant my beloved packer automation would not work
Worse, Extra DHCP servers authorized in AD that have bad scope options set that provide incorrect DNS info
Citrix Hypervisor (XenCenter) any $ you saved on using Citrix HV instead of a proper hypervisor like Nutanix / VMware , will certainly be lost to troubleshooting efforts / downtime down the line
Messy AD environments, clients still using FRS , clients who’ve not updated ADMX templates in 10 or more years
In-house apps that require manual efforts to get working on newer OS versions: Win 1x, Win Server 20xx
Lost private keys to TLS certificates
Internal windows update servers running against new OU’s , and no-one has access to manage the servers to stop it from happening
The above is just a sample from what I’ve seen. Bottom line, 🐵monkey business🐵will happen, it’s just a matter of when and how much time you’ll lose trying to stop said monkey’s from jumping on the bed
Mitigating monkey business
For the first year of my job, I mostly followed a ‘ronin / cowboy’ method of delivering projects, I did what I thought was best and called my co-workers for help when stuff went wrong. My work was more chaotic and stressful as a result
I would estimate a lot of my projects ended up looking like this from a pie-chart perspective:
Coming into year 2, me and the other dudes on my team started to do weekly ‘best practices meetings each Friday afternoon. We didn’t just talk, we wrote stuff down into our WIKI, and made it LAW! As such, I started to follow a more pedantic method
Following the OSI layer of troubleshooting, this has been drilled into since taking a Cisco CCNA course in 2004. I never did sit the exam to get the accreditation, but the troubleshooting steps remained with me. Here’s my related blog post on the topic
TLDR; start with the basics when troubleshooting an issue, start at the physical layer and work your way up, troubleshooting layer 7 ‘app issues’ is not where you want to ever be, or to start
Follow an SOP for your project deployment. In my job, we use Perfect WIKI, which is a teams add-in. Within your document(s), define how you deploy images, GPOS, install apps, etc
A tie-in to the above, automate your standard operating procedures (SOP) as much as possible. Hand-installing an OS/apps/windows updates isn’t the way to go anymore, it’s error prone and slow
Review the client’s environment during initial meetings, ID any potential sources for monkey biz, and if possible, have the client to agreed to resolve these issues before you start
Following these 5 points for my last 2 Citrix implementations, I’ve ended up finishing each project EARLY. I’d estimate my time spent on 🐵🐒biz is down from 50% to 30%.
As a result, I’ve got more time for the new items listed in the following pie chart:
1) Reviewing / improving the implementation 2) Providing hand-off docs to the client
Following an SOP that uses automation makes it easier for your co-workers to take over your work if you’re away on vacation/sick leave/etc. They can refer to internal docs / github to ID how you did your stuff, and if your automation uses logging (I hope it does) then ID when you did your work to the hhmmss 🙂
Now! I am just one person. In the comments, let me know how you do your own project work and/or any interesting examples of ‘monkey business’
In this post, I will review how to get it working, and most important of all , how to measure the impact of implementing the feature using numbers. I will use simple / free tools to show the impact: Citrix Connection Quality Indicator and a single PowerShell script from ControlUP contributor Guy Leech
The story. I had a client in March of 2022 who went ‘all-in’ with Citrix Cloud for their control/access layer. As part of the work, they had requested to review the best means to route traffic intelligently based on internal / external users using Citrix Cloud workspace/gateway. To be honest, I slept on both V1 / V2 of ‘Citrix Rendezvous‘, as well as ‘Citrix workload connect’ when they were respectively announced over the past few years. Rendezvous I thought had something to do with Apple ‘bonjour’, so, in my mind, I though it was related to optimizing audio/video steams that ride on HDX? Kind of correct! However, what if you want to reduce the time your internal users spend waiting on the below screen? Will rendezvous help reduce your normal Citrix Cloud session brokering of 5-13 seconds? After much testing and one slack post , I confirmed it will not:
While we work from home a lot more these days, many of us are back in the office, and want to full speed access to the virtual resources we were using from home
The steps to set up the ‘Citrix direct workload connection’ are easy enough. In fact, my man Eric Trond had an enthusiastic post from last year on the topic with a related script which I used for my first crack @ it. In the above post, he described a huge ICA RTT reduction following implementing it. Eric is in Brazil, his Cloud Instance was in the states. He was able to reduce his ICA RTT from 769 MS to 17 MS. MAN! That’s great. His results are provided by the free tool ‘Citrix Connection Quality Indicator’ should be on all your Citrix VDAs
To start, let’s capture some BEFORE examples, in this way, we can measure the before/after impact of enabling the direct workload connection feature.
We will do this in 3 steps:
1st: Follow this ControlUP article to set all the required GPO settings to enable audit logging so that you can run their powerful ‘analyze logon duration’ script. YES, there’s a lot of settings that you might not have set before, NO, don’t skip any of them
2nd: download the Control up analyze logon script to the VDA you’ll be using for testing, for me, i’m keeping this script in a local folder on any golden images I deploy. If you don’t want to do a PVS/MCS update just to have the script on the c:\ drive, you can just copy it over for the duration of your non-persistent session, you are using non-persistent machines for your clients, right? :p
3rd: install Citrix Connection quality indicator on your target VM, as above, you should have this in your golden image already, it’s free / very useful. If not, you can RDP to your target VM with an admin account and install it, you’d then need to configure the related GPO to enable it.
Once CQI is installed, copy over the related ADML / ADMX to your AD from here: C:\Program Files (x86)\Citrix\Connection Quality Indicator\Configuration
….to your PolicyDefinitionsFolder on your AD. Don’t have a central policy store? Create one via the following MS KB:
With the above 3 steps completed, logon to your VDA via Citrix cloud as you normally would. Open CQI to collect the ICA RTT, take a screenshot and save your desktop / Onenote / wiki / etc
I’ve gone though this exercise 3 times now, one on my lab, twice for clients, here’s one of the results BEFORE enabling the ‘Citrix direct workload connection’. It’s not terrible, but considering the connection is being made from a PC from within the same internal network where the datacenter VDAs reside, it’s not great
Next, you will want to capture additional metrics via the ControlUP analyze logon duration script you downloaded in step 2. Once logged on fully to your VDA session , open Powershell as admin, and CD to the directory where you downloaded the analyze_logonduration.ps1 script
Run the script against the session you logged on to VDA via ICA in the following format:
Record your results! Here, we have 14.9 seconds of brokering time, yikes, that’s almost enough time to call your significant other and tell them you’ll be late coming home, because your VDA logon was slow! 😂🤣
With your results saved (don’t forget!), you can follow the Citrix guide for the required steps to enable workspace direct connect on your Citrix Cloud instance
You will need to setup network locations for all the public IP (egress) points for your office(s), for the client I setup last week, there were just two, you could have lots. You’d only want to add the entries for locations where internal office workers are connecting to Citrix resources via Citrix cloud. Once the locations are set, it’s time to test!
For troubleshooting, it’s a good idea to Follow this KB from Citrix to enable workspace app ICA logging for the currently logged in user on the internal network PC you want to test on
With the above logging in place, let’s repeat your VDA logon tests
Logon to Citrix Cloud again, open the ICA logging file you set via the above Citrix KB for enabling workspace app ICA logging. Within the ICA log file, CTRL-F to ‘address=’, you should see the address appear as the name of the VM you’re connecting to. Here’s a snap of a working direct workload ICA connection:
If it’s NOT working, and you took the long route to your internal VDA via Citrix Cloud, you’ll see the familiar STA ticket reference in the ‘address’ field. This is a clear indication of 🐒🐵biznuzz. It happened to me when I was first testing this on my lab, I had set an incorrect public IP / subnet re-check your network location and re-test to resolve monkey business
Now, back to benchmarking. If it works, you should see MUCH less time on this screen
But what is ‘less’? it’s relative, let’s re-run our tests and capture some AFTER metrics
Start with CQI. Oooh boy, what do we have here? ICA RTT @ 2 MS / latency of 1 ms! We are off to a great start!
Next, time to fire up the ControlUP analyze logon duration script again!
What do we have here? From 14.9 seconds to 2.4 seconds from the time you clicked launch to actual session launch. An 83% reduction! SNAP. Pat your self on the back, you’ve made a measurable impact to your internal Citrix Cloud users
That’s more like it. You ever see the bills for the new hosting hardware that runs your VMs? I sometimes do, they aren’t cheap. Giving your internal Citrix Cloud users the fastest possible logon times will make it easier for mgmt to justify ‘going to the cloud’ when the annual renewal comes up