EMC World 2011 : AKA the last 3 months of my life

This post will cover so much of the last few months of my life that I don’t even know where to start. I am sitting here in the Las Vegas airport trying to remember all the details of how I got to this spot. It really started at the beginning of March when I moved to the vSpecialist Technical Enablement team. The TE team is tasked with arming the front-line vSpec with information, tools, messaging, and more to do their jobs more effectively.

Another aspect of the TE team is they usually provide many of the keynote and session demos for the conference and partner events like EMC World or VMworld. Also, the TE guys have been running the EMC Hands On Labs for over a year now at events across the US.

Within a week after joining I had a discussion with my boss, Chris Horn, about the how the existing Hands On Labs (which he pioneered and maintained) could be improved and expanded for EMC World. The goal was to do something much more like the experience the VMworld labs provide. Chris had been talking with the EMC Demo Cloud team about using cloud-hosted labs they could provide to the users in Las Vegas at EMC World. After a few long conversations I committed to write the lab automation that would take a session in the Demo Cloud and automate it straight to a user at a desk across the country. Not a small feat and honestly I realized shortly after that it would require a massive amount of design and work.

Step 1: Design

I had multiple aspects to look at to accomplish what I wanted. My goal was to be as completely hands off as possible. I am a firm believer that Cloud automation should just work to a point that it can be forgotten about. This meant I had to adopt patterns that allowed for failure and self-healed without human intervention.

I also had to consider the fact that a 200 seat lab would need to have multiple people running the workflow and I would have little to no time to train anyone on how to use the system. This meant control interfaces must be simple to understand and bulletproof.

This system would communicate via SOAP to a Cloud automation system in Durham, NC. It would have to not only inventory what types and sessions were available, but gather relevant info on the session (type, IPs, etc) and also revert used sessions for reuse by the next user. This connection had to be robust and self-healing also as it was a critical failure point for the whole system.

For the influx of lab users I had to design a object structure representing the lab user, the lab sessions, and the metadata I wanted to capture for reporting both during and after. This would be critical for post-show stats and live stats I planned on showing during.

I had to write a registration system that would allow the user to choose a lab out of a dynamic set that could be managed on the fly. This would have to be separate from the main system to allow it to work regardless of whether the main *machine* was up or down while allowing handoff to the machine when it was up.

I also had to write a lab console that would work on a Wyse think client and would provide a robust experience. This would have to support some sort of lab *guide* in some format and support RDP for connections into the lab VMs in their session. This lab guide should allow for automatic loading, the ability to change content on the fly, and be easy to read. I also needed to add interactive buttons for asking for help “Raise Hand” and detecting when the user was finished “Finish Lab” to allow the Machine to invoke a reset of the lab session they were using. The RDP connection had to be robust also allowing for slow connections or having to reconnect the user should a network blip occur. The lab console had to inherit the user and lab session information, update it with what occurred during the lab and return a completed lab session detail that could be used for analysis later. This lab console had to auto-detect and auto-login to the Machine allowing for dynamic naming to meet any floor plan and any number of clients (within reason).

I needed console applications for registration, lab assignment, admin functions(controlling the Machine), and more. I also wanted to have display reporting

And on top of off the above I needed to be able to test the entire thing without having 1,000+ people to register or 200 WYSE terminals.

I named this project: Bottle Rocket, and got started on March 21st 2011.

Step 2: The Work

In between March 21st and EMC World 2011 I spent an average of 80 hours a week working on Bottle Rocket. I made a couple videos of my progress for my teammates since I was locked away.

Bottle Rocket Assignment Console from Nicholas Weaver on Vimeo.


Bottle Rocket Registration Console from Nicholas Weaver on Vimeo.


Bottle Rocket Lab Console Walkthrough from Nicholas Weaver on Vimeo.


Also in the time period I built my Vplex Geo Maps/vMotion mashup application. Here is a quick video I made on this. This video was early in the development and you can see the final version when the Chad’s World (Live) it was featured in is released to view.

Vplex Map Demo from Nicholas Weaver on Vimeo.

The videos above are actually alpha versions and I added more stuff to most of them (like ‘Raise Hand’ on the console).

I wrote a ton of testing tools including lab console simulators and registration simulators. I actually simulated 1,000 lab consoles and several thousand registrations live against the Demo Cloud from my home office. This allowed me to stress test each component and be sure things would not fall over on the floor. I also built in a “Sim Mode” into the lab console software where the consoles could actually simulate as if a user was sitting at the desk and work the lab steps. This would allow us to simulate stress on the Wyse terminals on site if needed.

Another piece I wrote that I thought I would not need was a Demo Cloud provisioning tool. I had used this for my testing environment but when we had issues early on (see below) I ended up using this to maintain the quota of needed lab session automatically. You basically told this program what you wanted, and it attempted to provision to your level while adjusting to the rate the Demo Cloud would allow and healing from connection issues or errors. A real slick fire and go to sleep tool.

The last couple weeks before EMC World I spent in the lab with the TE team in RTP, NC. While I had been furiously coding, my team had been making the magic happen by building 19(!) different labs around EMC products. They also had enlisted multiple individuals across EMC to build lab guides for each one. This effort was massive and even more time consuming than my own. The greatest part is that these labs are now able to be used for other events down the road.

I ending up before EMC world with 37 different unique code projects including Bottle Rocket and all its components and the Keynote demos I was involved in. Some of the Keynote stuff was: a crazy fast VM cloning tool based on Virtual Selection, a RDP window matrix of up to 100 real RDP windows borderless on a single screen. And a SQL Performance dashboard that showed real time transactions, MB/s, and depth queue stats. I also built an Isilon VSA (internal only) and new UBER VNX in this period for the lab and my team. I am pretty sure I have never created such a massive about of personal output in my life in such a short time.

Step 3 : The Show

I showed up to EMC World early to help setup. We had 36 hours of lab guide building madness converting a ton of word documents over to the format my custom viewer uses (sorry guys). We had a dead Cisco C-series blade we fixed with duct tape. And we had a 48 hour marathon to try and fix a late occurring bug in the Demo Cloud that would mean changing core functionality in my code. During this period Chris Horn didn’t sleep for at least 40 hours and Simon Seagrave almost the same. We ended up launching Monday without having the bug fixed. I had to rewrite my code that morning to work completely differently and pray it worked. Monday ended up being a rough day. We got almost 200 labs done but we realized that the bug HAD to be fixed to do what we really wanted to do.

All night Monday and early Tuesday morning the Demo Cloud folks and my fellow vSpecialist, Clint Kitson, worked on an idea Clint had on how to bypass the error. This would mean going back to the original code and plan. And I almost did a back flip when Tuesday morning testing worked. I quickly reverted the code and fired up the system. And Tuesday was when I saw all the hard work and planning pay off. I watched as the system just worked. No intervention and no management needed. People registered, were walked in, sat down to a deployed lab, and watched as the labs reset after each user got up. My stats screens looked awesome also showing live views of progress and breakdowns in the lab (video below). I ended up updating several console tools based on suggestions from the vLab crew and even writing a couple new ones on the fly.

Here is a video of the final stats screens from the end of day on Thursday. We reached over 1,300 labs and 3,000+ VM’s. All of it hosted 2,000+ miles away inside an EMC datacenter. I think this really sets the standard for within EMC on how an interactive experience can be provided from a cloud,

EMC World vLab Final Stats from Nicholas Weaver on Vimeo.

The Chad’s World Live keynote was also just as aggressive and crazy. I will let Chad cover the behind the scenes on this. But I have new respect for live productions and antacids.

All in all EMC World was a major success. Both as a large team and personally I believe we pushed boundaries and did things we have never done before. I threw together a quick video of the entire EMC World experience from my lowly viewpoint. It is long at about 18min but definitely has some cool behinds-the-scenes shots and after-the-show fun.

EMC World 2011 from Nicholas Weaver on Vimeo.

To be honest I don’t know what is next for me. This was just a monster effort I am now on mandatory down-time. I am going to take some time to maybe go outside for a bit and keep away from the keyboard. I wouldn’t trade the experience of the last few months for anything but I also wouldn’t recommend doing it in such a short time to anyone.

Comments and questions are always welcome,


EMC Featured Life/Work Tools Videos VirtualAppliances

11 Comments Leave a comment

  1.  Nick – absolutely stellar job.  I wish you would have come over to the TM team sooner.  If you’d like my 2c, I really think the next move for you and Chris would be to establish a rapid engineering team that you lead focused on VMware big-thoughts that result in a software application. It will be a challenge, but if you can base it on an agreed upon and common framework the handoff would  be pretty easy.  Keep it up my friend – great work.  Oh, and one other thing… I’d have Chad sponsor you on a patent for this. If nothing else, you deserve that feather in your cap.  All the best.  

    • Thanks Scott. I wish I could work on all the teams at once 😦

      Also, wish I could have got some time to work side-by-side with you. Maybe in the future sometime 🙂

      Thanks for the compliments and good idea on the Patent.


  2. I’ve replied to a lot of twitter posts stating this but the work on the labs was greatly appreciated. There were a number of hands on labs that required you to be in line more than 30 minutes ahead of time in order to secure a spot, which I would prefer not to do given the limited amount of time. Personally I’d rather get in as much technical content as I can and these labs made that possible. One of the most interesting labs to me was the Symmetrix Tier Advisor, which was probably one of the “lower” tech labs in the bunch (when compared to having to simulate Isilon/VMAX/VPLEX/VNX/VNXe/Greenplum/etc).

    I hope these labs have a larger presence at future events, possibly offering exercises as long as some of the “traditional” hands on labs. Don’t hesitate to reach out to the community (maybe EMC Proven folks) for testing if allowed; I think that there are a number of people who would be happy to help test the labs/environment.

    • Thanks Jason,

      Our locking issue in the Cloud caused us to be behind on the number of VM’s we need. This caused the waits on Mon and most of Tuesday. Wed and Thurs we had almost 0 wait times after the first hour as we were running as planned. Next year will be even faster if I can help it in any way.

      Thank you for the feedback,


  3.  Nick – I’ve said it before, I will say it again – you rock, man.   Killer work, super-human effort.  What’s next for you?   Well, beyond mandatory vacation (and Wade and I say that dead seriously – trust us – there’s so much going on all the time you could find yourself working non-stop unless you stop to smell the roses), the answer is simple:

    Whatever you want.

    The opportunities are so vast, so much stuff to do – and your skills so useful – there are lots of things 🙂

    And, yeah – I would happily sponsor a patent application.   Again, from one friend to another – THANK YOU.

  4. Nick,   Awesome job by you and the rest of the team.  Maybe my conversation a few months back paid off in an unexpected way… 😉

    Great finally meeting you.

%d bloggers like this: