Demo Showcase Suite – A History – Part 9

Continuing on my series recounting how we got going on Demo Showcase Suite

7.  I wanted a reliable build and deployment process from the start

Okay this is pretty basic, and something I’ve insisted on for each project I work on.  You can’t afford to build production software on a developer’s machine.  It’s a no brainer though I’ve seen it done many, many times.  What we did on this project, was to convert from using CruiseControl.NET to TFS Builds.  This was new for me.  Why would we do this you ask?

First, we had converted from VSS to TFS.  I thought it would be better to be consistent and use the integrated tools we had.  Second, we were also having scaling problems with our current CruiseControl server, and we were going to need to upgrade it anyway.  So we made the leap to TFS with the plan to retire the CC.NET Server.

The other thing converting to TFS did was to take me out of being the expert on the builds, and I was able to delegate learning about it, and doing the setup to someone else.  I didn’t have time to do it, and as long as we were using something that I had put together, the likelihood of me continuing to be ‘the build guy’ was too high.  My very first job when I started in IT in 1994 was managing the nightly builds and version control.  In the 14 years leading up to the start of the project, I was always the build guy.  Now, I’m not the build guy anymore. 

We ran into some issues with deployments to Azure later when we tried to build on a 32 bit machine, so we had two build servers, a 32 bit machine for the client application and a 64 bit machine for the server apps.  This is probably not needed.  With a better understanding of configuration manager in Visual Studio, we probably could have everything working on just one, but we had so many problems with the vsmdi files that by the halfway point of the project, we were seriously considering other options.  Having a split build, and running the tests only in the Test build, not Prod and QA, helped a lot as well.

As far as the deployment process went, we scripted everything to get as far down the Azure path as we could, but we still had to manually upload to Azure, deploy and swap. There were two of us who became the experts on this, and kept everyone else’s fingers out of the Azure pie, just to be safe.

We spent a lot of time building the MSI for the client to have it do everything we wanted it to do.  A lot of what we learned we are able t0 apply to other projects right away, and though it took far more time to get it perfect than we wanted to spend, it sure is nice to have a process we just always trust.

I’m feeling ambitious tonight (and wanting to get onto technical stuff sooner, rather than later), so I’ll cover #8 and #10 tonight.  #9, as it turns out, is a whole series unto itself.

8) I wanted the team to be able to provide me with feedback quickly and easily and to give both myself and the PM a heads up well in advance of a red flag issue becoming critical.  No surprises.

I think I’ve covered this already, but this was all about communication.

• We implemented the scrums, as previously talked about. 
• We used task tracking in TFS to flag blocked items.
• When someone is worked up about something, we dug in and broke the task down so I could see and understand what the problem was.  Don’t let people stew, whether it be a technical issue or a personal one.  Get it out in the open and deal with it.
• We had whiteboards in the middle of the work areas.  If there was a major issue, we grabbed people and whiteboarded it out with input from everyone.

I think we did really well in this one.  There were almost no surprises from within the team, and the few that were came from me not listening carefully enough when tasks were starting to slide in the early days of the project.  I learned to listen for the signs that trouble was brewing, and made an effort to head it off earlier.

10)  I wanted to enrich my technical knowledge base.

I had no idea how much I would learn on this project.   I still can’t believe how much I was working on that almost no one else in the world was doing.  When it came to doing the first part of the security system, I think there were two or three of us on the MSDN Forums bouncing ideas back and forth.  Dependency Injection, Rhino Mocks, WPF MVVM.  I learned so much in the first few months, I thought I was going to drown.

For the record, I am not going to claim, as I begin writing about the technical aspects of the project, to be the expert in any one area.  I am sure (absolutely positive, actually), that for each area I will get into, that there are a dozen people, if not a thousand, who know more.  I’m just going to show you what we did, for better or worse.  Some will laugh and point and snicker at some of the code.  Hopefully I can get some feedback on better ways to do things.  Hopefully someone out there getting started on Azure or MVVM or Silverlight or Demo Showcase will get some information that saves them some time or money.

If you have a specific area of the Demo Showcase architecture you want to hear about, let me know, and I’ll try to cover it in an upcoming entry.  If you have a question about how to do something in Demo Showcase, or have a suggestion for a new feature, send it in.  I’ll get it on the list.

Comparing Crafts

I’m going to take a brief diversion from my series on Demo Showcase to discuss a parallel I found this weekend between developing software, and writing. As you can probably tell by the length of the entries on this site, that I love to write. I love it so much, that I have a web site completely devoted to my writing at www.joebeernink.com. I write novels. I write short stories. Hell, I like writing specs. But I also like to code. I like to dig into new technology. I like to solve hard problems. I thrive when faced with tough challenges. But back to the writing.

This weekend, I attended the Pacific Northwest Writer’s Association Conference new Seattle. This is a four day writer’s version of NerdDinner. Instead of hanging with Scott Guthrie and Scott Hanselman, I got to hang with Terry Brooks, Robert Dugoni, Will North, Joseph Finder, Richelle Mead, Caitlin Kittredge, and a thousand other writers and aspiring writers.

People talk often about the craft of writing software, and I think that has obvious connections to the craft of writing. Writing crappy software is easy. Writing crappy stories is easy. Writing good software takes education, practice, persistence, self discipline, structure and patterns, research. Writing is exactly the same.

You need to be skilled at the language you are writing, and know the vocabulary. Writing a great novel doesn’t happen on the first try. Just because you have spoken English your whole life doesn’t mean that a good novel will result when you put pen to paper, or fingers to the keyboard.

You have to keep writing, even when people give you negative feedback. If we all stopped writing code every time a user found a bug, or the compilation failed, or took it personally, we’d never get better at it.

You need self discipline to always strive to learn, to read that dull book on ASP.NET MVC, even though you think you already know ASP.NET. I mean, nothing will ever make VB 6 outdated… right?

You need to understand the structure of a good story, and the structure of a good piece of software. There are patterns and practices that have evolved, and yes, you may find another way to do something, and it doesn’t mean you are wrong, but there are some things that are just not worth reinventing. At least know what options are out there, and use the power that’s in them until you find a need to diverge. This isn’t meant to stifle progress. It simply means that not every piece of software is going to be a unique and brilliant creation. We get paid to turn out software that works. Novelists get paid to turn out writing that sells. If you don’t want anyone to read your book or use your software, go crazy.

Research the stuff that interests you. Read code from similar programs. Novelists need to read other authors from within their genres, not to copy, but to learn. Know what that code does before you copy and paste. Understand why it works. Find the patterns in plots of great novels. What tools does the author use?

None of these are spectacular revelations, or at least they shouldn’t be. But something hit me, a parallel between my career as a writer and my career as a software developer.

I started programming computers on a Commodore 64 in 1982-83. I was 11 I think. Actually, I had programmed on a PET CBM prior to that, but the C-64 really got me started. I took programming courses in high school and college, did an internship in college, and spent the years 1994-2007 working in the field taking a few courses here and there, picking up a book when I needed to, but never really taking charge of my career. But at the end of 2007, I realized my career was at a dead end. I had let my technical knowledge stagnate, and was blaming everyone else for my lack of opportunities. I made the leap from IT department at a large airline to a small consulting firm at the start of 2008, and vowed that I would spend a minimum of 3 months reading everything I could, as fast as I could, about the technology I was going to be working in, to get better, I wasn’t going to take courses, or expect someone to teach me.

I read on the train. I read at night until I couldn’t keep my eyes open. I read on the weekends. At the end of the three months, I realized that I had learned a lot, and had a lot more to learn. I filled up my wish list on Amazon with all books on every area of software development that I wanted to learn. I got a subscription to MSDN Magazine. I read the articles, instead of skimming them. I added dozens of blogs to my RSS feed. I read whatever I could there. I gradually brought my skills up to the point where I felt that I could talk intelligently about software development again.

Similarly, I started writing stories in grade school. I wrote them freehand until I had a Quick-Brown-Fox word processor for my C-64. Then I typed them out and printed them on my MPS 801 printer. I took a lot of English classes in high school. My schedule was too full to take classes in college, but I did write my first novel while there. I loved writing for the escape it provided from the pressure of my classes. I wrote a bit after college, but it was in fits and starts. At one point, I stopped altogether, until August of 2008. That’s when I got the laptop I am typing this on (a Toshiba U405-S2854 which I love).

I started writing a new novel on August 8th, and finished the first draft on January 1, 2009. But I missed something. I thought that writing was something I could just jump back into, and do without reading technical books on how to. I didn’t bother to figure out what genre I was working in. That has huge implications on what rules you need to follow. I didn’t bother to read other authors in my genre to see what was working. I didn’t practice my craft and look for feedback. I just assumed I was good, and smart, and that this writing thing wasn’t that hard.

This weekend was an eye opener for me. The successful authors study their craft. They practice, sometimes for more than a decade, before they get something published. They treat their hobby like a business. They are the CEO’s of their own corporation. They get feedback from their customers. They manage their brand. They work their asses off. You could see those in the audience that weren’t prepared to do that. They blamed agents and editors for their work not getting sold, or argued with the agents and editors about the quality of their work. For a new writer, that’s like arguing with the C# compiler that it should know what you meant, not what you entered. The compiler will be right 99.9999% of the time. If you go to three different machines, and the problem is repeatable each time, it’s your problem. If three agents tell you your writing sucks, it’s your problem. Fix it.

So why am I writing about this on a tech blog? It’s because it took me 13 years of my software career to realized that no one is going to make me a better coder / architect but me. It took me a year of writing, and a brutal slam from an agent to realize that I need to work as hard at writing as I did at improving my tech skills. And that takes time, and a plan. I filled up my shopping cart at Amazon last night with great books on writing, and expect to spend the time between now and Christmas reading those, and using the lessons I learn from them to improve my writing.

Where does that put my technical career? Right where it was. I just need to do both. To find more hours in the day. To work smarter. To learn the lessons I’ve been talking about in all these other blog entries so that on the next project, I don’t spend days regression testing manually. To do it right the first time. There’s just so much to do and learn out there, why screw around reinventing the wheel? Hopefully, after reading these entries, I help you to save time you could better use to bring your skills up to date, to create that vicious, incredible cycle of learning and doing things better and faster to give you the success and the time to do the other things in you life that mean something to you.

Good luck!

Demo Showcase Suite – A History – Part 8

Goal #6:  I wanted to plan for change, in scope, in the composition of the development team, in technology and in time lines.

Change on a project is inevitable.  Even on short projects of a couple of weeks, there will always be a small change that causes some consternation amongst the team.  I was conscious of four types of change:

1.  Scope. 

Everyone has dealt with this one.  You can’t fight it.  The best way to deal with it is to track it, to have good processes for handling it, and for providing feedback to management and the customer as to how each change in scope will affect the project.

Prior to the start of this project, our change tracking was minimal, if nonexistent.  We used a web product called Gemini for tracking bugs and some feature requests for existing projects, but we didn’t track deliverables and tasks for new projects getting started.  We used VSS (horror of horrors) for our source control.  It was quite clear that as an organization, we were outgrowing these tools. 

We made the decision to try something else as soon as the project was announced.  We had access to Team Foundation Server and Visual Studio Team System Team Suite without needing to spend a bunch of money, so we used that.  It was a big step forward from VSS, and it took quite a while to get up to speed on how to use it, and how to use it properly.  There were definitely some very painful moments, and we’ll all be glad when we can upgrade to TFS 2010 to get things like hierarchical task trees and better integrated reports.  We went back and forth on the best way to use the system, and I think, at the end, we were doing the best we could, though there are more features that we can probably gradually integrate into future projects to make us more efficient.  We didn’t take advantage of  the Estimate / Actual / Work Completed feature, and that is the one thing I would really like to change on the next project, to not just get a picture of how many tasks are outstanding, but how we are tracking effort to the goals.  It would also greatly improve our ability to provide feedback to everyone when a scope change comes in to see how it really affects the project time line.  Right now a ten minute bug fix carries the same weight as a 40 hour feature implementation on the task list, and that is obviously misleading and wrong.

2.  Team Composition

We only had one change in the team in terms of losing someone from the team from beginning to the end of the project, but we did gain two new people as well.  On a small team, it’s essential that in one person’s absence or departure, that another can step in and fill that role, or at least be able to understand what that person was working on so the work does not come to a grinding halt.  The only way I know how to do this is for everyone to be doing things the same way.  If everyone uses the same tools, writes code to the same standards, there’s a lot less chance that code will be a mess when someone turns it over.  I harped a lot to the team about a couple of things in particular

a)  Everyone had Resharper and was expected to make that little light at the top of the file go green every day.  Okay, part of this is due to the fact that I am anal-retentive when it comes to clean code, but part of it is a matter of enforcing standards.  I can’t sit over every developer’s shoulder and watch them name their variables, and set their tabs.  But Resharper, and the lovely Ctrl K D really left us with some pretty clean code.  It’s amazing how much easier it is to work on someone else’s code when you are not scared to open the file.

b) Nightly Builds:  We used TFS to do our nightly builds, right from the start, and constant check ins were encouraged.  Everyone feared the words ‘Who Broke the Build?  UNACCEPTABLE!” being shouted across the office.

c) Unit Tests:  As previously discussed, we could have done much better here, and we are working on making this better for the next phase.

d) Unity:  We used the Unity Application Block for dependency injection (DI).  I’ll go into more of the Unity overview in a later post, but because we had Unity set up early in the project, we were able to create mock objects and able to swap code in and out easily so that itwo people were rarely dependent on one another’s work.  This was especially important prior to bringing on the new people to fill a void in some of the back end code areas so that UI development could proceed until they were up to speed.

I also spent some time creating a ‘Technical Overview’ document that summarized some of the technical decisions we made early in the project, some of the issues that were left to be resolved, and a quick reference for some of the more complex processes we were using.  I don’t know how much this helped the new guys, but it made me feel better that I had at least tried to help them come up to speed without them needing to spend a week constantly asking questions we should have documented.

When I started my first IT job back in 1994, my boss gave me ownership of the New Employee Orientation / Standards book, and told me to update it when I found a question it didn’t answer, as I would be handing the book off to the next new person, who would then come to me first to ask questions.  I love this chaining of documentation pattern.  It gives the new people immediate ownership and responsibility.  Of course, some people weren’t as robust in their documentation as others were, but if we held true to the pattern, everyone would see the benefit of keeping the document up to date.  It made sense back then, anyway.

3.  Technology

We knew the technology was going to change, and change rapidly during this project.  Managing this change was a constant challenge.  Again, Unity and DI played a large role in helping to resolve this issue.  There were many times where we had issues with Azure, or did not know where the data would actually reside at the end of the project, but that didn’t stop the developers from continuing their work.  We built interfaces and mocks, and swapped the managers in and out using the configuration files. 

During most of the project, there were only two of us who ever touched Azure, just to insulate the team from the environmental changes.  We defined the processes and worked through the issues, but the rest of the team barreled forward, not needing to know the specifics of how the system would work on Azure.  This was true for deployments, data storage, security and configuration.  Almost everything we built could run directly in IIS7 or in the dev environment without Azure being available.  Pieces were converted over to run on Azure one at a time as they were ready with a few configuration changes.  I can’t emphasize enough how important this is when working with dependencies that are constantly changing.  Isolate them, mock them, keep the team moving.

4.  Time Lines

The final date was not moving, but that didn’t mean that there wouldn’t suddenly be a need to give a quick demo of what we had to the customer in the middle of a big dev cycle.  The builds saved us there, and having two (I would have preferred 3) environments to run the app in from early in the project allowed the devs to keep working, while the ‘PROD’ site was only updated as we achieved more stable releases.  Having the multiple environments saved our bacon more than a few times.

Demo Showcase Suite – A History – Part 7

This is the next in the series of entries about Demo Showcase Suite and the goals I wanted to accomplish.

5.  I wanted the team to feel the urgency of working hard and smart early in the project to avoid the death march mindset at the end

Here were the things we knew, right at the start of the project:

a)  The deadline was July 13th.  It was not moving.  No matter how many times I wished we had a few more months, weeks, days (or even hours), we were launching during the keynote on opening day of WPC.

b)  We had huge technological hurdles to overcome on areas we had never seen before.

c)  The customer wanted frequent updates on the work, samples of what we were doing.

To cope with all of this, we implemented some project management processes

1)  We broke the project into 3 week sprints.  This was our first time doing this, and I think we took the word sprint a little too seriously.  We were always up against a deadline of some sort, but that was driven by the final, immovable deadline.  When we saw our iterations slipping, we weren’t able to slide requirements to follow-on releases, but we did see that we were behind, and worked harder earlier in the project to get back on track.  We also were able to keep the developers focused on the important features instead of getting wrapped around the axle on something that wasn’t supposed to be started for a couple of months.

Iterations worked for us as well, as we were able to better manage the customer expectations for the functions they would see in the next release.  They knew what to expect, and knew what not to expect.  That went a long way in keeping down the surprises for both parties.

In the end, we did one ‘Alpha’ release, two ‘Beta’ releases, and 3 ‘RTM’ releases.  In retrospect, we should have called them Alpha 1, Alpha 2, Beta 1, Beta 2, Beta 3 and RTM 1 to properly set everyone’s expectations not just to the functionality, but to the quality, but that’s a lesson we learned the hard way.

2) We held a team scrum every day at 1:00 PM.  We tried to keep this to 15 minutes a day, and I, as scrum master, tried to keep the team focused on three things:

i)  What did you complete since the last scrum – Just a list, not a detailed dissertation.  This helped everyone else know what was ready to test, or what dependency they had been waiting on was now complete.

ii)  What are you working on between now and the next scrum  - Just a list.  If this was the same list you provided the day before, we hadn’t broken the tasks down far enough.  If that continued for more than a few days in a row, it was time to take it off line, break down the tasks some more, and add resources if needed.

iii)  What red flags are still outstanding that are keeping you from completing your work – Details as needed, especially if the task was taking much longer than expected or progress could not be made without more help

We tracked all of the tasks in TFS, though not as consistently as I would have liked, and added details / blocked status as needed.

At the end of the scrum, I stepped in to try to work any red flag issues to get the tasks back to green flag with each developer.

I’m a morning, person, and do my most effective work early in the day, so holding the scrum later in the day enabled me to break my day into two pieces.  Some days, some of the red flag issues carried over to the next morning, but usually, I was able to focus on the big picture tasks that were assigned to me in the morning.  The afternoon was spent solving problems, to keep the developers moving.  It stopped a lot of wheel spinning (I’d like to think) and made sure no one was stuck on an island.

Having the daily scrums gave me, and the rest of the team, constant feedback on how we were doing to meet our next milestone, and how much effort we needed to drive into the current iteration that week to get it done.  Sometimes that meant a weekend in February or March was spent in the office trying to clear up the backlog.  But we knew we couldn’t grow the schedule, so it was better then, than to wait till the last minute when other problems would undoubtedly arise.

I know for absolute certainty that without these two processes, we could not have been successful.  I also know that far more ‘project management’ time was charged to this project than was planned because of these scrums, but we probably saved just as much dev time by stopping the wheel spinning, and allowing me to focus on my tasks in the morning without constant interruptions.  With a team of more than two developers on a project, I would absolutely do this again.

Demo Showcase Suite – A History – Part 6

Honestly, I’m trying to do these once a day, but I took 3 days off for a vacation, and it’s taken me a couple of days to get caught back up.  Here’s the next installment on my series of goals that I had when we stated on Demo Showcase.

4. I didn’t want to get to a week from the delivery date, need to make a change, and have to regression test everything by hand.

As I write these entries, what I thought were distinct topics, all seem to be related.  This is talking about unit testing, right?  No.  Wrong. Different topic altogether.  Application regression testing has always been a thorn in my side.  I’ve never had the chance to put a good test automation tool to work for me to help with UI regression testing.  Along with unit tests, I wanted to have a tool that would let the QA’s quickly and efficiently automate their tests so that the tests they did in March would be instantaneously rerunable in July.  Sure, unit testing will get you part way there, but nothing beats a human looking at the screen and seeing if the right result is a reasonable result.

If there was a swing and a miss on a goal on this project, this was the big one.  In fact, I didn’t even get the bat of my shoulder.  In the post mortem discussions I’ve been having with the team, this one popped up to the top of the list for the QAs as something that directly affected their personal satisfaction with the project and with their jobs.  With the rapid pace that we were kicking out builds each day, the QA’s rarely got a chance to work on a stable version of the application for more than a few hours, and some tests would take an hour to go through by hand if they didn’t run into bugs.  They had a ‘happy path’ test they repeated each time, and there were many versions of the application where  they never got to complete the full test.  If we were to look at the number of hours spent testing on this project, I would think that we spent far more hours testing than the value we got from it.  That is, since so much time was spent testing the same thing over and over, little time was available to test edge cases until the build really stabilized.

So coming out of this project, we still have this as an open item to find an automated regression test suite that works for what we do.  I don’t think you can really evaluate the return on this investment prior to becoming proficient in the use of the tool, so it may take a few projects before we really see the benefits.  But it is definitely something we will be carefully considering before the end of the year, and likely before the next release cycle.