Daniel Cazzulino's Blog

Go Back to
kzu′s Latest post

Extending Visual Studio: Good Test Coverage Is Possible

A World of Pain?

It’s not quite uncommon to see developers despair when doing Visual Studio Extensibility (VSX) work. In addition to a myriad new APIs to learn (which can be pretty intimidating), it seems up-front so alien to the typical development work you do: you have to plug into someone else’s environment, you have to learn when and why your code will be invoked, but worst of all, there’s no plug-in architecture per-se.

Unlike other frameworks that implement certain patterns (i.e. ASP.NET MVC, WPF MVVM) or "plain" apps/services where you decide more or less up-front what pattern you’ll apply (DDD, Active Record, SOA, etc.), doing VSX feels more like doing transaction scripts over the VS APIs: hook to an event or implement a callback, and when called, go off do your thing and be done. And more often than not, cleaning up the mess is done by trying to apply DRY (dont repeat yourself) by moving repeated code over to all sorts of helpers or extension methods. I’m as gilty as anyone of having created yet another DteHelper.cs in the past.

This seriously hinders testability and maintainability of the resulting extensions, since the "automtion script" ends up being a spagetti of calls into whatever API or helpers that seem to get the job done (or more typically, the first that showed up on a web search). You just want to get done with this crazy thing and go back to doing TDD, running nice and tidy unit tests that run in milliseconds time. But on the VSX side, there seems to be no alternative to testing an extension than slow and convoluted integration tests that take forever to run and spin up VS instances, maybe even using UI automation. And of course nobody wants to spend time doing that, so when deadlines approach, they are quick to be dispensed with in the name of shipping in time.

And this typically makes VSX work not fun at all. Nobody who likes programming really enjoys working this way.

Fortunately, there is a better way.

Single Responsibility Principle (SRP)

There’s absolutely NO reason why sound design principles cannot be applied within a VS extension. SRP is one of those that you have to keep reminding yourself whenever tempted to add just one more method to that helper class.

Components should have a single reason to change. The approach of "automation scripts" that I mentioned, typically end up doing a whole lot of unrelated stuff: analyzing code, generating files or configuration, reading configuration from the registry, showing up a UI, logging some diagnostics (hopefully) to use when things end up broken, invoking web services, setting up deployment environments for testing the app being developed, copying various assets around, and on and on.

Just like any app, you should strive to partition and decouple mercilessly all of the different parts. You’re NOT doing it for the potential reuse in any future VSX work. You’re doing it so that you can TEST those pieces in isolation, in nice, tidy and FAST unit tests that will bring some of the joy back.

Yes, some VSX experts you might bring or have in the team who have been doing the "automation script" way for years could say "I can do that with my eyes closed in < 100 LoC in an hour". But the end result is something that despite working (apparently), is legacy code the moment the guy finished writing it. And I wouldn’t want to be on the shoes of the guy who has to fix a bug or add a feature to that when the expert has moved on to another company or another project.

So, if you have a VS extension that (say) conditionally evaluates some rules while unfolding a template to determine what content to include or not, depending on some value entered by the user in a wizard, by all means resist the temptation of just creating a 100+ LoC class that just implements IWizard and call it a day. You’ll want to split such a beast in (for example):

  • Rules reader (i.e. from XML or whatever)
  • Rules evaluator (pass in a file name, return true/false according to rules)
  • Wizard view model: no markup/UI at all, just the model behavior, validation via data annotations automatically ideally from a base class
  • Wizard markup: ideally with NO custom code, just plain XML with two-way data-binding to the view model.
  • Dialog window factory: creating dialogs and properly parenting them in VS is not particularly trivial and it’s easy to get it wrong.
  • Actual IWizard calls all the above.

Except for the last one which will require an integration test, all the others can be extensively covered by unit tests.

Abstractions and Dependency Injection

Visual Studio is not always friendly in the way it exposes the underlying behavior, and lots of APIs look completely alien in a .NET world. And in many areas, it’s not something that I expect will change. There’s already the “hardcore” IVs* way and the “automation” DTE way and I seriously doubt the VS team will be willing to introduce a third way of doing things, no matter how unintuitive the existing ones may be.

So, don’t be afraid of creating your own abstractions that make it easy to reason and test your code. For example, listening to solution events, showing error messages to the user or what not. The fact that there’s a particularly ugly API that nobody quite understands, is not justification for forcing everyone coming to the project to learn that crap. Spare them the pain and abstract it, create good tests for it (integration  tests if necessary), and you’re done. Hopefully, nobody else in the team will ever have to touch it again. But if they have, it’s well isolated behind an abstraction and not sprinkled all over the code base.