A blog about Open Source, my work at the Gates Foundation and those I am fortunate enough to collaborate with

You can scroll the shelf using and keys

Making the transition to Open Source – a proprietary view

September 14, 2011

Recently had a fabulous conversation with Tony Galluscio Healthcare Solutions Product Line Manager for Harris about his work with David Riley, Vanessa Manchester and Brian Behlendorf on the Health Information Exchange known as Connect!  Purpose was to understand the journey he and his team took in making the shift to Open Source with a broad community.

Fair to say that there were some clear commonalities that popped out during discussion between our work at Gates on the Shared Learning Infrastructure and the Connect project.   You’ll also see themes that will remind you of some of Karl Fogel’s guidance from his book on OSS production.

Here are the highlights:

What were the key actions Harris took to organize for Open Source?

  1. Assigned 2 specific members from each core dev team to act as point for the OSS Community. 1 on point.  1 as backup.
  2. Partnered with a leading OSS advocate who acted as “gardener” to the community.  In Harris’ case this was Brian Behlendorf
  3. Organized Codeathons where a majority of the core team could get together with developers, share info and bond.  Dev team planned to have lower velocity during those 1-2 day periods.  Proved critical to sustainability and ongoing innovation further down the backlog

When did Harris make the decision to open up?  Did they wait till they had a critical mass of code/stories addressed before doing so?

  1. ~3 months into the work David Riley started to evangelize the idea to open up across the team and the project’s stakeholders
  2. ~6 months into the work the code started to be fed out to select individuals/strategic partners like Mirth via email with invitation to focus on contributing code for patches and bugs
  3. ~12 months into the work Harris launched a full SDN with code/commit functionality and ability to download branches.  This proved critical.  For example, Mirth downloaded the code into their own dev environment.  Mirth is an OSS shop with its own community of 6500 followers which helped A LOT.  Gave Harris a Huge leg up in terms of understanding OSS and gaining traction in the ongoing dev process  

How did Harris organize for distributed agile teams?

  1. Wherever possible, put devs together into local teams and identified a leader among them who they could interface with in person.  They’d often form up sub-teams within the team
  2. The broader group was handled with good tooling: Skype, GoTo Meeting.  Harris used TargetProcess + Teams Board for project and code management.  Story cards provided a common basis across teams

How hard was it for Harris to open up the
work, given they were developing a secure PII exchange network for 26 different
federal agencies?

Harris had some early concerns

  1. They were initially very uncomfortable just taking requirements and posting them on the web.  However, Brian convinced them of the benefit and David managed the stakeholder’s expectations.  Harris made the switch as soon as they found a way to write the stories in a sufficiently generic way that they avoided disclosing a particular agency’s schedule.  They prioritized the stories, took direct ownership for those that were prio 0, cleansed all stories of agency names, stripped out sensitive data from the backlog and published it
  2. They were afraid that if they opened up they would give up some of the rigor and certainty of being able to meet deliveries to agencies.  Maintaining velocity was a must.  Harris overcame this issue by:
    1. Introduction of the community point person role in the dev team who could manage community expectations while the core team stayed on track with the prio0 stories.
    2. Scrum leader controlling access
    3. Community having to earn commit privileges by submitting code for consideration by Harris dev team leaders.  Harris developers were all grandfathered in and did not
      have to go through this process.  First approved committer was Mirth.  Required ~6 months to achieve full commit privileges

How did the codeathons work out? Wasn’t it a challenge given there were competitors in the room?

Codeathons exceeded expectations and have since taken on a life of their own (for the good)

Key contributors to success:

  1. Had reps for each agency attend – this was important to helping agencies and their technical staff come up the learning curve on what it means to actually work in open source
  2. Made a point of inviting competing vendors. EOD the whole project was about commodifying data layer so value could move up the stack.  All able to differentiate based on individual strengths.  Result was an ecosystem where they could all do business.  That was the context behind every invitation.  Mirth was one of those “competitors” who emerged as a core
    ally and committer to the code

Once the code was made public, what actions did Harris take to sustain momentum among developers while continuing to advance the larger project?

  1. Allocated 1-2 engineers to oversee community coding further down the backlog.  Once code public, it became quickly apparent that the community was prepared to tackle stories further down the backlog.  For variety of reasons Other OSS stakeholders in the project had different priorities from the core dev team.  Harris therefore invested in engineers to spend time further down in the prio stack because Harris knew that while it wasn’t an immediate prio for its agency stakeholders, it also knew that the broader industry needed
    it.  Using its in house community advocates, Harris was then transparent with the community about why Harris itself was making the prioritization choices it was.  Having a Harris advocate was crucial here
  2. Harris made a distinction between Open Source vs. Open Program, e.g.  what needed to be shared broadly with the community and what didn’t.
  3. Harris set up an area of the SDN where it could keep things private among core stakeholders
  4. Harris adopted a principle of “Local autonomy”, e.g. each agency responsible for determining what info went into and out of that agency.  Harris simply ensured an audit trail was in place to show local autonomy preserved
  5. Harris built in an adapter layer on top of the core stack.  This created a more “plug-and-play” environment that vendors could plug their solutions into. Also allowed Harris to make a distinction between what could afford to evolve rapidly vs. what should not

Finally, wasn’t there a risk that the community would take the code in a completely different direction from the core effort?

That is always a possibility with OSS.  However there was one instance which started out bad and ended very well.  This concerned development of a branch of the code to deal with query retrieve gateway capability.  The work quickly attracted a vocal base who felt that the branch represented a lighterweight solution to the entire project.  There was noise and debate in the developer community for a while, not always constructive.  EOD Harris ended up incorporating the code into their Florida HIE project. Led to a win-win.  Wouldn’t have happened without the fork


Sinofsky saves MSFT again

September 14, 2011

I wasn’t sure how to title this one (after all, it’s a 03:30:00 keynote!!!!)  There’s so much in what Sinofsky lays out (summary @ 01:42:52 in the video.  WinRT = equivalent of AMZN primitives for native apps.  From a business model perspective his team has made Windows into a software-licenseable equivalent of a “cash-and-carry”.  It’s a smart, smart preemptive move on behalf of the OS)  Even Mossberg caught on to the fact that Windows 8 is the biggest step fwd since W95. “Touch-first” or “Tap to sensor” anybody? (wave my business card over the PC screen and it goes straight to my blog)

When Sinofsky talks about reengineering from the “kernel on up” and “fundamental performance at the base kernel level”, he’s talking about improving a literal platform, not the hackneyed figure of speech that gets tossed around these days.  Cold boot in under 8 seconds and then straight onto your web apps (not forgetting the hardware accelerated graphics)

What said, note that the seamless in-page UI experience is all Apple, while the equally impressive system experience is uniquely MSFT.  Care to take bets on which company will capitalize on all this advancement with consumers??

EOD, it is exciting to imagine what a teacher’s start page might look like powered by something like the SLI API inside of Win 8 (or Lion OS++).

Not exciting enough??  Well imagine that capability coupled to this  Want to log in to work?  You lie sucka!!!!

Can’t resist:  final note (a prediction) – Sinovsky will be the next MSFT CEO and will be responsible for breaking the company up and unlocking 100%+ shareholder value.

Final, final note: you don’t need to watch past ~01:50:00.  Although if you want to see a developer geek out on his own geekiness fwd to ~02:45:00.

That’s it.  Doodles passed out long ago.  Off to bed.

Gerardo Capiel and Benetech are coding4good. Can the Gates Foundation help?

August 7, 2011

While at Oscon Danese Cooper and Brian Behlendorf kindly invited me to sit in on discussion of a new initiative that Jim Fruchterman,  Gerardo Capiel and the team at Benetech are cooking up to mobilize OS developer talent against the societal issues we face in areas like Human Rights, Healthcare, and Education.  I was very glad that I did and I am hoping you can contribute to my thinking about how we at the Foundation can help this very worthy initiative.

Gerardo is Benetech’s VP of Engineering.  He’s motivated, passionate for the cause, and a marvellous bloke to hang out with.  In 30 minutes I was given 3 demos, the history of Benetech, and a nagging sense that if we could find some excuse to pair Benetech’s savvy with Gates resources then crazy good things might result.

Gerardo led off the discussion by painting a pretty compelling picture: Foundations and other NGO’s could publish issues in need of developer talent.  In return, developers would have access to new opportunities to diversify their programming skills and build profile through contribution to critical issues.  What would make this different from or similar source exchanges is the ability of the venture to leverage its brand in order to present meaningfully curated opportunities to the community, rather than just random job posts.  Second, Benetech would leverage its network to pair up sometime significant Foundation resources against shards of developer time that are going to start to emerge as a result of the new paid time off (‘PTO’) policies being instituted at place in the Valley like VMWare.

From my own experience I am somewhat familiar with the messy and frankly limited-value experience developers risk getting from donating time to non-profit software projects.  In my case I acquired it at MSFT while organizing a skunk works OS project housed in Codeplex which was aimed at producing a lightweight SMS-based reporting app for health extension workers in Ghana.  It was sort of like how I imagine sport sex to be: anonymous, casual and in the end, naggingly unfulfilling.  Everyone exited the project unsure of their ultimate contribution and wanting to feel better about it than they actually did.

This is where a group like the Gates Foundation could bring resources to bear on behalf of Coding4Good and raise the quality of the developer experience while contributing to the likelihood of sustainability for the venture.  I wanted to give you a sense of where I think a Foundation could contribute (with thanks to Theo Schlossnagle for validating some of these points during the discussion):

Leverage the Foundation’s megaphone on behalf of the developer community – getting the word out and sustaining it can be an expensive proposition these days.  Foundations will have resources for the community to use on its own behalf

Budget for the time volunteers will need to ramp -It’s notoriously hard to get a resource spun up in a timely fashion.  Providing a developer with context for why their contribution is needed (and valued) can make the difference between the project enjoying the efforts of a committed and motivated developer versus a work-for-hire mechanicalturk

There is no substitute for a full time PM/ScrumMaster – I learnt this first-hand.  Consultant or part time won’t wash.  Experienced PMs are hugely valuable (and a frustratingly scarce commodity in today’s overheated Valley environment)  Foundations can often dig deeper than non-profits to secure and retain the right PM talent for the duration of a full release or sprint sequence.  Khan Academy recently demonstrated this fact hiring away the amazingly talented John Resig from Mozilla (Footnote: I do worry about sustainability of Khan’s hiring approach. With the perks starting to fly around the Valley, Khan’s hiring halo will begin to wear off and they need to plan and budget accordingly.)

There is no substitute for a full time UX either! – One of the first casualties of almost all development processes is the UX.  It’s a real discipline and so often falls victim to other priorities or the tyranny of budget and “deliverables” (this is why I LOVE UX and Agile development process and why I advocate for it at the Foundation!)  1 UX + a strong PM can equate in many instances to a more traditionally staffed development cycle if it is tightly coupled to developers

Federate but know where to tightly couple – and in those instances where you need to tightly couple, marry your PTO resources with the necessary resources and a sequence of clearly delineated sprints

Let developers stay in touch with the code and see the impact of their contribution – this is where thoughtful use of control systems like Subversion can make or break a collaboration.  Foundations will have little to no idea of how to manage a community and its contributions.  Benetech – and talented souls like Gerardo – will

Please ping me with any refinements to the list above.  I hope to have Jim and Gerardo up to Seattle to discuss a collaboration around their new venture and would appreciate any thoughts you might have on how Gates could contribute to a meaningful developer experience coding4good.

Alex Szalay – Progressive Rocker to the Stars

August 4, 2011

Alex Szalay

Alex is the Alumni Centennial Professor at the Department of Physics and Astronomy, Johns Hopkins.

Thanks to Roy Pea, I had the happy opportunity to sit with Alex during a dinner I held after a workshop of Big EdData and the future research ecosystem we would require to support personalized learning.

A leader in cosmology, Alex has been behind the development of applications that have democratized science and opened it up to enthusiasts and spacehackers.  His projects include the Sloan Digital Sky Survey, his ground-breaking collaboration with Jim Gray and Curtis Wong on the World Wide Telescope, and the Galaxy Zoo which crowd-sources galaxy classification across some 250,000 contributors worldwide.  As an integral part of his work, Alex is also blazing a trail in the use and application of ultra-large databases which I will characterize here as 40TB+ with over 1 GByte/sec I/O speed, using various scalable designs.

That and Alex’s warm personality and quiet humour would be enough to make him a fabulous dinner companion.  However, Alex is also a product of Communist Hungary who sought an outlet for his feelings in Progressive Rock.  Alex was first turned onto guitar in 1962 when he was able to see the Beatles play live in Stockholm during their Hard Days Night tour.  Alex remembers hearing very little of the set for all the screaming going on around him.  Alex elaborated in a recent interview that later that summer, he won some prize money from a Hungary-wide high school math contest. “Much to my parents’ disappointment,” he said, “I spent it on an electric guitar.” Having developed a taste for the blues, Alex fingers the Hungarian composer Bartok as the reason he fell into ProgRock (Emerson Lake & Palmer, King Crimson).  Alex still plays today with his son.

I am very glad to have Alex involved in our thinking.  You can learn more about Alex and his passion for music here.  His official site can be found here

Great community leaders are born not made

August 4, 2011

Bruno Souza

While at OSCON I had a chance to sit down with Bruno Souza aka ‘Javaman’.   You can find Bruno here

Bruno is an awesome guy with a more than passing resemblance to Russel Crowe who spends his time evangelizing among the Java and Open JDK community.

Bruno’s network extends to about 70-80,000 users worldwide.   He is currently in the middle of the US leg of what appears to be an annual world tour that he has been on pretty much ever since he joined Sun way back when they launched Java.

I asked for his thoughts on managing large user groups and any advice he might have for Gates as we start to engage with developers.  As I listened to what Bruno had to say, I quickly realized that Bruno is a what they call a “natural” with an unceasing
passion for his space and the mission he is on.

Here are nuggets I caught from our chat:

Only connect – ok so EM Forster said it first, but anywhere and everywhere that Bruno goes he is reaching out ahead of time to arrange a meeting or dinner with the local user group. No agenda required

Conversions come one at a time, (and not where you always expect them) – Bruno shared a great anecdote about how he broke into a vacation with friends in northern Brazil to give a presentation at the local university.  They closed down the entire CS and Science departments to turn out and listen to him. During Bruno’s talk, a particular faculty member grilled him relentlessly and aggressively.  He walked away feeling pretty down about the incident. Years later he ran into a young Java developer, the product of a new program in Java at the very same university Bruno had presented at.  The head of the program?  The very same lady who grilled Bruno…

Make people important locally –  Don’t try to control the activities of local user groups or try to be THE LEADER.  Support the emergence of local leaders and give them the room the need to operate as they see fit.  Make sure your contribution is an input to their activities rather than a directive.

Make sure people feel ownership for the message – while folks won’t always get it word for word, they should feel able to adapt your message to suit their audience and express it as they see fit

Give groups time – managing user groups and community is a journey.  It takes time.  Don’t go into it unless you are prepared to commit for the long term