OpenFeature Pt. 2

In this episode, we interview Justin Abrahms, the Principal Architect/Governing Board of OpenFeature, to discuss the diverse landscape of open-source development and its impact on the tech industry. He discusses the early days of open source, when companies like Google and eBay were just beginning to recognize their potential. Justin explained how these companies, although lacking the breakout successes of Kubernetes, played a crucial role in the open-source ecosystem.

He believes there's room for improvement in open-source management, advocating for a more systematic approach to stewardship and organizational support. Diving deeper, Justin explores the driving forces behind different approaches to open-source projects. Steering toward the future, Justin explores the potential paradigm shifts in open source, including how Rust rises to prominence and more.

---

In this episode, I’m pleased to have Justin Abrahms on. I’ve known Justin for a couple of years. We tried to plug him into some software when we first met. We've been working together on an OpenFeature, which is part of the CNCF process of progression in terms of their pipeline. Justin, thanks for joining.

Thanks so much for having it.

The reason that we contacted Justin is that one of the things that we haven't discussed on this show that Justin has a fairly decent amount of experience in is how large organizations approach open-source, contribute to it, how they manage resourcing it, paying for it, and all that stuff. To set the scenes for that, it might be good to introduce yourself and talk a little bit about your career and the sorts of companies you've worked for.

I’m originally a self-taught programmer. I’ve worked my way through agency life, working for startups and some of the biggest companies in tech. I’ve worked for Google, Amazon, Walmart, and eBay as my global corp experience. They're all very different corporations, as you might imagine. Walmart is different than Google. They each take a different approach to open-source.

In the period that you were working for those guys, they all built their entire tech platforms with hundreds of open-source projects, libraries, and platforms. Is it fair to say that it would be unusual for them to have a commercial agreement with Oracle or some big CRM platform or something?

Everyone has an Oracle license and they desperately want to not have an Oracle license so there's a lot of work to migrate away. That was a thing at Walmart, eBay, and Amazon.

For those companies, was working on open-source something that you wanted to do, they offered, or they had a policy around that interested you and was one of the reasons you joined those organizations? In terms of your involvement with contributing to open-source, what's your personal history of that?

I got my start in PHP land and wanted a better way. I ended up in Python. Through Python, I got involved in the Django web project, a bunch of that side of the community, and the Python web framework world where open-source was critical to my growth as a software engineer. I already had a pretty strong we-should-do-open-source personal ethos coming into those companies. Some of them have supported that work better than others.

As you can imagine, Google has a strong open-source publishing track record. When I was there, at least, there wasn't quite as much of a consumption of open-source at that time. I’m not sure how that's changed in the time since because things didn't work at the scale that Google was operating. Since then, the industry has leveled up quite a bit. They may make more use of open source.

You mean they were operating at a scale that they had to build their stuff.

You couldn't get my sequel and put Google through it. A lot of the things that we take for granted in our industry are coming out of Google. We were doing map produce stuff before I had heard of map produce anywhere else. There's a lot of that thing. Kubernetes is an example. That's based on an internal Google system. A lot of the software and open-source didn't work for use inside Google but Google had tons of cool stuff that they were building and needed to push it out, which has been an interesting experience. I’ve not been involved in those massive projects like Kubernetes. Companies like Amazon were users of open-source pretty widely.

There are a lot of things that we take for granted in our industry now or came out of Google.

As part of those roles, did you join those organizations with a personal goal or commitment to work on those? How did it work? My experience was that we've been working together on an OpenFeature, and you were contributing or putting some of your time, at least while being paid employment, into a purely open-source project. If you looked at it in one way or being countered, you might be like, “Why is he doing that thing?” Was that something those companies had policies around that you wanted to make use of?

Every company I’ve worked for has had an open-source policy, at least in global corp land. They're different, depending. I was fortunate enough to help write the open-source policy at eBay, where I did my OpenFeature contributions. I wouldn't say that I necessarily set out to do open-source development but that's how I work. It's a natural consequence of me doing the work that I do. It might be worth talking a bit about how the contribution of the OpenFeature came about as maybe an example.

My job at eBay spanned a fairly wide amount of work but it fundamentally was helping people deliver software more effectively. One of the things that we did was do an analysis of where the bottlenecks are. One of the bottlenecks we found is that people didn't want to deploy because deploys were risky. They didn't want their feature to go out because it wasn't ready yet. We started talking a lot about how deploys and releases aren't necessarily the same concept.

‍

We can use things like feature flags to limit what customers see but also integrate our changes and deploy those to production more regularly, which stops a bunch of merge conflicts and other issues. Internally, we didn't have a strong system for doing feature flags. It wasn't very robust. I did a big build versus buy analysis. Due to how internal eBay systems work, we couldn't use a vendor because we had this massive experimentation thing that had 100 engineers on it.

We ended up building ourselves but that seems risky from a non-invented syndrome perspective. We don't want to be in a position where we have to mutate open-source code when we consume it so that it can become experimentation-aware or feature flag aware. The OpenFeature project was getting off the ground. I got involved and started writing the Java SDK for use at eBay. That is how engineers at eBay are doing feature flags.

They've written a bespoke private provider for OpenFeature.

The team who runs the experimentation world has built an OpenFeature-compatible provider.

For those of you who have no idea what Justin and I are talking about, check out the OpenFeature project. It's a great project. That's how we both know each other. Both flags that are involved in it as well as a bunch of others are feature flagging providers. It allows you to use a single SDK, which Justin is talking about. He has written the Java one. Essentially, it helps you prevent vendor locking, which is a big problem in feature flagging. If you are going to spray this SDK all over your codebase, then you can do it to a common denominator rather than a provider.

I didn't know that. That's a testament to the design that the platform was originally built around, where you built a proprietary private. It is a private adapter to an eBay. It’s an internal eBay system. We flag with the written providers for different languages. That's great to hear. That's interesting. There was a very direct return on investment argument in terms of, “I want to work on this.” Can you talk us through the process? Was there some formal process you had to go through to say, “I’m going to spend two days a week writing code that's not licensed to eBay and is going to be part of this liberally licensed CNCF project?”

The process was pretty lightweight for me and that was due to two factors. 1) I was a principal architect at eBay. With that title, they give you a bunch of latitude to go do the thing that they think is best, which is very nice. 2) I complained enough about the open-source process, having open-sourced a couple of other things earlier to the person who was responsible that they then made me responsible. I had eBay run the open-source program. It was very easy to convince the four other people on my open-source committee that I should be able to publish my open-source code.

In terms of the other things that you open-sourced, are you able to talk about those? I’m curious to know. Were they things that you'd written internally that then someone was like, “We should open-source this?”

The things that I released were a couple of experimental developments in the open attempts. One was a project called GOOSE, which is a little tool to help with GitOps flows. There are a bunch of teams for whom we wanted to adopt a more GitOps way of working. We didn't want them to have to know what Git was from an integration standpoint.

The thing that this web service does is it monitors the change log through webhooks. If it sees the specially named file that you've registered, like experimentation engine YAML or something, it will then take the contents of that file and send it to that web service. They can speak HGEP but they get the benefit of GitOps. That was an example of something that I open-sourced while I was at eBay.

Was that something that you had designed and built with the intention of open-sourcing it? At what point was it like, “Let's go and open-source this?”

That project was built with the notion of open sourcing. If you go to it, it's not done. It's missing critical things like authorization because it was an experiment to see, “Is this useful? Is this an idea? Do people care about it?” We're experimenting with it internally and making it clear to others within the company. We’re trying to demonstrate that you don't have to have a fully baked production-grade, a million queries per second piece of software to get it open-sourced. Open the open has benefits to see if anyone cares. No one seemed to care about that, which is fine. We didn't invest in it and moved on.

‍

That's interesting that you say that because I’ve noticed, especially with projects that have been open-sourced by larger companies, tend to have a lot of polish at version 0.1. At the agency where I was working, we loved React Native. We got excited about React Native back when it was an iOS project. Facebook was working on the Android one.

It took them ages to get it out because they were like, “It's not ready yet.” They didn't want 1,000 useless bug reports for stuff they knew was already broken. I don’t know if it's a cultural thing or maybe people or companies are worried that their code looks half-baked. Most codes are half-baked. It's very unusual. It’s almost unheard of to find unbaked code.

Done open-source is a myth. It's never done. In my experience, you either end up with source code that is deeply entwined into what production means, and a bunch of things get added to when something is ready for production. On the flip side, you have a toy system that will then grow up to be at some project. Both of those are fine things to open-source but they involve different things. If you were to grow a toy project in the open, we could build a community in the open-source world around it. If you show up with a fully productized system, it's difficult to get people involved because the point of entry is so much harder.

Done open source is a myth. It's never done.

For me, it depends on what the goals of open sourcing the thing is. We had a notion on the open-source team at eBay around strategic versus not. We have a lot of projects that we would consider not strategic. GOOSE, being an example, is the thing that I released. There are others that are important to how we do our business. Some of those are probably going to be released later. I won't say too much because it's their news to share. OpenFeature would be another example of something that's strategic. There's a large number of people who are working with OpenFeature and are building tools to support it internally that makes sense for us to contribute. There are contributors from eBay that have come on since I left.

Are there strict criteria around this thing in terms of that process? I’m hypothesizing but it is much more common for big tech to open-source stuff. It doesn't matter if you're Netflix, eBay, or Microsoft. Think of Netflix. They've pushed out tons of stuff that huge amounts of engineering effort have gone into. Your career over the last several years, did you feel like there was an inflection point where it went from being a struggle to try and get this stuff out of it or being a reflection of CIOs or people saying, “What can we open-source?” I’m wondering what caused that cultural shift. Is it about hiring talent and wanting to attract people who respect the company because they're open-sourcing stuff?

Open-source is a way to reach an audience or a hiring pool you're interested in. A lot of companies treat it that way or recognize the benefit of open-sourcing their internal tech because if you can get an open-source community involved in it, which is a lot of work, then people show up knowing how to use your software. Your internal proprietary crappy software, people know. That's useful, not to mention the benefits of free bug fixes and things of that nature.

Open source is a way to reach an audience and a hiring pool that you're interested in.

Over the past decade or so, open-source has increased in prominence in both its consumption and the publishing of it. The one thing I’ve not seen is a similar increase in contributions. A lot of companies are excited to release their shiny new project and take your shiny new project and use it for their purposes. I see a lot less involvement in big companies contributing to existing products or projects.

The internal process to do that is difficult sometimes like, “I want to release this three-line change. I want to fix this bug in an open-source project.” You need to file a ticket and legal needs to review it. Comms needs to review it to make sure that we're not going to be saying things that our publishing of this change does not suggest something about our company that we don't want to be said. It's a pretty onerous process. I went through that at both Google and Amazon.

Is it for a trivial pull request?

Big companies are optimized around, “I want to get involved in the Linux kernel. I will be contributing there for the next six years.” Not that they're drive-by pull requests across fifteen different repositories because I noticed something. That is not a thing that big companies do very well. It’s because they have the stuff to manage that they need to know.

It is a shame because the fifteen drive-by pull requests, to a large degree, are the lifeblood of a lot of projects, progress, fixing bugs, and improving the quality of those projects. In terms of GOOSE, for example, what was the process in terms of the internal due diligence that you guys or the organization had to go through before they were happy to press the published button on GitHub?

eBay purposefully had a very lightweight process. I’ve gone through a bunch of bad processes and I wanted one that was a little nicer. eBay is quite nice. You file a ticket and we run security analysis tools against it to make sure we're not shipping malicious code. There's a review of the licenses that are used by that software to be sure that you're not releasing something as MIT that then has AGPL dependencies. Make sure you have a good ReadMe and hit the go button. That's it. It's a lightweight process. Tickets can move through the queue in a day.

Unless there's a finding, then there's back and forth, compared to some previous organizations I’ve been involved in where it’s weeks of process. I was involved with the TODO group, which is Linux’s Open-Source Program Office foundation. There are a handful of people there from companies in the financial sector or Wall Street and hearing their process. They have a pretty heavyweight process, including trademark, making sure they're not infringing on any trademarks that anyone has by the name of their open-source project. It’s a litany of other things that they concern themselves with.

That process for GOOSE, was that weeks rather than months?

It’s days. Do you have a ReadMe? Is it good? The license is okay, which we had scripts for. Does it pass security analysis, which we had scripts for?

It sounds quite like a production line of openness.

We wrote up that whole process. If you go to OpenSource.eBay.com, the whole how to run eBay's open-source program is written there. There's not a lot of theoretical how we think about open-source stuff there. It’s a lot of like, “Do you want to release something? Click this button and run this script.” All is there.

I’m hypothesizing. I’m assuming that the majority of those projects didn't have large amounts of community involvement, traction, and things like that. Maybe some did. What was the policy? How did the organization manage that? Google prints money. They've got a lot of engineers who know what they're doing. I’m always curious about Kubernetes. It must be this interesting dynamic whereby someone suddenly looks at P&L and sees that Kubernetes has had seven engineers working on it full time or however many it is.

I don't know how many it is. Maybe it's something like that number. Someone is going to go, “Are we getting seven engineers worth the value out of that being open?” How did eBay deal in terms of if you release it and then six months later, people are using it, filing bug reports, and folk needs to be there to drive that project? Did that happen often or did they have a policy around that?

I wouldn't say that eBay had a lot of breakout successes like Kubernetes. There were two big projects that were in eBay's open-source world. One was called Marko.js, which is a JavaScript web framework that focuses on server-side rendering and fast rehydration. That was donated to the OpenJS Foundation a handful of years ago. It predates me. There are a lot of folks that use that.

There's a Discord community for support and there are engineers who are full-time on it. There was also an implementation of the Raft protocol that was quite popular. That was handled by a couple of people on eBay's database team. eBay has a couple of custom databases. They maintain it and deal with bug reports. It was honestly a pretty hands-off process from an open-source management standpoint. It was up to the people who were passionate about the project to be good stewards. That's something that could be improved.

Having a more systematic way to steward a project and having organizational support would be beneficial but that wasn't the world that we had at the moment. It hadn't reached that point. That's how eBay's approach was. It’s like, “Do you care about this open-source project? Great, then be the community manager and go get people to use it.” There was work as I was leaving to change that approach to get more support from the open-source program on these sorts of things.

Do you have any thoughts on what it is within those organizations that causes them to have different approaches from each other?

It's what kind of behaviors they're trying to elicit. For me, drive-by pull requests are important. I make a lot of contributions that way. That's not necessarily corporate. The CTO doesn't necessarily care about how we want to run open-source. The question becomes, do we want to train people on internal tech? Do we want more users so that we get better bug reports? Do we want to show off how cool our stuff is? Do we want to fundamentally change the way the industry operates in some sectors? Those are all different outcomes we could have with an open-source project.

Depending on which one you want but if you want to show off cool stuff, you're probably not going to put a lot of effort into optimizing the onesie-twosie pull request thing. You're going to do a lot more like, “We make deep contributions to Maven and the Linux kernel.” If you're wanting to be visionary in your open-source in the way that Google is in a lot of theirs, they have a different approach to how they want to do this. I’m sure there's support internally around it. You don't make a Kubernetes and have it be a thing. You need a team of folks to make that work.

Kubernetes is such an absolute monster. It's such an outlier as well in so many ways in terms of its size, dominance, and popularity. To me, it feels like there's a long tail. When you're writing software, you quite often find random libraries that have been written by some random tech company. It's easy to get sidetracked by Kubernetes in terms of that project as a thing. You're going to work for a smaller company. You're still heavily involved in OpenFeature both as a contributor and on the governance board. What's going to be your approach in that regard? Do you think you're going to do more work in your time? How are you going to balance that?

In my job search, I prioritized companies that had an open-source ethos. The company I’m going subconscious networks has made a pretty big bet on open-source. They're building this open knowledge graph. Think of it like a social Wikipedia or a federated Wikipedia where everyone gets their own Wiki and then interlinks nicely. They're building that but it's on top of a protocol called News Sphere. They've released News Sphere as this Rust thing with a bunch of bindings to Swift, JavaScript, and everything else. That open-source protocol layer is something that they're interested in and passionate about and is important for their company.

You purposefully filtered on organizations that would accept and encourage you to work on other projects.

One of the nice things about being at a small startup is you talk to the person.

There was this shift in companies putting out large successful projects like React or Next.js. There are dozens of them. Do you think we're more of a resting point for those larger organizations in terms of their approach to open-source and their involvement or it's still developing and building?

It's still developing and building. We may see a temporary slowdown, given the economics of the season. Especially as newer companies come up, a lot of companies are using open-source as a wedge for how they build their business or grew up in a world where GitHub has existed, using open-source as the way we do business. That's a big departure from the early 2000s when it wasn't quite the same. Open-source was much less known. We had poorer tools for dealing with it. A mixture of the company makeups is different in terms of how they feel about open-source, especially this new crop of companies coming up. Open source is being viewed as a competitive edge.

It's interesting you mentioned tooling because I’m trying to remember the pre-Git world or the pre-distributed version control system world. I always prefer mercurial, for the record. I still don't understand Git UI. Back before those tools, there was subversion and source forge. The pull request hadn't been invented yet. I’m trying to remember what the process was for contributing code in a subversion source forge world. It needed to have permission to push because it was a single repository. There were email patches but they were pretty unwieldy from a UI point of view as well.

There's a company called sourcehut that does email patches. They're Git forge and email patches. It is how they do poll requests and they're trying to take it back to old school, which is interesting. Lightweight branches are a feature of Git and those didn't exist at subversion. My memory was if you wanted to commit code, you got on the maintainer list. There was no code review per se. That's my memory.

The tooling aspect is something that comes up a lot on the show and how much that has changed things. It is an interesting point you make about the culture of engineers who don't recall a world. I’m old enough to recall a world before source code control. I started my career and there wasn't anything around. There were a couple of tools but they weren't commonly used. Visual SourceSafe was one of the first.

It's interesting to think about because that's a generation of people that are going to have their approaches changed by the fact that this stuff exists. I’ve felt that sometimes, especially with an OpenFeature, there's tooling left to be discovered or invented around working on projects with more of a structure around design, strategy, and things like that. What do you think about that?

For sure there's a lot of room for tooling. I’m also a tooling nerd and I love to optimize processes. OpenFeature has a Google summary code person whose whole job is, “I have this idea pipeline better.” This project is a little rare in open-source projects in that. There's specification work that needs to be done, which is not very common in open-source so there's a lot of consideration to be done. From a tooling perspective, there's always more tooling to write because we can always do our job a little bit better than we did yesterday. Wouldn't it be helpful if we didn't have to think too hard to make it better? That's where tooling comes in from my perspective.

What do you see yourself in terms of that open-source world over the next few years? Do you think there's going to be any more paradigm shifts or big changes?

We're already seeing Rust take over a lot of what JavaScript was doing. Over the past few years, I’ve seen this massive proliferation of putting JavaScript everywhere. Now, there’s a current of people who are like, “What if we took that good enough tool that we wrote in JavaScript, invested, and made that into Rust so it's fast and useful?” That's going to continue. The Rust ecosystem will improve, develop, and get a bunch of new users who are lower-skilled. They won't have that systems programming skillset. The tooling will necessarily need to get simpler and easier. That's great for everybody. We'll see tooling get rewritten in Rust. We'll continue to see that. It's a big one in my mind.

The Rust compiler is pretty famous. A lot of designers have been put into it around the ergonomics of spitting out useful error messages. It is interesting what you say about that. I hadn't considered that. JavaScript is a pretty complex language as well. There are a lot of guns hanging around aimed down at feet. The mental model of JavaScript, I always find it quite complicated and hard to reason with. I feel like it has made some good design decisions around things like packaging and releases. When you look at JavaScript, everything is on fire.

Each community has the way that they want to do. Some are faster and looser. Some are a little more reasoned about. Rust gets the benefit of hindsight. It existed when they wrote it. They got to see, “That's how people are doing in package management.” I’d probably call Python my favorite language but the package management situation there has always been a little dire. Even in 2023, it's a little dire.

Flagsmith got a pull request that's been sitting there for a couple of months. It was contributed by someone outside of the company to migrate from pure pip to poetry. There are constant deliberations amongst a small team about what we are doing here. Everyone is terrified about making a decision. I’m always amazed at how much Java got it right way back around that. Packaging software wasn't even a thing when they designed that whole area of the language.

It was their whole goal. Packaging was so critical to the design of the language and the intent of the language that JARs are pretty good.

I was a visual C++ developer before I was a Java developer. It’s moving to a world where it was whatever was in the folder and that was it. It's amazing how many languages have got that badly wrong, even after someone invented JARs and a central repository. Justin, that's been super interesting. I feel like I’ve got a much better understanding of the hopes and fears and pulls and pushes from these larger companies. Thanks so much for your time.

Thanks for having me.

‍

Important Links

• Justin Abrahms – LinkedIn
• OpenFeature
• Flagsmith OpenFeature provider
• GOOSE
• OpenSource.eBay.com
• Marko.js
• OpenJS Foundation
• sourcehut