Ee Durbin is Director of Infrastructure at the Python Software Foundation, overseeing a host of internal and external tools including PyPI and Python.org. For the last 3 years, Ee was US Chair of PyCon and has organized the Cleveland Python User Group since 2014.
This podcast does not have any ads or sponsors. To support the show, please consider visiting LearnDjango.com, Button, or Django News.
Carlton Gibson 0:05
Hi, welcome to another episode of Django Chat, a fortnightly podcast on the Django web framework. I'm Carlton Gibson joined as ever by Will Vincent. Hello Will!
Will Vincent 0:13
Hi, Carlton
Carlton Gibson 0:14
Hi Will. And this week, we've got a special guest, he Ee Durbin. Hello Ee.
Ee Durbin 0:19
Oh, it's, it's very good to be here.
Carlton Gibson 0:21
Good to have you. So he I'm almost lost to words now to introduce you because you're kind of like a, one of these figures in the in the Python community that I've seen and been like, you know, major, major cornerstone of everything that goes on. So that's on
Will Vincent 0:39
most blushing infrastructure at Python. Right. That's the quick take. So that means pi pi python.org. We'll get into all that. That's a lot of what you do. Pretty much one that you could do tell us how single handedly that is. But my sense is you drive a lot of that. And that's, that's truly internet scale. We often joke about Django and sites trying to do things internet scale, but, you know, running pi pi, for example, that is truly internet scale.
Ee Durbin 1:06
Yeah. Well, I am, it is it is a portion of my full time job. And that is supporting, you know, a pretty large swath of volunteers that that are also taking part but Yeah, we do. There are three people who are really, really, really actively involved in pi pi itself. So it is it doesn't have backstopping on me, you know, occasionally.
Will Vincent 1:33
Very modest. Yeah. Carlton, sort of like you with Django and releases.
Carlton Gibson 1:36
No, no, that's Marius and releases. He's the he's the bankstone.
Will Vincent 1:41
Yeah, we're recording this the day Jenga 3.2 came out. So that's the I saw
Ee Durbin 1:46
By the way, congratulations.
Carlton Gibson 1:48
Yeah. So yeah, I
Will Vincent 1:49
see. It's Hacker News. You know, they had their annual Oh, gencos exists. And it's cool. And I filled out
Carlton Gibson 1:55
the comments on that thread, and it was the first couple of comments, and then you get into Oh, you know, I like flask. And then it's like, No, you mustn't use flask, you must use this.
Will Vincent 2:04
Micro frameworks, and then you get into JavaScript is better. And then you get into rust. And it's
Carlton Gibson 2:08
like, why did I look at the comments?
Ee Durbin 2:10
I'm very lucky to have a friends who look at that website for me and helped me know what I need to actually go and read. So I try to stay away.
Will Vincent 2:21
How did Oh, I, I don't have an account anymore. I I only glance once a week now at this point, it is it sort of. I mean, that goes for social media for me in general to I've pulled away because I take a glance, but that feels okay, but anything more than a couple minutes, and
Ee Durbin 2:38
it's not good. Once in a while there's a there's something posted there that actually gets any amount of praise. And most recently, it was it was actually related to peipsi. And it was when we rolled out the implementation of their token token scanning. So basically, if you accidentally upload one of your API tokens to GitHub, pi pi will now revoke it almost immediately. And the people on Hacker News seem to like that. So it was it was oddly satisfying to see pi pi, and not something relating to all manner of, of angry thoughts.
Carlton Gibson 3:16
So I've got this terrible thing come because now because when I was uploading Django 3.2, today, and I, you know, twine upload, and it goes up, and it's like, legacy dot something, I'm like, I've got about data setting somewhere. Now I've got you in front of me, I'm gonna be like, God, can you open up my config and help me sort it out?
Ee Durbin 3:34
I'm hoping that was probably twine or setup tools, whatever using upload.
Carlton Gibson 3:40
So but it was saved using the legacy endpoint, and I must go into the, you know, the docs and find the, you know, find the what's the what URL string? Do I need to change to get it to use them? The latest one? Because, yeah,
Ee Durbin 3:54
otherwise, it's actually the same. It's actually the same implementation. So the reason that warning is there, if so, eventually, when we do change the implementation, we can use a different URL. So at the very least, you're not missing out on any features. You're just talking to the wrong URL.
Carlton Gibson 4:13
Yeah, well, that's fine. I think I my feeling at the time was, oh, I must address that. But be if it was serious, it would be in a different color.
Ee Durbin 4:23
If you're serious, it would just give you an error and say it's too late. You've waited too long.
Carlton Gibson 4:29
Yeah. I'll fix it then.
Will Vincent 4:32
So a lot of podcasts don't go deep on technical stuff. But since the three of us who we are I really want to do that today. But maybe we quickly get your your backstory. So how does one get to this position with the Python organization? You were telling us? You're based in Cleveland, I believe you you went to school there you from there as well.
Ee Durbin 4:47
Originally, I grew up in southwest Ohio, and I came to Cleveland for school. And then I briefly moved to San Diego after school because it was one of the only jobs Second find that would help me pay for those dastardly student loans. But I went to school for physics and math and graduated into a recession that we're all pretty familiar with. And the junkyard job that I had for the first like, six months after I graduated, wasn't paying for things. And I took a job with an appraisal management company. And I was just supposed to do like magical spreadsheet things for like their business reporting. Because you know, physics and math, yeah, I wasn't very good at it. And pretty quickly ended up just just through, you know, hobby type interactions with computers over the years, and just being decent with configuring them. And such, I ended up sort of a sysadmin for, you know, sort of a standard cubicle farm type thing. And that company had like an in house programming startup that was doing pretty neat piece of software. And so the lead for that looked at me and said, You underscore physics and math. I said, Yes. And so Kanan said, well, that you can learn PHP. So we'll just go ahead and have you start programming with us, since you're kind of done with automating all the workstations and stuff. And my the first time ever Python was at that job. And it was because I searched for how to programmatically access stored procedures from my SQL database. And lo and behold, I ended up on peipsi, looking at the MySQL library, and I was like, this is the first thing I found that can connect to MySQL, I might as well use it. And that's when I wrote Python for the first time. So I eventually desperately wanted to come back to Cleveland, I got a job here, doing sort of Linux sysadmin type things, you know, and continuing to write Python to automate stuff. And through a local community, and like the local Python user group, I was introduced to an opportunity to volunteer, I'm sorry, to do part time work as a sysadmin, for the PSF. I took that part time job. And that's how I got introduced to like, you know, pi, pi, and Python and Oregon such pretty much immediately quit billing. And, like, submitted almost no invoices, and pretty quickly turned into mostly just a volunteer at the infrastructure, and did that for a number of years. And, you know, that is really, to me, how I got introduced to so many people in the community is, you know, as volunteering on the infrastructure, which covers, you know, all manner of technical concerns for the community, as well as community concerns, you know, so I would interact with the Python staff to help get the Python site going and things like that. And then just, just get at it. And I mean, there's, there's ton of wonderful people that you know, all have a pretty common direction and goal. And they tend to be kind, and the community shows a lot of what, in my opinion, the best of the software industry, or at least aspires to it. And so I kept on with that. And then years later, there became an opportunity. And I was offered a job at the PSF to take on the infrastructure role. And I've been there. I've been here I guess, for it's either two or three years. And I should probably know, it'll be three years, three years in June.
Will Vincent 8:46
Wow. We had Eva on last week to talk about her own progression from part time to full time and, and sort of that side of things. It's sort of wild to think that the PSF was ever small, because you know, Django land the Django Software Foundation, we look at Python and Python Software Foundation is this Godzilla of a thing? But that's really pretty recent, that outgrew the volunteer realm. And yeah, I
Ee Durbin 9:12
mean, I, I personally, remember when the PSF really was just sort of Eva as far as full time roles go. And so it has been interesting myself to watch it grow. And then, you know, we just, we just had a new a new person start recently. And so my eyes light up, because I'm learning so much about, you know, the community and nonprofit operation side of things, if you will. So, you know, I am community oriented in a lot of ways, but not necessarily in the way that the PSF is. And so it's been it's been a really amazing opportunity to stretch my legs a little bit outside of just sort of sitting at a keyboard and pounding my head against it occasionally. And sometimes it's, you know, you actually get to talk to people or You know, interact with the community in a way that is not just the software, it's been there.
Will Vincent 10:05
And you have these office hours where you've had them because I think we first spoke two years ago or something like that when I was setting up a site and Jeff Triplett recommend I get in touch with you around python.org, which is a Django site. And that was it. So I guess my question is, do you still do that is that something you did a lot of because it was very generous of you to just have open office hours office hours
Ee Durbin 10:28
were initiated during my time as the icon chair. And so this was before I was the PSF, staff member. You know, it says, and I was also a volunteer, chairing HP icon. And a big aspect of that is,
Unknown Speaker 10:45
you know,
Ee Durbin 10:45
I recognize the position that Python has in the community. But I also personally, very much recognize the immense value in regional organizing. And so the office hours were a really good opportunity for me to provide, you know, that that low, low intensity space to meet a lot of folks from, you know, different regions in different parts of the community that that sometimes can't even make it to Python in person. I kept doing them for a while. And that I think I took it down about a year ago, as the stress of the current world circumstances, just became a little bit much for me. So it's actually I appreciate the reminder, because it's something I'd probably in a much better place to put put back into
Carlton Gibson 11:36
into service. It's something I've been thinking about, because I saw some Wilson said he does. He's doing office hours for his data set project. And he's, I really recommend this I was thinking, well, maybe I could squeeze one. How did you run it? Did you do? Did you just like have a video call? Or did people book a meeting? Or did they
Ee Durbin 11:55
I used a service that allowed for integration from my calendar to a set of slots, but pre determined to be like,
Will Vincent 12:04
the I think it was was a calendly? Or that's one of those one of those
Ee Durbin 12:09
brand names, but yeah, it was cowardly. Yeah. And then
Will Vincent 12:14
I had that that's what I do, Carlton, yeah, you just say this is when and then so on. And I assume Actually, I don't know, on your end, can you sort of say no, like, What's the last minute someone can do it, right? Like, there must be some sort of way you can tell it like you can, you can only schedule
Ee Durbin 12:29
24 hours or seven days or whatever, in advance. And then my favorite part is that you can say, don't allow for somebody to book like, within 15 minutes of a previous thing, and during or before the thing. So it's really nice, because you just open up the little windows and have a Windows of time. And then it just sort of takes care of the rest. And you'll just get an email when somebody books it, and it shows up in your calendar. And, you know, it's cool, I do recommend, you know, some automation around it, because otherwise you're, you'd be doing something you'd be doing a lot of work that to offer that up, and it'd become probably more stressful than it's worth. It's, it's
Will Vincent 13:09
so funny that it works, I think really well in a nonprofit or in a volunteer setting. But on the business side, I've I've had it used in the business side and there becomes more of a, it's totally a power move. It's like who's gonna set who, you know, someone will ask something, ask for a meeting and be like, here's my dates. And it's, it's so it's so nice, actually, when it's just like, okay, like, you want something for me, like, here's my calendar, it's so much simpler than in the NBA world, which I spent some time in it, it sort of weirdly becomes this pissing match of sorts. So it's nice to hear that it doesn't have to be like that. I just have some scar tissue around. So one would send me one, I was like, What?
Ee Durbin 13:50
There were a few times where people would schedule it to try to sell me things, or convince me to do weird stuff. But like, you just you just hang up on the call and you are
Will Vincent 14:02
one click. Right. Well, so can you talk about then the so what is the scope of infrastructure? So I, I guess to me, I know pi python.org. But But there's more than that. What How do you define it? And then in terms of what takes up your time? Sure.
Ee Durbin 14:19
So the the PSF operates infrastructure, both for the community and so that would be a lot of the public facing stuff like python.org and pi bi and the wiki and the Python site and, and such. We also operate some infrastructure for our own operations. So yeah, so internal sites for membership data and things like that. And then we also in the last few years have been either providing infrastructure or collaborating or supporting infrastructure for our fiscal sponsor ease. So the PSF recently took on the ability for organizations who aren't quite big enough to be their own nonprofit, but need that structure around, you know, the, the tax incentives and such to apply to the fiscal sponsor ease. And so we have a couple of those, like, the most recent one is the Python package authority is the most recent fiscal sponsor. And that allows that will allow them to, you know, do fundraising in a nonprofit sort of way. And we've already been supporting the PI pa infrastructure, so that things like packaging, grep, python.org, and such, so it's realistically, you know, almost everything online on the python.org domain, as well as pi pi and a few other things. So, it all and then, like, really mundane things, like, who holds the registration for a given domain. So the PSF, you know, we make sure that these domains remain registered, and that there are funds to renew them for a number of boring, you know, a number of smaller projects.
Will Vincent 16:13
That all sounds very familiar.
Ee Durbin 16:17
But, you know, I guess the the less than glamorous way to describe it is, you know, the, the PSF infrastructure is realistically here to make sure that the really boring mundane things do get done. And, you know, occasionally, and it's exciting, always, of course, there are opportunities to build out new things, or improve or, you know, rebuild existing stuff. But there's a lot of running around to do to make sure that TLS certificates remain valid and
Will Vincent 16:52
such, and with volunteers, especially, right, I mean, we have that with Django, Django cons, not to mention other things. There's, you know, quite a bit of that at the board level. Yeah.
Carlton Gibson 17:03
So, but the the ops team, you know, it's so there's Maris and I, who were, you know, we contracted but the rest of it's all volunteers, and, you know, they're busy rebuilding servers, you know, on the weekend, and all the rest of it, it's like, you know,
Will Vincent 17:17
it's amazing that it works. You know, it's sort of a bunch of people who, you know, swoop, no, they need to, and then hopefully, otherwise can kind of, I mean, they all have, you know, they don't get paid for it. So, right.
Ee Durbin 17:31
Yeah. And so there's a lot of that, too, so facilitating various working groups, you know, a really active one is the documentation working group. So Doc's dot python.org. You know, there's a lot going on behind the scenes there. And, you know, realistically, you know, it's my responsibility, if infrastructure concerns of that have a problem. So whether it's the places where the code runs and such, but, you know, it is volunteers that push, basically all the forward progress that's visible, is driven by volunteers, you know, most of the time, and then the infrastructures, ideally, the thing that no one else is worried about, unless they want to be
Will Vincent 18:11
well, there's also there's the, you know, technical infrastructure, and then as you were saying, just the infrastructure of a nonprofit or community thing. I mean, we, we spoke, obviously, a lot with Eva about that, but so much of that is people don't think about in terms of, I mean, she mentioned, you know, going out for grants, and Python is built the PSF is building up the the capability to support those properly. I mean, those are things we can't do right now. And even the fundraising, Carlton spent quite a bit of time updating our stripe change their, their end of things, you know, donations to Django, it's been interesting to see GitHub sponsors, you know, GitHub, as I'm sure by the time this launches, they're going to have corporate sponsors as well. So they're making a big push there. On the one hand, it's great, because that's no easy thing to manage all that. But if it's Microsoft, managing every nonprofit in the tech space, there's, I suppose some concerns as well. But the demands are real, and largely behind the scenes around both sides of the tech side, and then just running a nonprofit with software and then there's keeping track of all the various ways that you know, contributions come in. And so that's where you know that the accounting team at the PSF has a lot of work to do because, you know, get up sponsors sounds so good on paper from a I want to give money perspective. And it sounds so good from I want to receive money perspective, but then there's the aspect of
Ee Durbin 19:43
Oh, wait, but that's a whole nother reporting pipeline for contributions and stuff. So yeah, the the infrastructure aspects like the, the non technical and infrastructure aspects are. Yeah, I mean, there's just a number of systems and all sorts of things going going on, and there's no one one. There's no, there's I guess there probably is like QuickBooks for accounting that takes care of all of that. This is where it all ends up. But it's getting there. Right?
Will Vincent 20:11
by that. I mean, yeah, it's just, it's non trivial.
Carlton Gibson 20:13
I mean, there's, that's why you all have three accountants, Carlton, you're gonna say on the GitHub front that, you know, thinking. So I'm based in Europe, and I'm in Spain called verbs, my status is autonomo. Like, is self employed. Basically, it's a particular type of tax setup. But in order to have GitHub sponsors, I'd need to be able to identify the payers and weather data and have to have a tax number and all of these things. And it's it none of that's available on its site. So literally, I can't accept GitHub sponsorship, money without breaking the the, you know, European and Spanish tax laws. And it's is like, so it's a nice idea. But the reality is, it's not compatible with you know,
Ee Durbin 21:02
yeah, moving money internationally is difficult. That's for sure. You know, we've we've been really lucky, you mentioned them earlier, but we've been really lucky to facilitate a number of grants, for improvements, and, you know, contracting, the people who are most apt to do the work has met, you know, international contractors, and that ended up itself is an amazing web of nonsense to cut through. So
Will Vincent 21:30
while we, I mean to, you know, so there's two Django fellows Carlton and Marius, and each of them gets paid a different way, in a different form manually. So I can we can only imagine the scale pythons that I hadn't I didn't think about the spit that sponsors thing, Carlton, that's, that's so you can't What if you you can't accept? Yeah, I don't know what's an equivalent, like, buy me a coffee things, you couldn't do one of those.
Carlton Gibson 21:55
Maybe you can, if you've got if they've got the right collection. So you could set the way you end up doing is to set up a stripe payment thing, and you make that you just get them to put in their actual details, and they can do it there. But like the services where they might try and make it easy, unless they're collecting the right data, or they act as the seller of record, I think it's called like the, you know, so yeah, like gumroad, does it so gumroad, you gumroad? Are the people doing the payment, and you just they pay you a royalty, and so that's fine, you can use to do that. But if they're paying you directly, you have to collect all these details. And, you know,
Will Vincent 22:32
well, I don't Yeah, 19. January, but I think that stuff is you know, Stripes, improving what it automatically handles around taxes and all the rest. I mean, it's a slow process. It's also the case that with sales tax, like on books, for example, a lot of times or digital goods, the rules only apply if you're at like a six figure or above level. I mean, they they recognize that it's impossible to enforce. So, you know, the Knights in the United States, you could argue that every state tries to get their hands on digital transaction in some way. But, you know, it's basically I mean, I spent quite a bit of time trying to figure this out. There's whole services that will calculate online sales tax for you, and it's impossible to do. But it's largely, you know, only for transactions, I believe it's generally high six, seven figures before they even say you're supposed to try to do these things, because it's nobody can. But anyway, it's a real concern.
Carlton Gibson 23:28
It's difficult, right? So I've
Ee Durbin 23:29
tried to still try to stay away from it.
Carlton Gibson 23:33
So I wanted to talk about pi pi, right? So I go pip install Django, and it goes and fetches it, and how on earth does that work? Because there's not like, a, you know, I know there's a server and it serves the files, but I mean, you know, in terms of that works reliably for how many people per day and, you know, how is that grown up and Hanuman
Ee Durbin 23:56
now this is, this is a place where I'm always happy to plug realistically peipsi exists as reliably as it does because of fastly, our CDN provider fastly is extremely generous in the the amount of traffic that they handle for us. So this is on the petabyte scale per month now. And so, you know, pipelines backends are by no means small, but relative to the amount of traffic we do at the edge, they are they are miniscule. So, for 9097 plus percent of PIP interactions with with with pi pi, you are probably get receiving a page from the simple index from fastly, and then receiving like in their cache, and then receiving the actual file itself, especially for a project like Django from the cache, at the edge, behind the scenes. You know, we're also really fortunate To have amazing infrastructure support from Amazon Web Services and Google Cloud Platform. So the file, the files themselves are stored in Google Cloud objects, access logs, and not necessarily access logs, I guess a better way to put that is access statistics are stored in Google BigQuery. So you can go find out all sorts of amazing stuff about who's been downloading Django from BigQuery. Now.
Carlton Gibson 25:29
And that's accurate, because the few a couple of years ago, I remember trying to look into this and this data wasn't really available.
Ee Durbin 25:36
It's been it's been improved dramatically from work by Donald stuffed and Dustin Ingram, in line Hall is the project is that takes the actual access logs from the files, as well as access logs from the simple index and converts them into anonymized records in Big Query that are as accurate as we can possibly imagine.
Carlton Gibson 26:03
And that's aggregating from the edges from the CDN. Okay, that's fantastic.
Ee Durbin 26:08
Yeah, so simple, simple index access is probably the most accurate because you have to talk to that, within with the advent of PIP caching, the actual file downloads have been minimized some somewhat via that. So you can read more about that at packaging that python.org. If you search for like, pi, pi, access, pi, pi, you usage statistics are something
Will Vincent 26:34
I'll find it and put it in the show notes. I think I've seen that article before.
Ee Durbin 26:39
Yeah. And then we have like, I don't know, 12 to 15, decently sized DC, two instances, running Kubernetes in Amazon, that run a handful of services that compose peipsi. So there's like the main one, which is the web app that you see and interact with. And then we also have the application that handles like, turning are enforcing TLS for all images that are accessed. And then we have another service that does URL. Magic, I guess, I don't know how to describe it. It makes URLs work, if you try to access a URL for a package file, in a way this sort of
Will Vincent 27:29
says probably a dictionary of some kind of match those up.
Ee Durbin 27:32
So it says he conveyor does, it also serves those old like, Doc stop python hosted.org. And that, then it turns the way that py files used to be stored, where like, slash capital Burke slash first letter, slash project name, slash, and then a bunch of files. And so you used to be able to reliably say, like, I want to go get this file. I don't even know its name. But I know that it's the project at this version. And so you just guess, because of the scale of peipsi, that that had to change, because like s3 buckets, and Google Cloud buckets and stuff don't deal with these, like, these, these sorts of or didn't used to deal with these sorts of files, file hierarchies. So now it's like slash packages slash, like a bunch of hash gibberish, and then the actual package itself. So that, you know, you kind of want people to be able to access the file the way they used to. So conveyor exists for that purpose as well. Yeah, and then we use pretty, we try to maintain, I think a big reason why pi pi has done as well as it has growing is right off the bat, fastly, that takes a lot of problems off of our shoulders. As far as just the sheer volume. And then otherwise, we've made really boring choices around technology for how to get to where we are. And so we're still just using, you know, a moderate, moderately powerful Postgres instance and some Redis and, you know, unfortunately, not Django, but a very Django inspired implementation of pyramid Donald has often said that he accidentally built like, you know, a bunch of what he knew from Django into warehouse to make pyramid work more like Django so yeah, it and then just g unicorn and nginx and
Carlton Gibson 29:34
keep it simple sort of all works.
Ee Durbin 29:38
We've done a lot of learning on how to push What's there to to the scale as well and you know, things break, they will always continue to break but simple choices and try to keep trying to keep our footprint small.
Will Vincent 29:55
Can you talk about the growth, I mean, I maybe it's in that where you referenced, I've seen the chart have, you know, just downloads from pi pi in the last couple of years? And it's almost exponential?
Carlton Gibson 30:06
And how much of that is ci? Is it? Is it people
Will Vincent 30:10
that's Carlton's thing?
Carlton Gibson 30:11
Yeah, nobody is Python that much more popular because it is much more popular. What was it like? You know that we're all running 58? Get home action jobs every time we Yeah,
Ee Durbin 30:19
I'm not, I'm not. I guess I've never, I've never seen an analysis. And I've never, like been able to perform an analysis to pinpoint one thing. And it also depends on what we what we talked about, we talked about growth. So if we just look at requests to the service, or if we just look at bandwidth, or if we just look at number of projects and stuff, you know, there's sort of different axes, right? When it comes to bandwidth, our biggest bandwidth growth has been in data science and machine learning, projects and the binary wheels that support, you know, computing on GPUs, right? Like that's contributed massively to our bandwidth. When we look at requests, I would assume that like request to the service, a lot of that is driven by ci, you know, of course, some of the bandwidth probably is because caching and ci is a unsolved problem, it would seem. Yeah. And then you know, what one of my favorite one of my favorite metrics to look at is new project registration. And so, you know, we that, that that growth, I think, is probably the closest you could get to the popularity of the language, right. So the, the more people are publishing new and novel projects, the more the, I think that that that shows a healthy ecosystem. So you know, that that growth has been pretty steady, all things considered and accelerated significantly, dude, like cogeneration things, right? Yeah. So you know, some, some people like that one of their one repository might create hundreds of sub projects. Right. But that's, that's not super common. Yeah, and But yeah, I think data science and machine learning have been a, you know, the most obvious growth aspect for both Python and pi pi in my in my assessment.
Carlton Gibson 32:17
Okay, and talking about registering new projects. One thing that was crossing my mind, in the last week or so I was about namespaces. And like, the sort of name collisions on pipes, is that a big issue? Or is there moves in the ground to take, for instance, the swift ecosystem was they they they do it by URLs? So you've got your, your your repo URL? It's, it's namespace per user kind of thing? Sure.
Ee Durbin 32:45
So I it the namespace issue has a couple of implications. As far as like, when I hear that, when I hear when I hear that word, it's like it's bringing some bells and turning some lights on. So one of the most recent conversations on namespacing has been around some of the, you know, the published. I don't know if he did either of you see the article about like, dependency confusion?
Carlton Gibson 33:11
Yeah, there was a thing about poisoning dependencies and yeah, packages, which perhaps malicious name very similar to existing packages.
Ee Durbin 33:21
So yeah, or named precisely like an internal package that isn't actually published to the
Carlton Gibson 33:26
Oh, yes.
Will Vincent 33:27
Yeah.
Carlton Gibson 33:28
That's Oh, yes.
Ee Durbin 33:30
So you know, the, it's interesting, because you see these wonderfully named projects pop up on the index, and you're like, that wasn't taken How? So? You know, we're working with arbitrary like strings of characters. And so the human mind will always come up with things like, you know, to fill out that, that space, I think, so from from, like, exhausting the list of usable names on pi pi, I don't think namespacing is necessarily a big concern. But on these other aspects of, you know, provenance of a project validity of a project and safely managing installations at clients, I think namespacing has a lot, a lot of opportunity to help ease some of these these concerns. And it is something that, you know, we're we there's a there's an open issue on the on the, on the GitHub tracker for it, and it's something that, you know, we're really interested in pursuing and are hoping to be able to do so you know, rather soon. So there's some really exciting announcements that do pertain to pi pi and packaging that are going to be coming through I think, probably in the next week or so. So definitely recommend checking out the Python Software Foundation blog at pi found@blogspot.com.
Will Vincent 35:01
Yeah, I haven't even hinted at those as well. So, in fact, I think it'll be out by the time this airs. So I'll put a link to the PI NASA.
Ee Durbin 35:08
So, you know, I think with with with with time, we'll see a lot of these things develop. And we're really excited to have the chance to pursue like things of that scale. That's a, that's a rather big change the ecosystem, that's a rather big change to the service. But we're excited the opportunity to be able to pursue those more consistently. So pi pi itself has seen a large amount of investment over the years, you know, Mozilla funded, the Mozilla Foundation fund funded. The launch of the rewrite of peipsi. Open technology fund, funded some really big security improvements and user experience and air internationalization improvements to ipi. The Chan Zuckerberg initiative, and Mozilla again, funded a bunch of work on PIP last year. But those as amazing as the outcomes are, it does stink, because it's just like, we get all of this done. And then there's no more money or there's no more like, there's no more roadmap for what's what's to be done next, and Serbs, people sort of dispersed back to what they were doing.
Will Vincent 36:26
Or hope, right, the maintenance is less fun than the ever, you know, the Greenfield
Ee Durbin 36:30
Sure. So we're really excited to hopefully have, you know, improvements on that story, not just for pi
Will Vincent 36:36
API, but
Ee Durbin 36:37
for Python packaging more generally. So
Carlton Gibson 36:40
yeah, I mean, it's hard to create a roadmap when it's like, you know, this. This is, well, there's a bit of a, there's a bit of momentum here. And then Okay, well, let's finish what we do. Oh, yeah. As you say, wander off and go back to what you do.
Ee Durbin 36:54
Yeah, it's not commented, somebody just sits down and says, I want to go through a month long project is a volunteer to add a feature to a service, right. That's a that's a huge commitment as a volunteer. Yeah.
Will Vincent 37:05
Yeah. So you mentioned Pip, I want to ask you, so installing Python? I guess first just personally, if you had a new computer, what what operating system? Do you like to work? So you personally desktop
Ee Durbin 37:17
is Mac OS? Okay.
Will Vincent 37:20
How do you? What do you what do you use Pip pip m, like, you know, if you have, so if you have a, you bought a brand new Mac, and you're just going to install it from scratch? How would you do that? Because I was just updating my books on Django and being like, Okay, here we go. Like, you know, there's four or five different ways to install Python. I'm curious what you what you personally do, and then what you recommend, if you get asked, it's gonna
Ee Durbin 37:42
sound like propaganda, but it's not I have been using the installers from python.org as long as I can remember.
Will Vincent 37:50
It is well, that's, I think that's now the, you know, certainly on Mac homebrew has its issues, Carlton and I were texting about this earlier. So that, you know, for someone who knows what they're doing the installers, but it's not even on both Windows and Mac would be the way to just don't even know what you're doing. It's download it's double click the dmg file, it's run the installer. It's like it's a fully packaged installer for you same on pies for Carlton everyone, by the time they come to you has clicked around and half install away. Yeah, okay. And your path is messed up. And they're like, how do I fix my path?
Ee Durbin 38:21
Yeah, so I mean, my, my setup is this, I, you know, I still work on projects that use various Python language, or Python versions, I still have to worry about that. And so for a lot of environments, what I'm doing is a brand new computer gets like the latest two, seven, the latest, like 363738392, and 310. All installed, I set up a, I choose a choose an arbitrary Python three to be liked by my main environment. That gets a pip install dash dash user, virtual env, then talks and a few other things. And then, as I did the other pythons, I create virtual environments for them explicitly, so I don't use like a helper,
Will Vincent 39:10
I probably should just do use pi m.
Ee Durbin 39:12
No, I use then built into pythons or if I'm using a Python doesn't have been built in I use virtual hug.
Will Vincent 39:22
Maybe I'm not understanding how do you within the virtual environments point to the different Python? So I guess I'm very familiar with pi m for pointing to somewhere Oh, yeah.
Ee Durbin 39:30
What when I when when you when you initiate a virtual environment, or then which is sort of like a very like, it's a very pared down implementation of virtual environment. You can specify an interpreter. And so the result that you get is a environment that once you source it or activate it, you're using that specific interpreter and you have a empty site directory or empty empty site packages. So then wherever you reactivate that, no matter what Python interpreter, you're coming from. Shell whenever you reactivate that virtual environment, you will get that same interpreter version and
Will Vincent 40:05
whatever dependencies installed, do you use Do you? Do you do much with Docker in terms of like team settings?
Ee Durbin 40:12
Yeah, we use a lot of Docker compose and such for not just pi, but also for like some of the Python infrastructure as well. You know, it. It, it just alleviates so much of the concerns of reproducible environments. But the biggest pain point that I have had with Docker based stuff is it is really not kind to contributors who don't have monster computers. And so that's, that's an, that's an open problem in my book around like, how do you?
Will Vincent 40:50
How do you? How do you help you need 16 gigs of RAM, basically to run anything? Yeah,
Ee Durbin 40:56
It's troublesome. And so in the past, we've done Sprint's on pi pi, like Python, or just what I've been trying to, like, collaborate with folks elsewhere. You know, there are people who are very excited and motivated to contribute, but are almost literaria, literally unable because their computer can't run Docker, or because he doesn't have enough resources. So I don't know the answer, though. If anybody does, please let me know,
Carlton Gibson 41:24
I think that's a big issue is that there are just so many millions of people out there, they've got a computer that's 234, or five years old, that it's got some version of Python installed on it. And, you know, they don't necessarily know what and they don't necessarily have the technical chops to download the latest one and update and you still want to be able to say, look, you can get started, you can work you can do and you know, and they'll learn the hardest stuff as they get on. But if the barrier to entry is you've got to have the latest shiny, and it's got to be maxed out to the top. That that's not what it's about.
Will Vincent 41:59
Yeah, yeah, I want to I mean, there's, so I run this site, install Python three.com. To try to sort of help Oh, go Yeah, go look at it. It does. It's not aesthetically amazing. But I'm about to update it, again for this very problem. Because there's so many questions. And I think the Chromebook section is still on there. So I was gonna say, so Chromebooks, I looked at this, this fall in detail, you can, they're adding the ability to basically run Linux on them. And you can, you can install Python, and you can kind of get there. But I put some time into it. And I could set up a dev workflow, but it was pretty hacky. But I think that's changing with like, Visual Studio code, and some of these things will run in the cloud. So in terms of access, it's exciting to me that I think they're getting there with Chromebook, you can do it, I mean, I have a guide up there, that does work. But if you go beyond installing Python, like, you know, doing a Django project, it was hard for me to recommend it to someone else, I could do it, but it was pretty painful. But it was I'm updating that site. Because on that site. Now, I want to, you know, as I said, Do that update where I believe you, if you're a beginner with Windows, use the you know, use the Microsoft App Store, otherwise direct, I'm going to update the homebrew section. But I would love to have some sort of section of you know, there's probably only a dozen different ways to screw it up. But it's just it's the path variables is people just they don't know where Python is located. They don't know they don't understand the command line. And striking that balance of helping them without completely overwhelming them where they you know, can't get a foot in the door and the
Ee Durbin 43:36
the Microsoft App Store, those those installers are the same as download are not the same as but they're packaged by the same people who back into the Python org installers. So I'm glad to see that maybe
Will Vincent 43:48
they're getting there's only was it's just the temp drive, there's there's just a one or two things that are they don't have a full read, write access to everything.
Ee Durbin 43:57
I'm not super sure about that.
Will Vincent 43:58
So I wonder if they're gonna solve that. But I
Ee Durbin 44:00
am. But I was excited on install Python three.com. Because you recommend dead snakes. And I don't know if you listen to Anthony, but I just want to give you a huge shout out. It's like the most appreciated consistent effort in packaging Python for at least the Abood to Debbie and flavors of Linux. And I generally just reach for that I used to be a little bit more like the Red Hat side of things. And
Will Vincent 44:25
now I'm just saying I do recommend dead snakes. This is Yeah, this is something again, I try not to look at all the time, but every six months, I'm like okay, what's let me put my hat on huge shout out to Anthony. So yeah, well, cuz I think this is the problem is that. I mean, I was just looking at like real Python has probably outside of the python.org the most popular guide and that does quite a good job. But the issue is I want install Python, it's like well, there's five different ways and then you know, scroll down. It's like, it's almost like a recipes app. We have to scroll down all the way, you know, versus, you know, install Python three is just like okay, if you're googling around for this, you don't know What you're doing, you know, you can sort of trust me this will get it going. And then you can dive into these debates later on. Trying to take a non engineer approach to the issue basically. Anyways, yeah. That's always something I like to ask people. I thought you'd be on like Linux or something. No, I
Ee Durbin 45:16
so my biggest thing was, I think the first time I got a Mac laptop for, for work, I was suddenly like, I felt relieved of a lot of distraction around that there's a certain amount of like, hubris in younger software developers, right? where it's like, oh, this is software, I can change it. So if you're working on Linux, and you don't like something, you know that you can change it. And I fell into that trap guardrail that trap so many times. And so eventually, when I finally got like a Mac computer for for work, I was like, well, this is nice. Like, I just have to accept that this is the way this thing is. I'm also a big fan of defaults. And so you know, I like my vim RC has one line in it. It's like syntax color. And like, my bash profile has, like, almost nothing in it. And like, you know, I just yeah, this it's so it's like, it's, it's, it's it's okay, it's not perfect. And I and people get frustrated when they see that you don't even have like tab expansion open for them or something, right. I'm like, Well, I just press the spacebar four times and like this. So yeah, it's okay. And so yeah, guardrails is a good way to put it. But yeah, just not just accepting like, well, I can't change the way Apple decided to make this decision. So forward,
Will Vincent 46:46
yeah. Well, literally this morning, or this afternoon for Carlton. I was texting him because I'm, I go for my books, I've got these projects, and I go go through everything on Mac and on Windows. And my Windows machine is it's old and bad. Anyway, so typing on it, I just feel kind of angry. And so I have I do have a little bit of a bias against windows. And I've been told by many people that it's gotten better. And I think maybe if I have a better machine, I'll, you know, I won't just like ratchet up my stress every time I have to log into it, because most people are on. I mean, you know, the Python survey, more people are on Windows than Mac, obviously. Same with Django.
Carlton Gibson 47:24
I'm having fun with Windows for exactly the same reasons that he was saying, Mac is that you need
Will Vincent 47:30
to change it for fun. No, no.
Carlton Gibson 47:32
But like, when I pick up a Windows machine, I really don't know what I'm doing. So I can't spend time paddling around in the corners. Because I'm just like, Well, you know, I don't know what happens if I go down there. And it's all in a weird, you've got to learn this whole other shell and this other language. And I'll just do what I'm going to do. And I get more done. It's
Ee Durbin 47:51
the best part is, the best part is that somebody has already figured that out. There's somebody for whom it is it is rewarding to figure out how to accomplish the thing. And so you find them you use the reference, and then you thank them. And then you just go on with your time. Yeah,
Will Vincent 48:07
well, that's farmers and chefs. So the analogy I like to make, you know, some people love to one crop takes years and they do a really good job. And I'm definitely a chef, I just, you know, get a recipe and maybe I tweak it, but I don't want I want to go to the store and just grab stuff. And the creativity is in the combination, not in first principles.
Ee Durbin 48:25
The producing a usable first principle can take a lot of crate.
Will Vincent 48:32
For sure, it's just what creativity Do you want to do, right? I mean, like, you know, where the grain come from? I don't know, but I can make pretty good bread. So I guess the last thing I know we're coming up on time, but so Windows machines, the command line prompt like, Carlton, what is one use? Right? There's PowerShell, there's,
Carlton Gibson 48:50
if you're on Windows, use the new windows terminal app and use PowerShell.
Will Vincent 48:55
And you learn how they have a new cross platform, I guess. PowerShell is No,
Carlton Gibson 48:59
I haven't got around to installing that on that. But you can, you know, you can install PowerShell on Mac now. And you know, I might try it. And they've got the whole dotnet core thing is all you know, cross platform now it's available everywhere. So
Will Vincent 49:13
if you do if you check your Python version on a Windows machine, it will prompt you to check out the website for you know, on PowerShell. it'll prompt you to check out Yeah, I seen that this morning. There was Oh, last thoughts on Anaconda. So be what are your thoughts on that? I mean, I'll tell you I have It's fantastic. It's bundled together. I recommend people not use that. But a lot of teachers say just use that because it's it is a little bit simpler if you're starting out. So
Ee Durbin 49:44
Anaconda is ultimately solving a larger problem than then PIP does. Right. Anaconda is solving, not just how to install Python packages but also how to install all The things necessary to make that Python package work. And so it is, it is a larger problem space. And it's represented in the amount of, you know, labor and investment that has gone into creating Anaconda. So, you know, I personally have not used it much. But, you know, it does solve so many of the issues that you run into with, you know, software library requirements and dependencies that are not Python based, at the end of the day, stands for PIP installs Python packages. And that is really what what its goal is. And so until such a time where, you know, the broader Python packaging ecosystem has a solution for declaring for explicitly declaring how to install dependencies that are Python, you know, the the solution really is something akin to a combat but that when you look at the conda, forge project, which is also volunteer driven, or foundation driven, and community driven, conda forge has an immense effort to produce the results that it does. And they speak for themselves, it is truly like the easiest way to install some of the more complex projects out
Will Vincent 51:21
there. So yeah, well, and there's the mini conda flavor, too, which is, what I found was the best on a Chromebook. So it's, as you say, it's a whole ecosystem, we had the PI charm folks on a little bit recently against thinking of, you know, beyond just Python. So if you use Django with, with PI charm, it comes with a whole lot of other goodies. Yeah, and there's like, that make a lot of sense. And, yes, a lot of investment to make that happen,
Ee Durbin 51:45
it's just another shout out, there would be a long lines, like pi wheels. So I've been at all pieces, this project called pi wheels. And this is, you know, an index of wheels that are pre built for installation on the Raspberry Pi. Oh, yeah. And so that's a really interesting solution to a very specific target for installing, you know, complex Python packages, and we'll itself produces the ability to target that specifically. And so I think that the components are there, and the ecosystem, you know, there are so many people involved in the wide open Python packaging ecosystem, that building consensus and building a direct way forward to start defining and maintaining the specifiers. You know, and starting to roll that out and building standards that everyone can can comply to, I think that provides a really good opportunity in the future for wheels to step in to cover some of those bases. I don't think there'll be a point where the, I guess maybe this is pessimism, but it does, it doesn't feel like there will ever be a point where the volunteer and community driven Python packaging standards, meet every single nice use case, and match every single expectation that, you know, certain people would have. And so, but I think that there's a, there's the building blocks in place for us to see a solution that meets 80% of those needs are much more honestly, you know, with time I, I'm a big fan of wheels myself.
Carlton Gibson 53:37
So this is really interesting topic because you know, I'm a Python user, you know, when I'm working on Django, and you know, the Python packaging landscape just confuses the heck out of me and environment as a quote unquote, expert.
Ee Durbin 53:56
And I am an expert in one part of the Python packaging
Carlton Gibson 53:59
system. But right. You've got that, you know, but I'm not an expert on packaging, but I'm, you know, I'm supposed to and I'm just I, you know, what's going on with, you know, setup convict versus Python. Pi project got home? I don't know, I don't know. I see all these pets come out. And it's like, and I just have I kind of nice to hear you say that you think it will come together and meet the, you know, 80% plus use case? I
Ee Durbin 54:25
think we're already at 80% plus mean for for a lot of stuff. And I think there's a there's a there's what's necessary in place to see a better story with time. And a lot of that will just come down to facilitating conversations and building consensus.
Will Vincent 54:42
I think so as the last question, if you in your role at the PSF if you had, you know, the power to do whatever you want to what's on your wish list for, you know, fixing ongoing stuff or new projects that would be fun to tackle, who
Ee Durbin 54:58
I have to admit that you've caught Me in one of the busiest times of the year.
Will Vincent 55:04
So mine was sitting on a beach doing nothing or say sitting wherever doing nothing we know.
Ee Durbin 55:09
Like, you know, right, right now we've got pi con coming up in just about five weeks. And so, you know, that sort of turns into an all hands on deck amongst the PS PSF staff as well as all the Python volunteers. And so, yeah, it's very much like, what's what's what's top
Will Vincent 55:25
stack right now? Right.
Ee Durbin 55:27
But I think as far as what the PSF does, if I could wave a magic wand, and and have and have something fixed, I think it would be along the lines of, you know, the way that we communicate, engage with the community and let them know, not only what we're up to, but also what we can you know, how we can be there for them. So, but again, most recent hire actually is is sort of on that outreach, that outreach, so but on that fun, fundraise fundraising, fundraise, and that inherently requires some of that work? And so I'm really excited to see where that goes. Because I think to a lot of folks, the PSF is a little bit, you know, oh, they just sort of assume it has something to do with Python. That's really important. And I agree, but not necessarily the right would
Will Vincent 56:15
infer, well, I think Django has the same issues. I mean, in terms of, most people haven't don't know about the DSF. I'd imagine they don't need to, but how can we communicate, you know, the good things, things where we need help on, you know, growth areas? I mean, obviously, when a new version comes out, we can announce that but yeah, it's hard. It's hard to corral the volunteers to do that kind of thing and figure out what's the balance between pestering people and I think we could do more. But it's it's hard to say, I mean, we're very, very careful with how we use Twitter, very careful with blog posts, and anything official. So much so that I mean, this podcast is Carlton and me just on our own. There's a Django news newsletter I do with Jeff triplet, but that's deliberately separate from Django, because, like, who had the DSF, in a volunteer capacity would do a newsletter, like, it's a bit of work. So there's all these things that I think contribute on the edges and are better off there than being consumed within Django itself. Yeah, the same time, you know, we just added I guess, last thing I'm proud of, I keep constantly plugging, there's a forum dot Django project.com. Is that the URL, the Django form for? You know, that has a lot of really smart people on it as a way for people to ask questions, and it gets traffic, but it doesn't have anywhere near what it could I mean, if people knew that, if they asked there as opposed to Stack Overflow, they would have, you know, the right people answering it, but it's just not used, as much as it could could or should be less elegant way of saying I agree communication is tougher for Django as well. Absolutely,
Ee Durbin 58:01
no, but I think that was I think the one of the first times I became deeply aware of the DSF was when the Django Fellows Program was announced, and that, that, that that has continued to be, you know, really interesting, and apparently fruitful way of, you know, providing value to the community. So, definitely, well, there's
Will Vincent 58:20
a, there's a new c Python fellow fellow or something like that, right?
Ee Durbin 58:24
just said, the job posting that just went out yesterday. So the Python Software Foundation is, has gotten the opportunity through funding to fund somebody to work on C Python, and it's the first time PSF will have gotten that opportunity. So it's really exciting. And, you know, again, I think that it offers the opportunity, like I said, for people to recognize, like, Oh, this is like a tangible, a much more tangible thing for software oriented minds anyway. That to understand, oh, they're doing something. And so yeah, I think that the Django follow program is, you know, very similar and that, you know, I, I became aware of the DSF even because of it, so, hopefully, that hopefully, that'll help us with our messaging as well.
Will Vincent 59:12
Well, we've been lucky we're three for three I would say with fellows. So I know it's coming. We'll have a we'll have a fellow that requires a little bit of hand holding from the board, but not yet. In any event, thank you for making the time I know you're, you got a lot to do anyways, and then all the planning. We didn't even get to talk about you were a US chair for Python. At least 2019 I think couple years before that. 2018 2019 it
Ee Durbin 59:37
was honestly one of the biggest honors of my life.
Will Vincent 59:40
Yeah, I remember seeing seeing you running around a lot during that as all the staff are at these events. So thank you appreciate you taking the time
Ee Durbin 59:50
so I really appreciate you all and this was a so calming and relaxing and you're very pleasant to talk to so honestly,
Will Vincent 59:58
it's Carlton everyone always tells me though You know, well, it's okay but who's that Carlton person? His voice? He sounds smart. Like
Carlton Gibson 1:00:05
I should move to America. I should move to America.
Will Vincent 1:00:10
Spain they don't take the British. No, no. No, he doesn't like your Spanish with your Catalan with extra gravitas. No,
Carlton Gibson 1:00:19
no. Only goes across in the other side of the pond.
Will Vincent 1:00:29
Well, he thank you again. Yes. Okay. Jango chat, calm chango on Twitter and we'll see everyone next time. Bye bye bye bye