Jamie is a founder at dabapps, a Django consultancy in the UK. We talk about Django REST Framework’s origins and two open-source projects, Django-readers and django-zen-queries, that help craft performant Django queries.
50% off the list price of the books Django for Beginners/APIs/Professionals by William Vincent, co-host of this podcast and current Django Software Foundation Board Member. Offer valid through Friday, November 26th, end of day.
Carlton Gibson 0:06
Hi, welcome to another episode of Django Chat, a fortnightly podcast on the Django web framework. I'm Carlton Gibson joined by Will Vincent. Hello Will.
Will Vincent 0:12
Hi, Carlton.
Carlton Gibson 0:13
Hello Will. This week we've got Jamie Matthews with us through some DAB apps crater, various packages in the Django ecosystem, longtime Django user and whatnot. So we're gonna chat about all those things. Hey, Jamie, thanks for coming on.
Jamie Matthews 0:26
Hello, folks. Thanks for having me
Carlton Gibson 0:28
on. Thank you for joining us quite excited, you've been producing some really interesting things that I'm keen to dig into. But before we do, before we do all of that. Let's do our traditional, what's your backstory? And how'd you get in? And tell us about your your, how do you find Django?
Jamie Matthews 0:44
Okay, so I'll introduce myself first I'm technical director of dabapps, I suppose in America that might be CTO. We're a kind of web or mobile agency. So we build bespoke web apps, mobile apps, usually with API backends of some sort. And we use Django for all of our server side code. We're based in Brighton, which is on the south coast of England. And we've got an office in Wales as well, about 30 people 1718 devs, mixture of back end and front end full stack. And we've probably built over the years, I was trying to add it up. It's definitely over 50. And it might be kind of closer to 100, Django projects, anything from like small code bases, you know, a few 1000 lines of code up to our biggest projects are sort of 7080 100,000 lines of Python. So we have some some really big projects. Oh, well, relatively speaking.
Carlton Gibson 1:41
Yeah. I mean, you said 30 people, right.
Jamie Matthews 1:43
Did you say? Yeah, so yeah. 30 people total? Yeah. So that's quite a big
Carlton Gibson 1:47
agency. I mean, you've got, you know, there's a lot of turnover of work there.
Jamie Matthews 1:51
Yeah, indeed, yeah. Yeah. I suppose where we'd be considered fairly big. Do clients
Will Vincent 1:55
come to you asking for Django or because of Django? Or do they come to you, and then you say, this is our toolkit we like to use,
Jamie Matthews 2:02
it's a mixture, most of our clients don't care or don't know, or care about the stack other than the fact that, you know, they're, they're happy that we're, we're good at making decisions about what technology to use. We have we do sometimes have clients who come to us specifically for Django, and that tends to be clients who have existing code bases that maybe they've had someone else build, and they need that sort of taking over from people who know how it works.
Carlton Gibson 2:30
Okay, so before we dig in further, I've got Archie, so you did mobile, native apps and these kinds of things, as well as how about so could I just ask about the front end integration? Like how do you? How do you map between Django on the back end? And what's going on on the front end? Next?
Jamie Matthews 2:48
So most, most of our Well, all of our front ends are react, mostly written in TypeScript for mobile apps. We use React Native. So yeah, essentially, we build them API's with Django rest framework. And we have, you know, yeah, react and React Native apps that talk to them.
Carlton Gibson 3:08
So yeah, so Django rest framework. We have to we have to mention it here. Because the thing you didn't mention about DadLabs is that's where Tom came from. Is Tom Christie say, yeah, so
Jamie Matthews 3:17
I realized I didn't, I didn't really answer your question, which was how I how I came to Django. No, good. So well, I'll loop back around to that one. So I think I've found I've tried to remember where I first used Python, I definitely came out. I did a computer science degree in Manchester. And I definitely came out of that with an awareness of Python. I can't remember whether I used it as part of a course or whether I just sort of found it in my you know, in my free time. I then moved down to Brighton to do a master's and I use Python quite a lot. They're doing kind of scientific computing type of stuff. When I finished a master's, I went and got a job as a web developer, but doing PHP. But I used to get so Brian has this really amazing sort of tech scene with lots of meetups. And there was one which I used to go to a lot which believe it or not, was called Flash Brighton, right? Remember flash? Yeah. Well, so I didn't really do very much flash, but I kind of went there because when they didn't have flash topics to talk about, they got all sorts of other people and all sorts of other technologies. And one of the speakers there was Simon Wilson, who of course, is one of the original co creators of Django, who at the time lived in Brighton. So he was giving a talk about Django, this would have been, I think, I think it was early 2009. So it would have been just after Onedotzero came out and kind of just before 1.1 came out. So I went and saw this talk, obviously I was doing a lot of web dev at the time. I really liked Python and this was like, wow, now I can build websites in Python. This is the best thing I've ever seen. So I went away and kind of dabbled with it. Funnily enough, the very first bit of Jango that I use was inspect dB, which I imagine basically nobody uses. Because what I did was, we had this kind of internal tool that was some sort of inventory management system, which was like a PHP app that talked to a MySQL database. And I kind of went, Oh, watch this, you know, and went and pointed inspect dB at the database, generated all the models wide up to the admin. And within about five minutes, I just had this like working admin system for this for this app. And of course, everyone was very impressed. And they all went, Wow, well done. Let's get back to writing PHP. So, so yeah, I did that for a while, played with Django in my spare time, you know, got a book about it learned as much as I could. And then eventually, myself and a couple of colleagues from from my first job decided that we were kind of going to go off and do our own thing. Set up DevOps. From day one, it was it was all about Django. Our first project was, it was for a company who were a supplier of Tesco, which I'm sure everyone's heard of. And so our very first project was kind of quite high profile, very briefly featured on like the front page of the BBC News. And it didn't fall over.
Carlton Gibson 6:08
Let's just have a footnote for American listeners. Tescos is a large supermarket chain.
Jamie Matthews 6:13
Yeah, that's it. Well, yeah,
Will Vincent 6:15
I've been to your side of the pond. So I know. I know it is. But if I hadn't, I wouldn't. Yeah. So
Jamie Matthews 6:20
other supermarkets are available? Of course. Yeah. So it was briefly quite high profile. And, you know, it worked really well and was really successful project. So we kind of we got that we made a few more projects. And then at some point, I'm trying to remember when it must have been probably 2011. Maybe I met Tom, a Python meetup in a pub. And at the time, I remember him saying, so I'm, I'm working on this this thing. It's kind of it's like a framework for building rest API's in Django is like a Django rest framework, but I'm really struggling to think of what to call it. So I said, Well, you know, what about Django rest framework? And that's a good I bet, I bet that I do well, with like, SEO, you know, that'd be nice and easily Google. So, you know, step forward a few months, eventually, we ended up hiring Tom, he was our first employee, after the three founders. And pick up. Yeah, yeah. Yeah, now that it was really good, he was really kind of crucial in shaping a lot of the early tech decisions at DevOps. Sure. And he kind of, he was working on client projects for us. But at the same time, he was also working on rest framework. And he, if you remember, when, quite a while ago, he did a Kickstarter to fund some rest framework development. And he did that work while he was working for us. And so we used to kind of sit opposite each other and bounce ideas around. And I think I'm kind of I don't, I haven't contributed a huge amount to rest framework, but I think I'm responsible for, like serializer Method field, the original version of that was me and like, you know, things like that. So so so that was fun. So that was kind of back in the day. And then, yeah, we sort of transitioned over to because originally, we did, I guess you might call it a classic Django app where you do the templates on the back end, you know, renders HTML on the back end. That was, you know, for the first few years, that's what we did. And then we moved over to this sort of API driven, react app type approach, obviously, using using REST framework. And that's, that's what we've been doing ever since really, although I've kind of recently got quite interested in the whole on Poly HTML,
Carlton Gibson 8:44
just about, well, I've been using it on projects. And it's just amazing to be for me it really just to go back to writing templates. And just, I need to, you know, so I've got a, I don't know, a form view. And I just want to, you know, jazz it up a little bit and a couple of there and it just works. And you're like, ah, yeah, this is yes, this is good. This is good. It is. So I was going to ask you, what do you think?
Jamie Matthews 9:11
So I really like it. So we've recently been I've, I kind of, I guess, I ATMs. I really like the philosophy. So I think the guy who you know is the main developer and at Amex is amazing. And I think he has some brilliant ideas. And I think he's basically right about everything. But what we've actually been doing from kind of a practical point of view, we've been using something called on Poly, which is, I hesitate to call it a competitor because I think they're very much in the same vein, and they're very much they support each other. But um, Polly is, I would say a bit more full featured from the point of view of actually building web apps. So it's come out of an agency in Germany, I believe, who do rails back ends, but other than that, they're very similar to us. So it allows you to do stuff like it very easily. Lets you navigate around without doing full page loads kind of makes makes it into a bit of a single page app and that kind of stuff. So that's been interesting. And it's nice and easy to integrate with, with React where you need to drop some React components in, well, this
Carlton Gibson 10:13
was kind of one guy. So if you're, you're essentially a React shop, and you're fully invested in there. And then a new technology comes along, how do you kind of smuggle that in? Or how did you have to do a sort of separate project to experiment? Or do you know, how do you get going with the new technology? How do you give it a go? Because that's, that's always a question I'd like to use actually maximum, how can I, you know, the work, you already use some other stuff.
Jamie Matthews 10:36
So it is quite high risk thing to do, I would say change the tech stack, particularly in an agency where everything that you build kind of exists in parallel, if you like, you know, if you're a startup and you switch technology, once you've deleted all your Rails code and rewritten it all in Django, then you can, like, Forget rails ever existed, and you never have to think about it again. Whereas in an agency, if you switch from Rails to Django, you've still got whole load of Rails projects that sit around that you're never going to be able to rebuild. So it is a high risk thing to
Carlton Gibson 11:05
do very important that projects have a similar structure so that you can see exactly,
Jamie Matthews 11:09
exactly, and you know, people, you know, are context switching a lot. And so, yeah, it's very, very important that things look as similar as possible. So like I say, it's a high risk thing to do, I think that the key is to try it on a small scale first. So try it on internal projects, which we did, and we got really confident and comfortable with it being something that we could, we could experiment with a bit further, and then you start introducing it into sort of smaller scale client projects. And it's been, it's been really good. So I, you know, we're not, we're definitely not ditching react, because we do build a lot of projects, which are very much what React is good for, you know, that very dynamic user interface type of type of thing. But yeah, we're experimenting.
Will Vincent 11:53
I imagine that's when you switched over to react. That was similar calculation around, you know, is this something we can commit to is how sustainable react be? Right? This is like, pre, you've been using React for a while, I mean, react has changed quite a lot. Over the years.
Jamie Matthews 12:10
That's true. I think react was maybe a bit of an easier decision, because we knew that we had to build the sort of things that react let you build. But before react came along, we were building them in in all sorts of slightly janky ways, you know, starting off with jQuery, and then maybe we did a bit of backbone, and we tried out a few different things, and none of them really felt very solid. And then react came along, and it was oh, yeah, yeah, that that makes sense. Someone solved that problem now,
Will Vincent 12:34
Carlton, you're still doing Objective C, right?
Carlton Gibson 12:36
No, no, no,
Will Vincent 12:37
I just kill.
Carlton Gibson 12:38
I mean, I'm saying that we're sort of tearing my axe, I do miss it. But this sort of last 1218 months, it's all gone Swift. Now. It's like Objective C doesn't quite, you know, you can still do it. But actually, going back to it, it's you don't remember, it's not as nice as you remember, it being if that makes sense. Like you have these fond memories of Objective C, but then it's not as good as swift. It's just isn't. So I've kind of chopped up. Anyway, that's a look to Django.
Will Vincent 13:09
So, project structure, you kind of mentioned that. I'm curious. So presumably, you have within the agency default you like to use? I'm curious what that looks like around especially how you organize your apps, because we had Ned Batchelder on recently, and we were talking about, you know, is there an codebase, that's mature that people can look at as a best practice. And he was rightly saying every large thing is kind of has things baked in that if you could change you would. But since you're in an agency, you have maybe more of a chance to have some greenfield projects where you can structure things. So I'm just curious how you like to do it, because everyone kind of does it a little differently at scale, I find.
Jamie Matthews 13:47
Yeah, I mean, we try as much as possible to not deviate from Django, sort of best practices, I guess. So. We, it looks a little different to what you would get if you just ran start project. But But essentially, our our project template for the past probably 10 years has been essentially what Django gives you. With some react stuff in there as well, all of the apps go into a separate top level package. So instead of having just a big kind of list of apps in your root directory, they go into a separate top level package. That package is always has the same name. So it's always called Project rather than being called the name of the project. Right. So yeah, the apps are called Project Well, no, just sort of the top level package that all the apps area, if you like. So, yeah, I mean, it's not a million miles away. Having said that, again, sort of recently, over the past year or so we've started to experiment a little with slightly different structures. I'm very, I'm increasingly of the opinion that dividing things up into apps is not always a good idea. A couple of reasons for that. One is it's when you start a project, you often it's very difficult to figure out where to draw those lines between things. And if you're not sort of building for reusability, in terms of those apps, which usually are not when you're writing bespoke code for someone else, drawing those lines in exactly the right places is quite hard. And it's then quite difficult to change it if you get it wrong. So if you, you know, Django has migration framework doesn't really handle well, actually, I just want to move this model between apps very well, or certainly it didn't the last time I tried to do it.
Carlton Gibson 15:39
So okay, but yeah, so the opposite of that is you've got a giant models.py file, or,
Jamie Matthews 15:44
essentially, so you have a model.py package, which has in it a file for each model, okay. And then you import them all at the root. So you essentially, you have, you have a model that app if you like, which has got all of your models in it. And then separate to that you have an app, which has all of your views in it, right? Because really, you know, views and models are almost at right angles to the to the kind of cross cutting domains, right? Because often you want different views to all talk to the same set of models. And so packaging them up together, in an app always feels a little bit odd to me, particularly in bigger projects.
Carlton Gibson 16:18
Yeah. And you're always got the view that imports the model from the other app. And
Jamie Matthews 16:23
Exactly, yeah.
Carlton Gibson 16:25
Okay. Interesting. So I wanted to ask you about your open source work in January, because there's a couple of big projects that I asked you about a couple and then you can riff on those. And if there's others, you want to tell us about that? That'd be awesome. But the two that come to mind is queries. And then the newer one, which is Django readers. So do you want to tell us the backstory on those and you know what the motivation is? And we can chat, chat about those packages, because they're amazing functions? Check them out?
Jamie Matthews 16:57
Sure, okay. So Zen queries. Let me first explain what it is. At its core. It's a context manager, which lets you say, which lets you mark a block of code as not being able to run queries not being allowed to run query. So if you try and run a query, ie, you know, evaluate a query set inside that block, it will raise an exception, you're just not allowed to do it. Now, why would you want to do that? What we find, particularly in big Django projects, and I guess this is kind of the classic object, relational impedance mismatch thing, right? So Django encourages you to put your business logic in, in more in the model layer, essentially. So you have model methods, or you have query set methods, which you put your code in. And then you call that code from a, either a template, usually, if you're, if you're doing your HTML on the server thing, or you call it from a serializer. And often, you know, the templates can do things in loops, or they can do things in W triply, nested loops, or serializers, can be very deeply nested if you're pulling back quite complex object graphs. And what ends up happening is that you end up with the classic and queries problem, right? So you accidentally, without realizing it, perhaps access an attribute or call a function, call a method on on a model, which ends up incurring one or more database queries. And if you do that in a loop, or particularly in a nested loop, you end up with 10s, or hundreds, or I've seen 1000s and 1000s of queries from from single view. But there's, there's not really any sort of visibility of that, if you like, you know, you can't see that happening. And that's because the, the code that calls the methods or the calls, that the code that accesses the attributes is sort of over here in your templates are in your serializers. But then, the code that you have to write in order to make that efficient is over here in your views usually, right? Because you would usually define a query set attached to a view. And so what you end up with with is this kind of weird, like binding between the two things, and they're very dependent on each other, but the dependencies aren't visible in the code. And so what I kind of, you know, thinking about this for a very long time, you know, lots of years and I spent a lot of time tuning query sets and prefetching. And select related, you know, Django gives you the tools to solve these problems, right? But it's always to me, you create the product of the problems, and then you have to solve them. You shoot yourself in the foot, and then you wait for it to you splint it kind of thing. Why is this fast? Slow? Exactly. So So what I wanted was a way of saying, well, actually, how can we avoid this? How can we make this not be a thing? So the the point of jank of Zen queries is to allow you to say, okay, as part of rendering this template or as part of access Seeing this serialized data, you know, when you're kind of serializing, some some stuff with a serializer, you're not allowed to run any queries, right? If a query happens as part of doing that thing, it blows up, it'll give you an exception. And so what that forces you then to do is to think about it when you're writing the code. So you have to before you access that attribute in your template, you have to have select related or prefetch, the query set. And as soon as you do that, then it's fine. And it all works. So the idea is, so it gives you this context manager, but then it also gives you a few little tools sort of built on top of the context manager. So it gives you a replacement for Django dot shortcuts, dot render, which will allow you to render a template wrapped in the context managers that you just switch to import at the top of your views file, and it kind of, you know, all your all your templates blow up when you're running queries in them. Or it also can give you a serializer mixin and a rest Framework API view mixin, which will kind of magically mean that it when you try and access your data attribute on your serializer. Again, if that runs any queries, then it blows up.
Carlton Gibson 21:13
Yeah. And that's a good way of enforcing a certain amount of discipline.
Jamie Matthews 21:17
Exactly, yeah. And it's, it sounds arduous, but really, it's not all it's doing is pushing a little bit of work to the start of the project rather than six months in when the clients upset because they're their projects running really slow.
Carlton Gibson 21:31
And so just as a matter of best practice, use this on every project from the beginning.
Jamie Matthews 21:36
Exactly, yeah. Because that's the idea. And it gives you an escape hatch as well. Like if you need to do something, well, actually, you know, I know what I'm doing. And I really, really do just need to run a query here. It also comes with a another context manager called queries dangerously enabled. Again, just for that
Carlton Gibson 21:54
nicely, node nicely know. Exactly,
Jamie Matthews 21:57
yeah. And similarly, there's a template, a template tag, which which does the same thing. Okay, cool. So any any more questions on send queries? Have I explained it? For
Carlton Gibson 22:07
fun? I think I'll just say give it a try, because I've used it myself, though. So the other one, then is some that I'm really excited about at the moment. And I I'm still on the fence, I haven't you know, I've played with them a bit like, Do I dare use? Do I dare smuggle this into a real project? It's Jenga readers, can you tell us about what's going on that one? Because that's quite Yeah,
Jamie Matthews 22:27
I'll try. So this this is one of those things, which is is quite difficult to explain purely over audio, it makes a lot more sense if you can see the code. So this is if you like the next step on from from Django Zen query. So if same query says, Well, you can't run a query here. But it doesn't really give you any help of how to structure your code, how to how to, you know, correctly prepare, in order to render things by prefetching, and select relating and doing all the other things that you need to do in order to make that query efficient. So the origin of Django readers was in another project, which also came out of Deb apps written by Paul who's one of our engineers. And that was called serialization spec mixin. And as you can imagine, that was very tightly bound to Django rest framework. So the idea if that was that you would have a, what's called a spec, which is, I guess, kind of the closest thing that you can think of is it's a bit like a Graph QL query. But rather than being in your front end code is in your back end code, if you like. So it's a property of a rest framework, view, spec equals, and then it's a list. And in the list, you can have strings. And the strings are just field names on the model. And you can also have dictionaries and the dictionaries, the key in the dictionary is the is the name of a relationship. So let's say you have you know, a book with an author or whatever your your key would be author. And then the value of in the dictionary is another list, which specifies which fields of the author you're interested in.
Carlton Gibson 24:06
So there's two nested serialization format.
Jamie Matthews 24:09
Exactly. So it's very simple spec for saying, here's all the fields that I'm interested in. And then what it does, it generates both a query set, which is sort of surgically pulls out exactly the data that you want from the database. And not only does it automatically prefetch and select related and and all of those things. It also uses not only so that it only pulls back exactly the fields that you've asked for. So you know, normally a Django query set does the equivalent of select star essentially doesn't it explicitly lists out all the fields, but usually, that's fine. But if you know let's say you've got some huge text field in there, then all of a sudden that can be slow, or it can use lots of memory so
Carlton Gibson 24:55
or you're fetching your your custom user model, which has got 300 fields on because you've used it Oh, here we go. Exactly the running gag.
Jamie Matthews 25:02
It feels like you have some experience there.
Carlton Gibson 25:06
But anyway, the point is it just gets the first name. Last name. It does exactly,
Jamie Matthews 25:11
yes. So it pulls out only the data that you actually need to, you know, create your API endpoint shape. And then the other thing that it does is it generates your serializer for you. So it creates a serializer class with some nested serializers, which precisely serializes exactly the thing that you need. So you just write one quite concise thing. And you get efficient queries and serializers, kind of free. So Paul wrote this thing, amazing, like beautiful bit of API design, I don't think he would be upset with me for saying that the code was kind of gnarly. So it was very much one of those projects where it's like, let's get it to work. Of course, he understood it all. But it wasn't the easiest code to follow, it wasn't the easiest thing to maintain. And when we sort of tried to add features and fix bugs in it, it was like, well, actually, there aren't many people who really understand this well enough to do it. So that's sort of that's one side of it. If you just park that in your brain, that's the that's the high level thing. We'll come back to that in a minute. Okay, so that thing got me thinking, okay, so what would this look like if we, if we built it in such a way that it was simple? Okay, so rather than being this really complex, gnarly thing, rather than being bound to rest framework very tightly, and only working with rest framework, what if we could turn this into a bit more of a generic concept. So this goes back to that thing that I was talking about earlier around, query around model methods and model attributes being very tightly bound to query set calls. So the two places where you generally put bespoke business logic in a Django project for you know, for reads, If you like for, for returning data from the database, model methods, query set methods. Now, imagine if you, you have a model method, so you know, def method, self that does some stuff, you know, access to some attributes, and then munches them somehow, let's say it's a full name. And it concatenates, the first name and a last name, I know, that's a really terrible thing to do, because lots of people believe falsehoods about names. But let's say that that's what you're doing. Imagine, instead of that being attached as a method onto a model, you just move it to the left one level of indentation, right? And it becomes just a function in a module. Okay? So it takes a model as an argument. And it returns a value of some sort. Okay. So that's, that's one thing. The other thing is what, instead of a query set method, so a query set method being, you know, model dot objects dot, something, some custom bespoke thing that you've written in your application, again, move at one level to the left indentation have have a function which takes a query set and returns a query set. And in that function, it does some stuff to the query set, right? So it prefetches something or it only the field or whatever, whatever it's doing. Okay, so now I've got, I've got my business logic, and instead of being it bound, instead of it being part of a model, a part of a query set is just a function. Okay, what can I do with those functions? Now I can put them in a data structure. So if I, if I said, Imagine a two tuple. And the first item in the two top or is it tuple? I don't know how to Americans pronounce it tuple, tuple, tuple, tupple, something, I was gonna call it a tuple. And
Carlton Gibson 28:48
I said, you know, you said, I said, tuple. Let's call the
Will Vincent 28:52
four. But I've heard smart people say both. So a
Jamie Matthews 28:56
structure with two items in anyway. So the first the first item in the tutorial is the query set function. So the thing that takes a query set and returns the query set, the second item in the to topple is the model function, the thing that takes a model and returns a value. So what you've then got is you've expressed a dependency between those two things. So what you're saying is, in order to efficiently call the model function, you have to have first called the query set function on the query set. Okay, so does that make sense? So once you've once you've got that structure to things, you can then compose those into big trees of things, right? So you can have lots of query set functions, and you pass the query set through each one and each one does its thing, each one sort of, you know, modifies the query set in some way. And then you evaluate your query set. And you then iterate over the query set and call each of the functions that return the values and it kind of I'm skipping some of the details deliberately because that's find it hard to explain go and read the readme. Okay, so now we've got a way of expressing these kinds of dependencies between between business logic that operates on an instance and business logic that operates on the query set. Well, now I've got that I can go and build serializing, serialization spec mixing again, right? All I need to do is to take my list of strings. And I need, let's say, let's say we're doing books, and in that list of strings is title, all I need to do is replace that title string with a pair that has a thing that says to the query set, hey, I need the Title field. And another thing which says, go and get the title field off the instance. Right. So all of a sudden, you have this kind of elegant way of building the thing that we had before, but was a bit gnarly. And what it what it sort of suggests is a slightly different way of structuring your business logic in Django projects. So instead of putting things like I say, instead of putting things on models, and instead of putting things on custom query sets, you just write functions, you can put those functions, wherever you want, in your code base, we have a kind of emerging standard of where to do that. But it allows you to then only import the things that you actually need to actually, you know, call that view, you import exactly the bits of functionality that you need. And then you assemble them together using one of these specs. And then, yeah, it efficiently queries the database and serializes the objects.
Carlton Gibson 31:43
And you can do that before you pass off to the render function. So so that you know, exactly, you've got exactly the fields that you wanted from the database, you got them in, in a the minimum or close to the minimum number of queries possible. And you're not going to cook, you're not going to cause any more queries when you render the template or the serializer. Or, or well, or to cast it, because you get back a dictionary, right? You get back a something that's Json serializable.
Jamie Matthews 32:10
Yes, yeah. So that's the idea. It's sort of, I guess, solving two birds with one solving two birds with one stone, that doesn't make sense, solving two problems at the same time. Having efficient and highperformance views, but also having, you know, a better ways of structuring code in a Django project. Yeah,
Carlton Gibson 32:33
I mean, it's, I mean to come check
Will Vincent 32:35
it. I love that discussion, because I'm beginners who read my books and stuff off and ask kind of what's what's advanced Django look like? And I often say, it's going to be all about query sets. And I think your description just proved that
Jamie Matthews 32:48
I couldn't agree more, I think I think the ORM is by far the most important bit of Django, you know, like, you can imagine almost any other bit of Django looking kind of different or replacing it with something else. But the RM is really the bit, which is most of the complexity and most of the power is in there. Yeah, I
Carlton Gibson 33:07
mean, there was some talk about wrapping Django in a service layer. A couple of years ago, you know, a year or so ago, there's some blog posts going back and forth. And one of the points is where if you're not, but if you're not using the, the ORM, like, then why using Django, you know, like, you might, you might go somewhere else if you weren't gonna use the ORM.
Jamie Matthews 33:26
I completely agree with that. You know, I think that that part of I mean, Django readers, I suppose, is starting to slightly go in that direction of sort of being a little bit like a service layer. But very deliberately, I wanted to keep it very familiar to Django developers. So that the whole idea is that you can mix and match it with what you already know about Django. It's not it's not hiding Django away, it's still very much Django. It's just doing it in a slightly different, different way, I
Carlton Gibson 33:52
suppose. And what's really nice about it's the composability is like, you could define the individual bits to pull pull the fields out, or the individual query set methods, and then they just slot together like little Legos. It's lovely. Yeah. And so you kind of describe this, you use that you said, you mentioned the G word graph. QL. Before so you're building using this to build what nested nice nested responses that a mobile client can get all the data it needs in a single go and that kind of thing?
Jamie Matthews 34:19
Yeah, exactly. Yeah. So we tend to try to put as much business logic as we can, in the back end, rather than in the front end, you know, we try and keep our front end, quite minimal to the extent that they can be. So we do end up with quite complex and quite bespoke API endpoints. So we, you know, I suppose we're maybe veering a little bit away from that restful purity of having, you know, very basic sort of flat representations of things. And moving more into this API endpoint serves this particular purpose it it allows this this page in the app to render I guess, rather than the front end having to make millions of API calls and stitching them all together. there again.
Carlton Gibson 35:00
Yeah, I mean, one thing we talked about when we had Carson, who's the HTML creator on was that it's kind of easier for the back end developer to craft the API endpoint, than it is for the front end person to, you know, that you've kind of got a mismatch, where the front, the front end guys, person is like, I need this field and that field and that field, it's much easier for the backend person to put that together and say, look, here's your endpoint for that page or that request or
Jamie Matthews 35:26
Yeah, exactly. Exactly. That.
Carlton Gibson 35:28
Jamie, do you have any other great items that you want to plug while you're on? Because I'm excited about but what else have you got?
Jamie Matthews 35:36
Well, I find it a bit of an alien concept to plug open source packages, like I don't care if anyone else uses.
Carlton Gibson 35:42
What's on your list is that you give away your
Jamie Matthews 35:44
shot. So we very much we have a an approach of open sourcing, general purpose, utility things which are useful to us. And if other people find them useful, and that's great. So a couple of other things that we have, we have a project called Django DBQ, which is, if you like a kind of lightweight replacement for celery. You know, celery really does, too. I suppose you would use celery in two places. Very, very high performance, high throughput, highly parallelized, dealing with millions of jobs as quickly as you possibly can. Well, fine, but a lot of people don't really need that. Really, all I want to be able to do is to send an email when you press a button, but not have the person have to wait until the email has been sent before it comes back. Right. I just want to ditch it into a queue and kind of I don't care
Carlton Gibson 36:30
unless there's like 10 jobs a day sort of thing.
Jamie Matthews 36:34
Egg precisely. Yes. Yeah. So Django DBQ is a really lightweight, entirely ORM backed queue, which just kind of presents its worker as a as a management command. So you just managed up UI worker, worker will start up and then you can just create instances of a job model. And you can pass them some arguments and a function to call and it'll call the function and do the thing. And then it's and then it's done. So we use that a lot on all of our projects. And we don't currently have any projects at all, which is salary, which there's nothing wrong with salary, but it's a very big complex beast, and it's quite operationally complex.
Carlton Gibson 37:12
Sometimes it's like using a sledgehammer to crack a nut.
Jamie Matthews 37:15
Exactly. Yeah. Yeah. And then I rather probably our our most popular open source packages called Django log Request ID, which is popular because it's linked to from the Heroku documentation. Although they they specifically linked to like version 1.0, which is from years ago, which is a bit annoying, I don't really know who to contact. I don't think anyone
Will Vincent 37:38
work there in four years. With all seriousness, I mean, their build pack is four years old. Yeah, I love that are the Django Heroku thing that they linked to in their docs that yeah, kind of worked on is been closed for three years. Yeah,
Jamie Matthews 37:53
we use, we host all of our stuff on Heroku. And it's absolutely fast, fantastic. But it doesn't change very much, it seems to be pretty, you know, it works. And they don't do much in the way of maintaining it. So yeah, jet, what Django log Request ID does is very simply allows you to every time you write a log message, it will attach a UU ID to the at the level of the request. So if you've got a concurrent environment with lots of requests happening at the same time, and you're capturing the log output from that, it can be very difficult to tell, with lots of interleaved log messages from different threads or different processes, how they join up with each other. So what Django log Request ID does is it sticks a UID in all of the log messages that have come out from a single request, so that you can then grep your logs or search your logs for that UU ID. And that will give you a nice list of all of the log messages that that request created, which is which is really useful. Okay, so that's like request we have sort of, we have a few other small open source packages that do various simple things, but they're the main ones, I think. And the other
Carlton Gibson 39:00
thing you've got is a there's a direct blog as well, which I, you know, there's article of Tom's on there from years ago about wrapping your Django logic in a manager methods and whatnot to one of your canonical blog posts. It's no, it's just like, oh, wow, I will I will keep mentioning that blog post till here we go.
Jamie Matthews 39:21
directly contradicts the advice that I just gave in Django readyspace. No, don't do like that anymore.
Carlton Gibson 39:27
It's about I think the essence of that that particular post is, is about having a single point of control for say, creating a model instance. So that if there is business logic that needs to be applied, it's applied inside that method, and then all client code can call that one method. And then you know, it happened. Whereas if you if in a view, if you just go, you know, model, new site model, save, who knows whether the business logic got applied, because it's, you know, it's a bit wild west at that point. I think that's the long and short of it, but
Jamie Matthews 39:59
exactly And we still follow that principle. You know, it's not exactly the same to the letter. But, you know, general principles like that you're absolutely right are very much worth sharing and will improve everyone's Chango code bases. I think
Carlton Gibson 40:11
there's lots of super posts on it on the DevOps blog, so well worth having a look.
Jamie Matthews 40:16
I should probably write some more, shouldn't I? Yeah, no.
Carlton Gibson 40:19
Well, this is kind of what I'm getting to keep that going.
Will Vincent 40:23
Yeah. Speaking of improving your code base testing, can I ask just what your kind of default testing kit you throw it a project is like, what?
Jamie Matthews 40:32
Yeah, so So we're maybe in the minority in the sense that we just use what Django gives us. So we use Django O's, testing base classes, you know, test client, all that stuff. We don't use PI tests. We don't use anything like that. Again, mostly because it's all just there in the docs. And it's how you how it tells you to do it. And so it's really easy to just point people at that and say, there you go, you know, do it that way. We use a few, a few things on top of that. So we use model bakery to kind of create model instances, we obviously use the sort of the built in mocking stuff. But yeah, basically, just just Django.
Carlton Gibson 41:14
I mean, a lot of the test cases come with helpers, you know, transaction test case, and these kinds of things.
Jamie Matthews 41:21
Yeah, yeah. Yeah.
Will Vincent 41:22
I think that's great. Yeah, I mean, I've been, I asked just because great to share with others. And that's something that comes up if we're, you know, having a beer I'd ask you, and also, I often, in my books, like, I have quite a lot of testing. But I haven't yet gone down the PI test coverage. You know, everything else rabbit hole. Because it is a lot more. It's used a lot. It's helpful, but it is, you know, what you said was salary. It's like, well, it's this whole other ecosystem are rolling in. And it is true that you can get, I mean, I can test all the stuff I do for people just with the built in tools. Yeah,
Jamie Matthews 41:57
I mean, it's a bit weird, because it's that whole, you know, it came out of J unit. And it doesn't it looks weird, right? It doesn't look like Python code. But somehow I was thinking about this the other day, somehow, I kinda like it. Because you know that you're looking at test code, right? Rather than looking at business logic code, because it's all it's all slightly weird Python with like, you know, uppercase CamelCase methods and things like that. So it doesn't bother me, weirdly,
Carlton Gibson 42:22
I can never probably should could never remember the name of this search. So I'm always like, self serve, what am I serving now? Like?
Jamie Matthews 42:30
I, you know, I've been doing Django for 12 years or something like that. And I still keep the docs open all the time. You know, it's, it's just, it's a big library, isn't it? So there's a lot of surface area to memorize.
Will Vincent 42:41
So running a large agency, I'm curious how you train up your depths? Like what or do you do pair programming? Do you have internal stuff? Do you have other resources you like to use? Because you, you bring in more junior people? What does that process look like?
Jamie Matthews 42:55
Yeah, so we have a few different things that we do. The starting point for people who've never really done any Django before is always the Django Doc's, I think that the the tutorial in there is good enough. And that's a good place for people to start, we have an internal company handbook, which we call the DevOps docs, which sort of builds on top of that, and says, you know, given that you understand basically how Django works, this is how we do Django. So it just has DevOps, specific stuff. And it has a load of other things in there around how we run the company, and, you know, front end stuff and testing stuff, and operational stuff, and all that kind of thing. And then yeah, pair programming, mentoring, it's been quite difficult onboarding people, in the past couple of years with everyone working remotely, because it's so it's so great to just have a new person and an experienced person just sitting next to each other at a desk and, you know, shouting questions at each other. But we've we've adapted, and we try and have people on on Zoom calls as much as possible. And now we are able to go back into the office to some extent, we try and encourage new starters to go in at least a bit of the time and, and sort of, you know, work with the person who's trying to onboard them.
Will Vincent 44:12
I mean, that's one of the things. Having someone who's more experienced who's also interested in mentoring you is you can't put a price tag on it. I mean, I think, on some level, that's why all these CS grads can go into these large organizations. And it's not the organization itself has great training, I would say it's just that there are experienced people and there's sort of a there's enough slack that they can spend the time to level you up. Because if you sit next to someone who knows what they're doing, you know, in three months, they'll get you what would take a year on your own.
Jamie Matthews 44:42
Oh, yeah. And we really try and encourage a culture of asking questions. That to me is it's absolutely key. It's far better, to risk appearing stupid. And of course, there's no such thing as appearing stupid like people who've been programming for their entire lives as stupid questions. It's just how programming works. But If you can get over that, that that fear of of asking a silly question, it's so much more efficient than going away for a week and trying to figure out something for yourself and coming back with probably the wrong answer, but certainly an answer that someone solved before 100 times. And you could have fixed it in 10 minutes, if you just asked. And I think code review is really important as well, I think we use GitHub and we use pull requests, and we make sure that everything gets reviewed before it can be released to clients. And, and that's definitely not a sort of command and control, hierarchical thing. It's not more senior people reviewing more junior people's code. It's also junior people reviewing senior people's code, and that allows the genius to see how seniors think, and to read code written by someone with more experience than them. And I think that's, that's really helpful as well.
Will Vincent 45:49
Yeah, I mean, I would say, as people move from beginner to intermediate level programming was specifically with Django, it's learning where that line is, in terms of how long do I spend on something? And then when do I ask, because I think, as a beginner, you know, a beginner needs to put in some degree of work. But you know, don't spend a week on something that, you know, maybe it's an endless one issue, or maybe it's just, you know, a simple config thing. And I still do that I just Just yesterday I, I had a question around forums and views, and I did enough research that I was like, Okay, I think I kind of know, but I still want to know what the best practices. So I asked Carlton, some other people and, you know, obviously, like, I started know, what's a good question to ask someone. But I, to your point, I still ask all the time around certain things, but it's usually more, you know, the what's the best way rather than, you know, how do I get this to work? Because I can get it to work. But yeah, I do want to give a plug Carlton to wrap reviews, just because I am throwing it in the 3.2 update to my books, but it's also the
Carlton Gibson 46:51
question. Yeah, that's the question. Well, as yesterday was about, let's say, you've got a view that you want to respond to differently to something on regrets, like, you know, might be the content type, like a detail, detail view of the forum. Yeah. Okay. So you want to handle the same the detail view and the form view and the same thing, and then you know, you so you can have a nice to two separate views, one that just returns the detail view and one that return handles the form. And then you can have a kind of view that decides which to dispatch to in front of them, it's kind of a nice pattern, because you can do that with, you know, a ListView, that you also want to be an RSS view or, you know, or JSON response for the for the same thing. And it was a nice, it's a nice pattern. I was like, No, we'll do include it, you know, include that in the example because I think that's something that's really handy to learn is this, this, you know, if you put it all in one big, fat view, it gets really complicated. Like,
Jamie Matthews 47:43
I'm a really big fan of simple views, I have to confess, I'm not a huge fan of Django, generic views. Django as well. Django has kind of class based views, I guess. To me, there's a fantastic, quite short book written by Luke plant called Jango. I think it's called Django views the right way. That's a really, really good read. And it kind of argues that the way Django views encourages you to compose complex functionality is quite confusing, quite difficult to follow, quite difficult to read. And you only can really follow it. If you know the API really, really well. Or if you kind of almost have the source code open next to you. So you kind of go okay, so that method is called by this thing that's on that base view. mixin. And then that one calls that one, and it's just this huge nest of things, calling other things. And often, it's just easier to write the damn code out, you know, even if you're repeating yourself, it's just clear, and you can just see this happens, then that happens, then that happens. And then that happens, done. Right. And I'm a big fan of that.
Carlton Gibson 48:48
Exactly. I think the classmates views as they're, they're just too clever. They're the kind of the best example I know of what goes wrong with object oriented programming. Like, you know, this good code is great. It's super tested, it's super well documented, it's got classy class based views, but when you you know, when you need to use it, but that's not maintainable, easy to reason about, you know, software
Will Vincent 49:17
calls and I have to read your tweet, because we had this exact discussion where I even said to Carlton, like, I like I have it working function based views like, but I really, I'm somewhat committed to the generic class based views and getting the you know, tweaking it properly. And Carlton said, and I quote, If post loads all logic and all in one place else loads more logic and all at one place first breaking it up. So I agree with that. But that is that you know, to your point, yeah, you're gonna duplicate yourself a little bit, but you know, it's weighing not versus this big inheritance chain and, and part of why you split up class based views is that when you have a detail view and another mixin, another mixin. You know, that gets wonky fast with in subtle ways. So the as
Carlton Gibson 50:00
well, a lot of the mixing methods, they're like, it'll be just like self, the object is equal to self dot get object. And then that will be the sub method. That'll be the super method that you'll call. So you get so in your, your method, you go super, you know, this method, and then you write some other logic is that and then so when you're looking at that six months later, you have to go and look at what the super method is. But the super method was one line. Well, from that six months later, you is much better off having that one line in front of you, even if it was duplicated, then having to dig through two or three files to see what's actually going on in this method. I would and Tom's argued, and you know, Jamie, you're nodding. So you clearly agree.
Jamie Matthews 50:42
I completely agree. Yeah,
Carlton Gibson 50:43
come on, let's do our weekly plug for Django, vanilla views as well, because Tom Christie, right, this, this alternate set of class based views called Django, vanilla views, which is a nice example of a different way of implementing the same functionality, but in a more sort of inline declarative manner.
Jamie Matthews 51:00
Yeah. And again, Dan, give interviews another thing that Tom was working on when he was at DevOps, and I seem to remember he wrote a blog post about that, which is probably still on our blog as well, somewhere.
Will Vincent 51:09
As we wrap up, I always want to ask about deployment, you mentioned use Heroku? Are you using GitHub actions? Or what's what's the standard flow for deployments for you all?
Jamie Matthews 51:18
Okay, so we very much try to keep our sort of ops and deployment approach simple. And I don't like running servers, I've done it for a long time. And I think that it's, it's a big overhead, and you have to almost hire a team of people to do it properly, you know, with the proper security, patches and keeping things up to date. So we use GitHub actions for CI, what we do is kind of, we work in a pull request workflow. So we'll have let's say that we have a project that's been live for a while. And we're working on a new package of work that might consist of, I don't know, three features into bug fixes, each one of those will go into its own pull request against the main branch. Each one will be reviewed independently, will then build what we call a release branch, which is essentially another pull request, which merges all of those together. And that's another pull request into the main branch that will then get deployed to a staging environment. And in order to do our deployments, we just use the Heroku Web UI, you know, we go in, and we have a rule where we, we pair on deployment, so you always have two people making sure that you're clicking the right button on the right project kind of thing. Yeah, you know, you go in, you press the button, it deploys it to the staging environment. You test it internally, the client signs it off. And then same process for deploying to production, click a button, and I'm very keen on that approach. Definitely.
Will Vincent 52:51
Sounds rock solid to me, just always like to ask anything else that you want to bring up Google? And Jamie, you? Well, I
Jamie Matthews 53:01
mean, all I would say is, we're always on the lookout for good Django, people in the UK. So if there's anyone who fancies working in a, an environment with a lot of knowledgeable Django, people doing an interesting mixture of different types of client projects, you know, we really don't specialize in any particular type of client, we have all sorts of different projects, and completely different sectors all going on in parallel at the same time. So it can be really, really interesting. So if anyone is interested, drop me an email jamie@devops.com. Even if you don't see a suitable job ad on our website, just drop me an email anyway. Because we're always on the lookout
Carlton Gibson 53:42
for. Okay, super well, I'm gonna say thank you for coming on, like, you know, really exciting to talk about your projects and really interesting on, you know, I'm so hot about Django readers. And you know, I bumped into it a little while ago, and there was an issue that was is this usable in production, which he recently closed as, yes. 1.0
Jamie Matthews 54:01
Yes, I tagged the one that we're using in production. So that means that everyone else can Okay, super well, that's, that's, honestly, that's, that's really good to hear. Because I think, you know, I'm always slightly concerned that ideas that I have are a bit crazy. And so to hear someone else who's very, very knowledgeable about Django, thinking that it might be a good idea makes me very happy. So
Carlton Gibson 54:22
yeah, no, I mean, it looks it looks right in the in the ballpark. You know, there's this catters Is that as in Carter's is played out into this, you know, and I in my sort of project folder, Django readers now is in there as like, you know, these are these are the contemporary approaches to serialization. So super excited to
Will Vincent 54:39
be that's a good tagline, a contemporary approach to serialization. Oh, yeah,
Carlton Gibson 54:43
that's trademark me. Alright, let's call it quits that, Jamie, thanks for coming on. Thank you for joining us, everybody. We're Django chat.com and chatting on Twitter. Join us next time.
Will Vincent 54:56
Bye bye. Thanks, Jamie. Thanks, bye