Editing stuff in prod
Here's a concept: "ssh to prod and edit directly on the machine" is generally bad news.
I heard from a third party that someone was doing this at $COMPANY on one of my services, and I just thought (and said to them in chat) something like "ugh" or "oy". I left it at that. I did not reach out to the actual person who was doing it because I didn't care that much. This particular company had many, many problems and this one was comparatively minor. There was no reason to go out and deliberately stir things up, considering how poorly many people tend to handle being told "don't do that" in general.
I should point out that I didn't know this person and had no idea how they would respond to such feedback - it might have been fine! They might be a well-adjusted person who can take a request like that and not turn it around into a full-on nuclear attack. But, I didn't know either way.
Some time passed, and it turned out this person somehow heard about my random groan when it had been mentioned to me by that third party. I assume the third party talked to them at some point, or maybe it just wound its way through the grapevine and made it back to this person.
At some point, I actually had a meeting with this developer for some other reason, and what happened was kind of amazing. First thing, he brought it up and asked me about it. I said something like, "well, I didn't want to get into it, but since you asked", and proceeded into something along the lines of "obviously it would be best if you didn't do that - sshing in and twiddling Python files on the actual production AWS instance of the service".
I didn't stop there, though.
"But... I suspect you are doing it for some reason?"
He was, and said something like "we can't do (other method) because of (dumb thing inflicted upon all of us by $COMPANY's crap infra) and (other dumb company thing)".
My response was... I hope... reasonable: "So as much as I would love to say 'don't do that!!!', I'm not going to stand in the way of you literally doing your job. So please proceed with care, and keep an eye open for some day when (first dumb thing) and (second dumb thing) aren't problems any more and you need not do this any more."
This really happened. That person's hands were tied, and there was no useful way to do anything about it. The "infra" which had accreted at this company forced us into any number of terrible patterns, and this was the path of least resistance that still worked. He was biasing for useful outcomes and to his credit was being *very* careful about it.
It was either "edit in prod" or endure DAY LONG development cycles: make a change, wait 24 hours, come back tomorrow, and try it again. That's how broken it was.
In other words, this was someone who had been asked to give haircuts to hyperactive children and had only been handed a chainsaw, and still managed to actually DO it without harming or scaring anyone. That takes some serious effort (and skill).
The best part about this was that we had a totally reasonable conversation about it. It's the kind of thing which had become a rare commodity in those days at that place.
Months later, someone asked me what I would have done if I had been in his shoes and hit the same problem. My answer was something evil involving "punch a TCP port through ssh tunnels and/or socats to make it look like I'm coming from the prod machine". I would have done that because it's one of the things I know how to do.
Would I expect someone else to go to that level of insanity every single time they just want to get their job done? Absolutely not.
Editing files in prod to develop them? That's bad news. Deliberately rigging tunnels to spoof a connection from prod? That's also bad news. Having to do them to get your job done? You might be in the bad place.