The Euclid IRC server and what it means for us.

Summary:

This thread's probably a couple weeks overdue.

The Euclid IRC server and what it means for us.

pxdnbluesoul 14 Sep 2018 15:29

I wanted to lay out exactly what's been going on regarding our IRC network.

August 6th: We begin discussing options for IRC and collectively have the same idea.

+ show block

[13:04] <bluesoul> on a tangentially-related note to the discord discussion, i think i'm gonna put a thread up asking about running our own IRC network
[13:04] <bluesoul> the pros are that we are our own ircops and can handle shit without waiting on synirc staff, and i could theoretically add logging of all channels that would be accessible on the web
[13:05] <bluesoul> the cons are it would cost about $5/server/mo and we want some georedundancy, at least 3 servers, and we're our own tech support if shit breaks
[13:05] <@ChaoSera> I don't think that's a beneficial cost-benefit arrangement for us
[13:05] <bluesoul> i've done a proof-of-concept network for us in the past
[13:09] <@Tuomey> it's not that it couldn't work
[13:10] <@Tuomey> it's that it's not really necessary right now - we have a good relationship with synirc staff as it stands
[13:10] <@Tuomey> that said
[13:10] <@Tuomey> in the future I might be very interested in this
[13:10] <@Tuomey> particularly wrt logging 19 and 17 publicly
[13:11] <@Lily> yeah there aren't any problems with Syn and I don't wanna risk throwing away what we have now
[13:11] <@Tuomey> right now this is a very cool idea but it doesn't solve problems
[13:12] <@Tuomey> which I realise is probably kidn of a frustrating thing to be told
[13:12] <@Lily> yeah
[13:12] <@Lily> also i'm really against relying financially on anyone on staff
[13:13] <@Tuomey> I do appreciate people looking into cool stuff like this, even if it doesn't yield results a lot of the time
[13:13] <@Tuomey> the few times it does are worth it
[13:14] <@Lily> Also there's a lot of scp related channels on syn and I'm against uprooting all of that
[13:19] <@Lily> Anyway we've been here 10 years with no problems so there's no reason to miss it
[13:19] <bluesoul> Lily, we've had problems in the past
[13:19] <bluesoul> you may not have been around for it, but to say "no problems" is an exaggeration
[13:20] <bluesoul> now, here's a thought
[13:20] <@Lily> bluesoul: well, nothing large enough that's caused anuly trouble
[13:21] <@ProcyonLotor> Currently we have so good a relationship with network staff that I'm not sure it's necessary, bluesoul
[13:21] <@ProcyonLotor> I speak to sats pretty much daily
[13:21] <@ChaoSera> Yeah I don't think we have much to gain and in return have to do all the maintenance by ourselves
[13:21] <@Tuomey> bluesoul: oh, I agree there's been problems in the past
[13:21] <@Tuomey> but things have improved greatly
[13:21] <trets> I'm actually lowkey kinda surprised none of our long-term opstaff haven't pursued a position as an IRCop for syn
[13:22] <bluesoul> i could link a server and become an ir— yeah
[13:22] <bluesoul> sorry, got busy there
[13:22] <bluesoul> yeah i can talk to them
[13:22] <@ProcyonLotor> I would support bluesoul insofar as it's absolutely an option to have plans for in case synirc bites it hard and fast
[13:22] <bluesoul> ProcyonLotor that's kind of a concern of mine as well
[13:22] <@Tuomey> yea
[13:22] <bluesoul> now, what i can do
[13:22] <bluesoul> is have shit ready to go
[13:22] <@Tuomey> that would be literally amazing bluesoul
[13:23] <bluesoul> so what we're left with is i'm an ircop on syn and we've got an entire backup network prepared

August 7th: An ircop responded to my inquiry in #help and indicated that a server in Ireland would be good to have. I set up an Amazon EC2 instance in Dublin, filled out the application and waited.

August 24th: I get word that my server and another in NY are in consideration.

August 26th: I get word that the server has been accepted and I can begin setup.

August 27th: Completed configuration and got linked up to the rest of the network as euclid.ie.eu.synirc.net.

September 10th: After two weeks with 0 unplanned downtime, Euclid was put into the rotation for all clients to connect to.

Connections have been steadily increasing over the last few days, and even with 180 clients directly connected, the stats look great.

You're reading that right, 0.5% CPU usage on a single core server with 512MB of RAM. You could pretty well run these services off an old Nokia brick phone with a 3G connection.

In the short term, this means that we did get what we want out of this, which is the ability to handle our own instances of ban evasion and harassment without having to wait on #help which can take hours. It's already proven to be incredibly helpful.

This also means that for the long term, we've got a viable blueprint for making our own network. I have configuration files that can be repurposed with very little effort to run our own network should that be either desired or necessary down the road.

In the event of an emergency, where synIRC goes down and synirc.net doesn't work as a domain name, you can connect to Euclid at:

euclid.bluesoul.net
irc.j0lt.com

If we are in such an emergency state, we may not have the usual services (NickServ, ChanServ, etc.) available immediately, as those services are provided by a separate piece of software that I do not run on my server, it's only run on a couple in the whole network. I have set it up in the past though, and it is a pretty painless, fast setup even to do things like cloaking and vhosts.

If we were architecting our own network, which only needs to hold maybe 200 people tops, our needs are not too intense, either 2 or 3 512MB Lightsail or EC2 instances which can be as little as $2.44/mo per server when you purchase 3 years up front. I'd wager that setup could support probably 6,000 users, evenly distributed, before we would have any concerns at all about any bottlenecks. I'd probably leave my one in Ireland, set up one in the Eastern US and maybe a 3rd in the Western US if we really thought it was needed. Total outlay would be about $7.50 a month. I'm fine with defraying that entire cost myself if we need it, but the Bus Factor dictates that someone else also spends money on the deal should the time come. All code and everything needed for this plan will go on the GitHub when I have a "production-ready" copy prepared.

The not-chatstaff guy has chat questions

Eskobar 14 Sep 2018 17:21

Just so my math is right, we're talking about an upfront cost of $263.52 at the beginning of the lease, which would cover three years, and then the need for someone at the end of that period to…would we switch to a monthly fee at the same price, a higher monthly fee, or do another three-year deal? Or play it by ear in three years?

As an IRC dum-dum, the full implications are still somewhat obscure to me, but I am also not chatstaff, so this may not be a concern. Here are my questions.

1. Are you now a full IRCop for all IRC networks, or just SynIRC? I wouldn't think there was such a thing as a protocol-wide cop (that would be like being a jack-of-all-trades moderator for all Internet forums everywhere), but I'm also very dumb.
1.b. Did you just have to start this server in order to become an IRCop? Is that how that works?
2. If we move to the Euclid server, will other IRCops still have jurisdiction there? (This is more of a general "how do IRCops work?" question.)
3. Will the Euclid server not still be bound to SynIRC (if I'm reading that "euclid.ie.eu.synirc.net." configuration right)? If so, are we still not up a creek if/when the rest of SynIRC goes down?
4. Basically, are there any downsides to us other than "it costs things"? Is there something SynIRC does on the hardware end that we would be less capable of doing? Is there some service they provide that we couldn't get access to? Are there any chat clients that would need to jump through more hoops in order to use this server?
5. If this arrangement shits the bed in any way (say, in three years we can't find anyone who wants to or can foot the bill), will there be any downsides to moving back to SynIRC or another network?

Re: The not-chatstaff guy has chat questions

pxdnbluesoul 14 Sep 2018 18:03

I'm going to line-by-line this.

Just so my math is right, we're talking about an upfront cost of $263.52 at the beginning of the lease, which would cover three years, and then the need for someone at the end of that period to…would we switch to a monthly fee at the same price, a higher monthly fee, or do another three-year deal? Or play it by ear in three years?

I paid $88 and change for a 3-year reserved instance with Amazon Web Services. You basically buy the reservation and then assign an instance to it. So at the end of the reservation period, it would go to standard rates which is about double, around $5/mo. You can buy a reservation in advance so the time at the standard rates is kept to a minimum. They offer better pricing if you buy the whole 3 years up front but also have plans for $0 up front and half up front.

1. Are you now a full IRCop for all IRC networks, or just SynIRC? I wouldn't think there was such a thing as a protocol-wide cop (that would be like being a jack-of-all-trades moderator for all Internet forums everywhere), but I'm also very dumb.

Just synIRC.

1.b. Did you just have to start this server in order to become an IRCop? Is that how that works?

synIRC's charter has multiple paths to becoming an IRCop. You can either be nominated, or link a server. If someone on staff wanted to apply to become an IRCop, and the synIRC staff thought there was use for more staff and trusted that person, they could be brought on on a probationary basis.

2. If we move to the Euclid server, will other IRCops still have jurisdiction there? (This is more of a general "how do IRCops work?" question.)

If we delinked from synIRC and just ran our own network, we would have total control over who the operators are.

3. Will the Euclid server not still be bound to SynIRC (if I'm reading that "euclid.ie.eu.synirc.net." configuration right)? If so, are we still not up a creek if/when the rest of SynIRC goes down?

So, even if the rest of synIRC went down, if you connected to euclid.bluesoul.net, you would hit a working IRC server, just with no services. We'd need to pivot to running it as part of a new network pretty quickly, I'd say it would take a couple hours to modify the configuration and set up the Anope services software.

4. Basically, are there any downsides to us other than "it costs things"? Is there something SynIRC does on the hardware end that we would be less capable of doing? Is there some service they provide that we couldn't get access to? Are there any chat clients that would need to jump through more hoops in order to use this server?

The downside would be that synIRC staff are collectively better at running an IRC network than we are, and our inexperience is worth considering. In the event that servers act up, one of the IRCops is normally pretty quick to either reroute traffic or remotely fix the issue. Aside from the growing pains, the other concerns aren't applicable. Mibbit connects to euclid just fine, I see the connections pretty regularly.

5. If this arrangement shits the bed in any way (say, in three years we can't find anyone who wants to or can foot the bill), will there be any downsides to moving back to SynIRC or another network?

We would have total control over restricting IP ranges if there were persistent issues with targeted spamming or attacks. We would probably lose that with a new network. We'd also have complete control over the customization of things like cloaking, vhosts and so on. You'd also have the "long-tail" issues of people that have been away for a while and don't know where to connect.

These were all good questions, if you think of anything else or I missed something let me know.

Re: The Euclid IRC server and what it means for us.

pxdnbluesoul 19 Oct 2018 14:34

Euclid has passed muster and has been approved as a full link! I've also been added to the services operator list. This is a step below services administrators, but it does give me the ability to do things like see emails associated with accounts, set up AKILLs, and a couple other things. I've also been added to the SynIRC Administrative Council as a server administrator.

Last edited on 19 Oct 2018 14:35 by pxdnbluesoul Show more

Re: The Euclid IRC server and what it means for us.

pxdnbluesoul 23 Aug 2019 20:42

Noting I've been promoted today from services operator to services administrator and now have more or less the full set of oper abilities on SynIRC.

Re: The Euclid IRC server and what it means for us.

thedeadlymoose 24 Aug 2019 03:23

Congrats. :D

Re: The Euclid IRC server and what it means for us.

The Pighead 24 Aug 2019 12:56

Like Moose said. Congrats, man.

Re: The Euclid IRC server and what it means for us.

RJB_R 26 Aug 2019 15:13

Wahoo

---

Whether you like it or not, history is on our side. We will bury you!

Re: The Euclid IRC server and what it means for us.

ManyMeats 26 Aug 2019 20:46

Execute Order 66.