IMPORTANT: Migrating away from mailing list

SDL Development
1

tl;dr: We’re going to try to move the mailing list and web forums to a new piece of software. Please go to https://discourse.libsdl.org/ and claim your account. You can still use it as a mailing list or a web forum. This list will (probably) be going away soon.

So this is an experiment with Discourse (https://discourse.org/), to see if we can solve our mailing list and forum woes:

Problems we currently have:

  • The forum software is wildly out of date, and couldn’t be upgraded without losing the email bridge. Upgrading the software breaks a fundamental feature we rely on, not upgrading it is a massive security risk. We tried–and failed–to move the web server to PHP7; the forums needed massive work to just run at all. Serious problems, like not being able to register an account or post a reply, were failing silently, so we could never be sure if we ever really had fixed it up completely. For now we have dropped the web server back to PHP5, but being trapped on old versions of basic software to support old versions of custom software is doubly bad.
  • The spammers on the forums were brutal; they couldn’t actually post because we had to approve the accounts, but they’d just flood our SMTP server with backscatter trying to make thousands of accounts, all day long.
  • We have to manually approve forum accounts in the first place. :confused:
  • The mailing list was unreliable, and would lose mail, lose the archives, etc. This was largely a problem with the list provider and not mailing lists in general, but migrating somewhere more reliable was non-trivial, so changing direction entirely was a reasonable option if we had to do that.
  • The glue that bridges between the mailing list and the forums was not great in general.

As I see it, these are the primary benefits of Discourse for us:

  • It runs on our server.
  • Is well-maintained and constantly improving open source, concerned with what modern Internet communication looks like (silly things like Emoji and Gravatar, really important things like Unicode and REST APIs and downloadable archives of your posts, etc).
  • Is one central place where all communications happen.
  • If you like the mailing list, you can use it exclusively through email.
  • If you like the web forums, you can use it exclusively through a web browser.
  • If you want to use it through the web interface and occasionally shoot off a quick reply by email, you can do that too.
  • The web interface looks modern and responsive, with all that AJAX-y goodness. It runs well and looks good on mobile devices, too.
  • It has all our email archives imported! After a lot of manual cleanup and one-off perl scripts, all the existing mailing list content, all the way back to 1998 has been turned into Discourse topics with threading intact. More than 20,000 topics, more than 100,000 posts!
  • It has all the old forum posts imported too, thanks to a lot of manual cleanup and one-off perl (and PHP!) scripts.
  • It has all the users (more than 2000!) imported. We took efforts to try to merge people that posted from different email addresses over the years into one account, using the latest email address we saw, and deal with people that used both the forums and mailing list interchangeably over the years.
  • An actual search function that works.
  • Lots of bells and whistles and control over everything…not just as an administrator, but as a user, so you get the experience just how you like it.

So we’re going to give it a little time and see if we can turn off the existing list and forums. So that we don’t end up with two separate communities, we’re going to make this decision pretty quickly. As such, if you have problems (including a generic “I hate this”), please speak up loudly as soon as possible.

To get started: Go to https://discourse.libsdl.org/

If you sign up for an account and it thinks your email is already in use, we probably generated a user for you when the list archives were imported. Just tell it your forgot your password and it’ll email you a reset link so you can take control of the account. In this case, we assigned you a best-guess username, but you can change it yourself once you log in.

If you exclusively used the web forums, we have an account for you but not your email: hit me up and I can hand the account over to you, or you can just make a new one, if you like.

If you want to use Discourse as a mailing list and never see the website again, you can; there is an option to turn on “Mailing List Mode” in your user prefs. New topics can be started by writing to sdl@discourse.libsdl.org, or you reply to the address on whatever email you want to reply to, and it ends up in the right place.

If you have trouble, we can fix things (including getting you a username from a legacy account, assigning ownership of old posts to a new account, etc). Just let me know!

–ryan.

1 Like
2

For security reasons alone, this is a welcomed change.

Activation went without a hitch.
So far, I see my old messages/threads that I have made over the years as well, so, nothing is lost that I can see.

Overall, good move!

I am wondering, will the old mailing list be purged, since that DID leak e-mail addresses?

1 Like
3

I am wondering, will the old mailing list be purged, since that DID leak e-mail addresses?

The old list will definitely be deactivated if we decide this wasn’t a terrible mistake, but the list archives are out there and we probably can’t put the genie back in the bottle. :frowning:

4

This seems to be working great, thanks a lot Ryan!

1 Like
5

I was able to claim my account without any issues. I think this is a reasonable switch.

1 Like
6

(Cross-posting from the mailing list to be sure it’s seen, since my reply there isn’t showing up here. This is important.)

When this came up a while ago, I mentioned that I had previous experience with Discourse and it was a huge mess, and it sure looked like you and Sam acknowledged it and said thanks for the warning.

Since it looks like that warning wasn’t heard, please let me reiterate: I don’t deny that there are serious, severe problems with the current mailing list, but if you forge ahead with Discourse as the solution, you will just be trading one set of serious problems for another. When another community I’m part of switched to Discourse, we found this out the hard way.

It has huge architecture and scalability problems that you won’t notice early on, but will bite you in the butt once you get going.

XSS/XSRF vulnerabilities galore. (You say the current forum has security issues? Discourse is no better!)

The front-end is so sloppy that attempting to run it on a mobile phone will literally cause your phone to heat up in your hand as if you were playing a high-end game.

The team is exceptionally unprofessional and actively hostile to bug reports, to the point where Jeff Atwood personally made a point of banning anybody from our community who reported bugs in Discourse, simply because we were finding too many and making him and his software look bad.

Speaking of Jeff Atwood, even though he was not a member of our community, he took it upon himself to use a developer’s backdoor to set up a special administrator account on our community and go through policing things, arbitrarily moving posts from one category to another because he felt it fit better there, to the point where “Jeffing” became a verb among us meaning “moderator action to move a post to another thread/category”. I can’t help but think this behavior would be completely unacceptable for the SDL forum.

I cannot emphasize this strongly enough: if you switch over to Discourse, we will all end up regretting it. I agree that we need a better forum software, but please, please, find something less toxic!

Mason

2 Likes
7

(Also cross-posting my reply; wherever we end up, we’ll eventually not have to cross-post anymore. :slight_smile: )

When this came up a while ago, I mentioned that I had previous
experience with Discourse and it was a huge mess, and it sure looked
like you and Sam acknowledged it and said thanks for the warning.

I tried to find a better alternative and couldn’t. I’m not sure that the
technical issues you mentioned are still a problem, as development
continued on, fwiw, but this quote here:

Speaking of Jeff Atwood, even though he was not a member of our
community, he took it upon himself to use a developer’s backdoor to set
up a special administrator account on our community and go through
policing things, arbitrarily moving posts from one category to another
because he felt it fit better there, to the point where “Jeffing” became
a verb among us meaning “moderator action to move a post to another
thread/category”. I can’t help but think this behavior would be
completely unacceptable for the SDL forum.

Do you mind if I ask Jeff about this? That would be a pretty serious
breach of trust, and I’d like to understand the details.

I cannot emphasize this strongly enough: if you switch over to
Discourse, we will all end up regretting it. I agree that we need a
better forum software, but please, /please,/ find something less toxic!

The bulk of the hard work was cleaning up the archives outside of
Discourse; those are all sitting here as mbox files that can probably be
moved anywhere. If you have a better solution, please suggest it, but we
have to move quickly in whatever direction we go.

1 Like
8

So, this was what happened?

If so, that is really, REALLY in bad taste.

@icculus, what other forum software have you looked at?

9

If so, that is really, REALLY in bad taste.

I don’t know the details on that, and am not particularly interested; my specific concern was the claim about a developer backdoor. Fwiw, this was Jeff Atwood’s response when I asked:

I mean, it’s pretty clear Jeff Atwood and The Daily WTF forums strongly dislike each other, to understate it, but that doesn’t have any impact on our software choices.

@icculus, what other forum software have you looked at?

The number of things that function interchangeably as both a web forum and a mailing list is vanishingly small: Google Groups was the closest alternative we could find, and that had serious drawbacks. Once other needs: importing all the archives, migrating users, having the level of administrative oversight we desire, running it on our own servers, not having to pay a lot for it, etc…Discourse was the only option still standing, and even here I expect I’ll be writing patches and filing bugs for some desired features it’s currently missing.

But if the email+web requirement wasn’t there, though, there are probably as many possible web forum packages as there are web forums, but we had to eliminate most of them out of the gate.

10

Why do you even need a mailing list? Your choice of course, mailing list has benefits, many good old users are used to mailing list. But my two cents, as i see, very few people visit my GitHub project through mailing list, i would say it is 2% not more. And being anyhow associated with StackOverflow, imho, is a sacrifice too great to make. Let at least one place remain independent, there are vanishingly small number of places where to talk about software in the Internet, StackOverflow and such have taken over that all. For example, practically the only place where i can write about my widget toolkit and let people know that it exists at all, is here. StackOverflow has an immense number of utterly nonsense rules, like one cannot recommend any software, which is interpreted the way that when you mention some software, all the question with all its answers will be deleted. Just like Wikipedia, and this undoubtedly helps some people to force their views and suppress all others. Too many rules are evil, the only way to avoid the hell is to have as few rules as possible. Whatever you decide but, let some freedom remain.

11

So… yeah. Did you notice how he didn’t actually answer the question, and just deflected it with insults instead?

Jeff may have used the “normal moderation tools” provided by Discourse, but he didn’t come by them legitimately. He did not have a normal moderator’s account; those were for the moderators chosen by our community. He set up a deal with Alex Papadimoulis, the guy who runs The Daily WTF, that we would beta test Discourse for him. He invited a fairly large and active community to beta test Discourse. Among other things, this implies looking for bugs.

We found bugs. We found bugs galore, including several very serious bugs with security implications, and Jeff made it painfully clear that he didn’t want to hear about them. He may complain about the community’s antics, but here I’m gonna have to invoke the Parable of the Scorpion: he knew when he invited us to beta test that we were a community built around a site whose entire purpose is literally to make fun of bad code. And we made fun of his bad code, and he reacted very poorly.

As part of the beta testing process, he was set up with an admin account, for the explicit purpose of providing technical support. (Which, as previously noted, he utterly failed at.) Instead, he used this access to act as an unwanted moderator, attempting to forcibly reshape our community into something more aesthetically pleasing to himself.

One other thing to note: have you considered the cost? Discourse is not only massively inefficient on the client side, it’s also written in Ruby, which makes it massively inefficient on the server side. As one of our actual moderators noted when I asked around:

They should be looking at the server requirements for running Discourse. Discourse is very resource intensive, and they only really recommend using it on Digital Ocean droplets. Anything else, and you’re pretty much on your own as far as support goes. By the time we migrated off of Discourse, Alex was paying something like $300 a month just for the forum instances and we were still suffering from some serious performance issues.

Do you have $3600/yr for additional server expenses just to run a forum in your budget?

12

This is how most the people contributing code to SDL prefer to interact, and we’ve had an enormous amount of email users over the years. YMMV, of course; obviously other people liked the phpbb forum (or liked it better than email), which is why we needed interoperability here.

I recognize that Stack Overflow is useful and some of their rules are draconian (as is true for Wikipedia too, etc), but using Discourse isn’t the same thing as managing discourse.org; I don’t think much will change from how we would interact on the mailing lists, etc.

He had an admin account. You told me he used a backdoor. If the backdoor is “he made a deal with the website owner,” that’s not illegitimate.

It’s fine with me if you continue to not like him (I can clearly see that he’s often abrasive), but Jeff Atwood’s personality doesn’t change anything here, in the same way that I still use Linux even though Linus Torvalds is often rude to contributors.

We happen to be using it on a Digital Ocean droplet, fwiw.

If it doesn’t scale, okay, but it’s already supporting 2,000+ accounts, 100,000+ posts across 20,000+ topics spreading back to 1998 and not causing trouble. If our usage patterns change, we’ll deal with that, but I don’t see that being a problem in the near future. This sounds like a lot, but over 19 years, we’re pretty low-volume and a niche community. If this was a general purpose forum, then maybe we’d be blowing up resources.

Screen Shot 2017-03-29 at 11.31.31 AM.png1185×1474 195 KB

Honestly, from here, this transition is going smoothly and the software is pleasant to work with. I’m sorry that you had a bad experience with Jeff Atwood; I made an effort to investigate the matter, and none of this convinces me I’ve made a wrong decision. If the situation changes, though, we’ll adjust appropriately.

3 Likes
13

This all may change and it is very subjective too, but i say what i feel. The forum now looks like scrap and it is almost dead, and it was a big active forum.

14

Congrats on the huge move, I’m sure people will get used to it in time. Worked out great for the world’s premier free software FPS ioquake3 when we moved from PHPBB even though there was some grumbling at first and this move has been organized better

1 Like