Journal for the Camarilla Servers

mail (final update for this particular instance)

Fri, 01 Jul 2005 23:02:43 GMT

mail is flying out of the box like there's no tomorrow. Once it cleans up a bit, I'm gona stop it for long enough to fix a few more things. I know a few of you wanted mail though...so there it is!

a good update!

Fri, 01 Jul 2005 20:59:22 GMT

ok, so it's on a livecd now. Unfortunately, if I merely fixed it, that wouldn't be enough since there is a cd in the drive! I can eject it, but if I reboot to run off the system itself (instead of the cd) then when the system is coming up, it will close the cd drive bay, and boot from the cd again. Only, no one will be there to get it on the network, and start sshd...

So here's what I now have to do:

fix it

get all services working off the kernel booted from the cdrom

do item2 while making it such that once the system is rebooted without that cd, it will work

tada! No worries, I've done it before. Actually, will be welcome, since this means I can do some hard-core scrubbing on the real / and /usr filesystems (since they aren't even mounted)

going to try for getting it done tonight. I had other plans, but you all make me feel so loved I just thought, eh! Why not! ;)

an update!

Fri, 01 Jul 2005 01:37:37 GMT

some of you may not be aware, but yes - cammail is still down! It can't come up enough for me to get it back. All I need is a shell...

tomorrow, I'm going to walk WW through getting a livecd working enough to get me back on the box, even if it means I won't be on the box's native kernel. So hopefully, tomorrow, it will be back.

All from a power loss that lasted overnight, or at least several hours? So in this case, a UPS wouldn't have helped.

yes - we know (cammail)

Tue, 28 Jun 2005 22:10:56 GMT

electricity.

Sounds repetitive, I know. Apparently it's down hard, and isn't coming back up happily.

And I thought I didn't have anything to do tonight...

mailman on the web

Fri, 24 Jun 2005 22:57:51 GMT

up! was just messing with various things. Lemme know about any mailman problems you see now.

updates and features

Fri, 24 Jun 2005 15:38:44 GMT

an update about the server move and some additional features.

Server Move:

Rate of issues has slowed down a little... I'm trying to keep up and get them fixed as soon as possible, but I need lots of information from all of you who log issues.

I'm going to be spending a lot of time this weekend fixing issues (my girlfriend is out of town)... so if there are things you know are broken but haven't logged, do so now.

Log issues at http://camarilla.white-wolf.com/bugs/

I know bunches of people have the wrong (or weirdly formatted) cam number in the approvals database (or didnt have their second or third account linked). I can correct these, but I need you to log your current (linked) cam number and your old approvals login name in the bug request.

New features:

I know several people have had some tribulations importing data to the prestige system. Because of this, I've added a step to the import where you verify your entries before they get written to the database. You have to answer the question at the bottom for the import to actually do anything.

Also, some people have requested the ability to attach full emails, large details to prestige. I've added a "detailed notes" section that you can reach by editing your prestige. When values are added to this field, it addes a new link in the notes section to show the full details of the entry. This field is limited to 4,294,967,296 characters... so try and be brief :)

Since formatting would be impossible you can't import detailed notes, you'll have to add them after the import... sorry.

There's much much more... but I'm getting long winded...

Thanks for bearing with me, soon this will all be fun to look back on...

camarilla server move... complete

Thu, 16 Jun 2005 02:15:27 GMT

If you haven't noticed yet, camarilla.white-wolf.com has completed it's move.

Please remember you need to link your White Wolf account to your Cam Membership by going to http://www.white-wolf.com/camimport.php

Once that's done, you can use most of the features on the site that require a login.

Please log issues to http://camarilla.white-wolf.com/bugs

And please be gentle, the CRD (http://camarilla.white-wolf.com/crd/) replaces the member database (and adds some new functionality, too).

server update

Thu, 16 Jun 2005 02:11:51 GMT

If it makes people feel better, I (me, not Brent) had to do this too, and was apprehensive...but it worked, and was simple. :)

----------
Ladies and Gentlemen,

As you may or may not have heard (or noticed), the new server came online
tonight.

http://camarilla.white-wolf.com now will bring you to a new location with some
new features.

I'll let everyone explore for the most part, but there are a few things you need
to know...

1) In order to use most services on the new system you will need to link your
cam number to your White Wolf account.

To do this, first make sure you have signed up for a White Wolf account at
http://www.white-wolf.com and are able to log into it. Once you have an account
go to http://www.white-wolf.com/camimport.php to link your Cam Number to your
account. You will need to know the login password you used in the old member
system to link this.

Once you have linked your Cam Account and your White Wolf account, you should be
able to access most items on the Camarilla system by logging into your White
Wolf account.

2) The old ccdb / member database has been replaced by the CRD. The new link is
http://camarilla.white-wolf.com/crd . Please explore... but also give us some
slack... it's new.

3) To log issues against the CRD (or any of the tech services) please use the
new bug tracking system, Mantis. It is available by going to
http://camarilla.white-wolf.com/bugs . The login for this is seperate from your
Camarilla/White Wolf login for 2 reasons... it's a third party piece of software
and you can log issues anonymously if you like.

Thanks... and enjoy the new functionality.

~brent cerrato
US-NTA
US2002022439

server being transitioned

Tue, 07 Jun 2005 21:49:45 GMT

cam1/camarilla/ol' rattler has started its transition. The approvals db, amoung other things, is now offline. All logins (sans 2) are now offline. The only thing remaining is content under /mst (or /MST) and only because I threatened to shoot methuse if he forced me to take it down...in the end, I decided to just pay him $6 BILLION dollars. Approvals are one thing, but someone might have an actual rules debate tonight or something. They need the rules to be up.

Anyway, wave goodbye...will be back soon.

mailman updates!

Tue, 31 May 2005 13:11:45 GMT

Mailman should be happy now; if you are still having problems, leave us a note here or contact any of us through the chain.

/var had finally had enough, and thought it was allowing some writes, it was no longer stable. It also was beyond repair, so I had to restore and replace it.

I'll put a redirect back up so that cammail.white-wolf.com automatically goes to cammail.white-wolf.com/mailman/listinfo but beyond that, all should be avail. If you don't see something in the archive, and you never saw it on the list, then resend it; it didn't get through.

Tue, 31 May 2005 11:17:51 GMT

Hello there,

As you may or may not be aware, the Camarilla Web Server is slated to
be moved over to a new machine. This move will involve moving all
files and service which currently reside on camarilla.white-wolf.com
to a new machine, and then the redirection of the URL to the new box.

During this downtime, we are taking the opportunity to make a few
changes that we hope will make for a better member experience. I would
like to take this opportunity to describe a few of these changes in
more detail. Following the description of some of the new features is
an important request of the membership.

-Replacing the iCamarilla system with the Camarilla Resource Database-

Over the past few months the Tech Team, with the help of a few key
developers, has been working on a system that will be used to
integrate all of the member functionality currently distributed to
several applications. The framework of this change is the Camarilla
Resource Database. The CRD (as it has been come to be known as) merges
the organizational management of the iCamarilla system with the
ability to store characters, browse venues, make announcements, and
much more.

-Unifying the Logins to Various Applications-

With the growth of applications developed by the Tech Team and the
need to maintain a little more control over content, a decision was
made to unify the logins to as many of the logins as possible.

White Wolf has provided us with a framework to unify the logins by
linking your Camarilla Membership to your White Wolf account. This
will allow a central location for logging into services, as well as a
central location for personal data changes.

-Linking your Membership to your White Wolf Account-

As you probably guessed from the above section, your Camarilla
Membership will have to be linked to a White Wolf account. Since most
Camarilla Members already have such account, this was a natural
central login to use for this purpose.

This will also give White Wolf the ability to ensure that when you are
logged into their site that you are a Camarilla Member... which will
open up new content and sections of their site to the membership.

------------------------
Now... the fun part.
------------------------

In order to facilitate this move over, we will be removing access to
the machine known as camarilla.white-wolf.com. This access block will
occur shortly after NEGLECt. We hope that this outage will last no
more than a week.

During this time, services such as Approvals, iCamarilla, Ordeals,
etc. will NOT be available. This is to ensure that ALL data is
imported and moved to the new machine and is not changed after it is
moved. The cammail.white-wolf.com server will remain online during
this process.

What I need from the membership is for their Org Unit, Expiration
Date, and Cam Number are present in the Membership Database. These
fields will be used to link your current membership to your White Wolf
account.

Also, if you need to get an approval complete or to finish an ordeal,
it needs to be done BEFORE NEGLECt so that we can remove access
without interrupting your needs.

Please stay tuned to whatever method this message reached you, as
updates will be coming soon. Also, please distribute this information
to as many members as you can to ensure it reaches as much of the
membership as possible.

Thanks,
~brent cerrato
US-NTA
US2002022439

mailman!

Thu, 26 May 2005 12:15:38 GMT

a filesystem of relative import (var) got too corrupted, and is not available (and it was ext3, which should be stable enough darn it). It is preventing mail from going out temporarily...a week before NEGLECt! I'll have to reconstruct/restore it for it to work. This will be done tonight, with literally several entire days left before the con. /humour

today, yesterday, etc

Wed, 11 May 2005 01:19:42 GMT

an instant on/off power outage last week wasn't stopped by the UPS that was feeding the cam server last week, so it died. It's gotten beyond the point of wanting to deal with such things anymore, so it's not taking kindly to it.

Yesterday, the UPS just completely died. Power didn't go off at WW, the UPS just died (so I'm told). Today? No idea.

Net result - continuous power loss, which yields unhappy system. It's working again now, but I need to actually fix it again, which will require downtime sometime this coming weekend. I'm thinkin saturday morning, sometime around 8am EST. Consider that to be the schedule unless I post otherwise. Will be up by noon, because it's supposed to stay sunny and I'd like to go on a wine tour this weekend :)

updates, and such. Soon this (in theory?) won't happen anymore....?

that was fun

Thu, 21 Apr 2005 11:31:20 GMT

Forgot about this thing. Wouldn't have mattered; I wasn't able to get anywhere.

First note: do we live in a third world country or something? For the second time in 2 weeks, someone knocked down a power line (different places) and I lost power to my house abruptly and for many hours. I stayed home yesterday since I hadn't had a weekend in ages, and wanted to relax. I started doing some stuff to the cam server, and suddenly my laptop screen dimmed - which mean's it's running off battery. Crap. Nice sunny day, at least. Trouble is, I have a digital phone. No cablemodem means no phone.

Finally reached someone at WW, got stuff up - only to find device-mapper no longer saw any lvm disks in the system....?? I poked, I proded, I got in chat channels where absolutely no one had any suggestions (you'd think a channel with 1,100 people in it would have at least 1 person who knew...).

Without the large areas available, only the minimal stuff was there - which is more or less just enough for me to log in (and to display web pages, since they don't take much). All's well now though :/

If you're on a box and device-mapper suddenly forgets where everything is, lemme know and I'll tell you both what not to do, and exactly how to fix it. It ended up only taking less than 30 minutes once I figured it out...

hola!

Wed, 16 Feb 2005 15:23:33 GMT

Just wanted to let everyone know that both of us were not even turned on when Charles came in this morning to check us out. But with Charles on the scene, he was able to push our buttons and get us going again.

Seems we've had impotent power problems off and on lately, and yesterday one of us (cammail - the mailman server) was burping all over. Why? Well, papa will figure that out when he gets home and can play with us. Maybe it was a brown out? Maybe the UPS we're on isn't happy? Maybe we're just brats?

Either way, we should be talking to the world now. It may be a bit before you get an email you're expecting, but that's probably just because there's a backlog. Wait your turn!

Renewal Order Processing

Thu, 06 Jan 2005 11:33:39 GMT

Renewal Order Processing

When ordering renewals to your membership, please remember that your
billing information MUST match the credit card information for the
card used. If the address you enter does not match, the card will be
declined and you may experience errors ordering.

When you are renewing for someone else, you must still put YOUR
information in the order page. Give the Order Confirmation information
to the member you bought the membership renewal for, and they can
attach it to their account in the members section of the Camarilla
website (http://camarilla.white-wolf.com/members)

This should greatly reduce the problems with renewal orders.

Thanks!

-----
~brent cerrato
US2002022439

old camarilla.white-wolf.com emails

Tue, 14 Dec 2004 03:51:59 GMT

I did a couple things that I /thought/ would take care of the old mailman emails, without disturbing the emails that we wanted to continue. In my goal of being careful, I didn't catch them all. You'll see a few old emails going to the lists, coming from the camarilla.white-wolf.com server. If you reply to them, reply at cammail.white-wolf.com

Anyway - carry on! Approvals and ordeals should now be sending emails.

ordeal and approvals email

Mon, 13 Dec 2004 18:23:41 GMT

Greetings. When mail was moved, it was shut down on camarilla.white-wolf.com to prevent things from stepping on each other. I set to ignore new incoming mail not generated on localhost, let the spool empty for a bit, then shut it down altogether.

There were many reasons for this - mostly, so that when something wasn't working on one I'd know, and also to force people to make changes. I was aware that other, non-mailman emails would suffer - but it should be right there on the server still.

Tonight, I'll get postfix going on camarilla.white-wolf.com again, so that ordeals and approvals emails start going again. Its not a bug or problem causing it to not currently happen - its just that postfix is intentionally not running there. I might scale back to a more simple MTA for just sending out those emails regardless, just to make things as simple as possible on there.

update

Wed, 08 Dec 2004 01:03:04 GMT

ok. To move the lists from cam1 to cam2, I have to first ball em up so I can toss them over.

That ball has been being created for approaching 2 hours now. Once it is made, then it will be moved over to cam2. Then it will be unballed, and manually extracted from (at least, at first).

Just an fyi. The lists are intentionally down atm, since the ball is being created, and I don't want the files changing during the interm.

immediate plans this evening...

Tue, 07 Dec 2004 18:22:27 GMT

Attention: there WILL be lost mail tonight. No ifs, no maybes.

Simple reason: it will take *hours* to move the lists over. Once they are moved over, they will come up on the other box. I will set up camarilla.white-wolf.com to forward things over, but...once things are off camarilla.white-wolf.com, that box will be down for a few hours. That will mean no web stuff, no..well, lots of things. But mail will be back, as it will be on cammail.

I will make extensive effort to get all lists moved over tonight and tomorrow. They will be done in groups, with the newer IC lists done second, and the national and regional ooc lists done first (for communication). Those two groups will be tonight's goal. The remainder of the lists will be done tomorrow.

Assuming, of course, the extensive errors I'm currently seeing don't take the system down before I get home this afternoon...

as soon as...

Tue, 07 Dec 2004 13:01:30 GMT

the system is reset, it will be. I'm waiting (called in on friday) just like everyone else....sorry.

In the mean time, cammail seems to be ready to take the lists without any problems. Transfering the 11Gb of archives/lists over is all we're waiting on for that, which is waiting for the system to be up to make the move from.

sunday night, now

Wed, 01 Dec 2004 18:15:14 GMT

apparently there was a power surge sunday night (or was that monday? I worked 15 hours yesterday and 12 on monday, so my timing might be skewed from that) that took out both servers.

I got on cammail last night to start moving stuff over from camarilla, and found that...it wasn't there. Rather, it was, but it wasn't responding to actual requests for info.

Assuming it wakes up today at some point though, several OOC lists will be moving over to cammail tonight. An additional large chunk will move over tomorrow. The entirity of the OOC lists will be moved by the week's end. IC lists are another matter, esp those that are for the old chronicle.

Sorry for the delay in putting up this note - I've been with vendors all day down in the server room, far from any system that I could update LJ from, and I simply didn't think to do it last night.

mail issues, explained

Tue, 09 Nov 2004 16:46:46 GMT

There seems to be continued confusion about the mail issue. Let me explain it in a way that it will hopefully be understood.

When I post an email to the ordo dracul list, camarilla.white-wolf.com checks the mx record for lmco.com. It then says HELO (a mail server's command to introduce itself) to mailgwX.lmco.com (replace X with the actual number it goes to). It gives it's name. The ip is then gleened from the transmition, and mailgwX.lmco.com decides to find out who 206.65.59.149 is.

It asks the root server(s). This is what it gets in reply:

Asking a.root-servers.net for 149.59.65.206.in-addr.arpa PTR record:
a.root-servers.net says to go to indigo.arin.net. (zone: 206.in-addr.arpa.)
Asking indigo.arin.net. for 149.59.65.206.in-addr.arpa PTR record:
indigo.arin.net [192.31.80.32] says to go to auth00.ns.uu.net. (zone: 65.206.in-addr.arpa.)
Asking auth00.ns.uu.net. for 149.59.65.206.in-addr.arpa PTR record:
auth00.ns.uu.net [198.6.1.65] says to go to Z1.NS.SJC1.GLOBIX.NET. (zone: 59.65.206.in-addr.arpa.)
Asking Z1.NS.SJC1.GLOBIX.NET. for 149.59.65.206.in-addr.arpa PTR record: Reports camarilla.white-wolf.com. [from 209.10.34.55]

Answer:
206.65.59.149 PTR record: camarilla.white-wolf.com. [TTL 86400s] [A=206.65.59.149]
206.65.59.149 PTR record: www.camarilla.org. [TTL 86400s] [A=None] *ERROR* A record does not point back to original IP.

At least, that's what we *hope* it gets, and exactly 50% of the time that is what happens. The other 50% of the time, the last two lines (the answer) switch - the answer is www.camarilla.org.

So when someone does a reverse lookup, they get back that answer - but they only look at the first response. Since www.camarilla.org is the first response 50% of the time, that mail gets rejected.

What happens then? The email is resent. And resent again. And resent again. And again. And again.

What can be done about this? 3 things:

simply change 206.65.59.149 to a different ip, and point camarilla.white-wolf.com to that ip instead. WW issue.

remove the damn www.camarilla.org record. Earthlink (the authority on the name, apparently) refuses to do this. WW has lawyers talking to them, or so I am told.

Start sending mail from cammail.white-wolf.com

Which are we planning on doing? #3, currently. Which will mean that in a couple weeks, all email will suddenly be coming from somewhere else! This will take a bit of work in mailman to change everything over, and will break filters people have that explicitly name servers.

So when an email doesn't arrive right now, this is the reason. The emails are actually zooming through quite nicely with zero delays - email sent to a list returns in seconds generally (what your MX does with it then is their ball). The servers aren't bogged down, there is simply a delay in retrying a failed attempt at sending a particular email. At 50% a shot, that could -potentially- be a long delay.

Let me know if this doesn't make sense. No, not all mail servers do reverse lookups - those that don't are getting all mail instantly. Its those of us that have mail servers run by anti-spam admins that have the issue (since the reverse lookup is done for anti-spam reasons).

And yes - that means that we often look like a spam box to many servers out there...what with the volume of mail (sent to lists, not specific people), and the failed lookups.

updates saturday morning!

Fri, 29 Oct 2004 20:26:17 GMT

Saturday morning, approx 5min after I wake up (giving me time to eat some oatmeal or something), I will be updating apache and postfix on the servers. I am planning on this being ~7am EST, and it won't be started later than 8:30am EST. There will actually only be less than a minute of downtime with apache, and any downtime with postfix won't be noticable since mail will spool for the few minutes it is down (ie - no service impact).

yes, we know

Thu, 28 Oct 2004 03:15:28 GMT

sorry for the delay in verifying that we know, but yes...we do in fact know.

Both servers are gone, so failing over to the second one just won't be happening. It also limits the number of things that can be reasonably expected to be the cause (since cam2 wasn't even being used this evening).

Will have to wait until I can reach folks at WW in the morning...until then, relax and enjoy the moment of silence.