The Art Galleries of SL List is available at http://sasun.info/ArtGalleriesOfSL.htm.

New visitors, please read: About This Blog

Monday, October 24, 2011

Kiosks Back to Normal - Soon

Good news - I've successfully tested a new web host with the kiosk system and it seems to be working fine. I have not had time to do extensive testing, but the system is on the way to working again. It will take a day or two before the LL servers "catch up" with where the new website is, but they will just start working again... soon.

I decided that waiting on LL to fix this problem was simply too risky. It's been a week already, that's one week too long for the whole system to be down. The solution basically involved a ton of work to evaluate, find, and configure a new web hosting site, which means, it's a new IP address that, luckily, the LL servers don't have a problem communicating with.

Does this mean the same problem won't happen again? I have no idea, since we don't know the cause. Once LL figures this out, I suppose they will fix the problem so it doesn't bite anyone else, or me, again.

It's been a long night, I'm tired. This whole ordeal has fried my brain completely. But I think at this point the light at the end of the tunnel is rapidly approaching... it's just a matter of time before things start working again.

Saturday, October 22, 2011

LL Punts Me to JIRA

Got a reply back from LL support today. They said that I am not the only site affected by this problem, but gave me no details on that (which would have been extremely helpful) and that the solution is to file a JIRA. This is a significant blow and it looks like we are not out of the woods yet. I am disappointed that it took LL 5 days to tell me "file a JIRA". I have no idea how long it will take for them to investigate the new JIRA bug and fix it.

Today I tried one good potentical technical solution today (a fixed IP for you geeks out there) to no avail, that IP address is blocked by LL as well, so that didn't pan out. Plan B is to move to a new web host, which I am in the process of doing. With any luck they won't be blocked and the kiosks will just start working again.

I have created the JIRA bug and hope that all of you can go vote WATCH that bug (LL ignores votes, apparently!) to show that you care about art in Second Life and don't want to see this list and all the hard work I put into it over the years go down in flames. As well as my entire SL business and reason for being here :(

I am also sending out a personal message to everyone that owns one of my kiosks, so you may be receiving that as well, with an explanation and link to the JIRA to encourage everyone affected by this server bug to go vote on the JIRA. By demonstrating that this affects a lot of people I hope we can get this bug the attention it actually deserves. Thank you.

Friday, October 21, 2011

Linden Labs Shuts Me Down

Four days ago for reasons unknown, all communication between any scripted object anywhere on the grid and the IP address of my web application (that runs the Galleries of SL kiosks) at http://sasun.info stopped working. Testing shows that my web app is functioning perfectly fine and is in fact working for kiosks on the Avination grid. Avination is able to communicate to my web app, no problem. HTTP communication to other domains from the SL grid work fine. The only combination that does not work is http requests from SL to my website's IP address.

The only sensible conclusion is that either LL or my web host are blocking these http requests. A quick call to my web host confirmed that they do not block any IP addresses from their shared hosting IP addresses at all, it is up to their customers to block IP addresses on a doman-by-domain basis. In other words, if someone is trying to hack into http://sasun.info (which in fact just happened yesterday, unrelated to this case), I have to block any such offending IPs in my application myself. So... no blocking going on there.

So the obvious conclusion here is that LL is for some reason blocking all http requests from any and all of their servers (which means, the entire grid) to my web app. It may be possible that there is something in between that is blocking these, or some technical issue on LL's end that's not blocking them on purpose but some bad side-effect of the recent OS upgrades (hint, hint, LL) that's stopping http requests from working to my domain (and possibly many others, I have no idea).

All of this is perfecty understandable, though unfortunate. Software is buggy, problems crop up, upgrades cause problems, gremlins sneak into the wires and make things go crazy. I may be just the unlucky victim of bad luck somewhere between LL and my web host.

Or am I? It turns out I am not the only IP address that has been blocked by LL. One person I talked to had a very similar story and their solution was a lot of work on their part to move to a different IP address. For me, that would involve quite a bit of expense and work but is something I am going to investigate today since there is no way for me, or any of the "regular" support people at LL, to even tell me what the status of my issue is.

So here is the frustrating and critical failing of LL in this whole debacle - their unbelievably bad handling of this problem so far. As soon as I filed the case I jumped onto a chat session and was politely told that they would escalate the issue to from a normal "case ticket" to a "technical assistance" issue, meaning that it goes to the real support people that don't solve the stupid questions. Great! It's obviously critical to me and a bunch of other SL business owners so this should get fixed soon, right?

Days go by and I hear absolutely nothing. Every day I start a chat session to find out what is going on, and every time I tell them how critical this issue is and how vital it is that I get some kind of update or acknowledgement that it's being looked into. It turns out that the front-line chat support people have NO WAY to communicate with the "technical assistance" people working on any particular support case. From today's chat session:

You: It has been 4 days and affects well over 6,000 kiosks across the grid. Is there any way to get an ETA when this issue will be fixed? Every single scripted object (approximately 7,000) across the grid that communicates to my web site has not been working for 4 days. I've been in the dark this whole time.

RebeccaH: I do apologize but I don't have access to the technical support operations information.

You: so you guys are cut off from them, there's no way to update customers on what is going on?

RebeccaH: I'm sorry but that is correct, I don't have access to their information or a way to communicate with them.
Uh, what? Support team A has no way to communicate with support team B? Seriously, really, the live chat support people have no access to support information or any way to communicate with the technical assistance people?

Furthermore I was told that the technical assistance cases are unprioritized - there is no way to prioritize them. So every non-critical issue is in line with every severe "my business is down" case and is prioritized fist come, first serve. This is a disaster! Any support business in the universe needs to prioritize cases and get to the high-impact, "server down" cases first!

I can guarantee you that LL's own internal IT support department, for example, doesn't prioritize strictly by first-come-first-serve basis. So if an LL intern requests a different colored mouse from the IT department, he's ahead of the CEO whose video card died and is stopping him from getting his job done? I seriously doubt it!!

So why in heavens name would they refuse to prioritize customer cases and make the critical cases that are going to have wide-spread negative effects on the success of Second Life (there are products and content creators that falling into this category, obviously) higher priority? In effect what they are saying is that if LL stops your entire business from functioning, we don't consider that more important than anything else, so you'll need to wait a week for us to get around to it. Great!

I can't begin to express how frustrated I am with LL. I have been sweating over this the whole time, and working like crazy to supply customers with some level of functionality for their broken products. I'm obsessively checking to see if things are working again, since it drives me absolutely nuts to have so many customers unhappy and unable to use my products like this.

I have done some work to port my kiosk network to other grids such as avination and inWorldz, which so far as been very successful. This incident has convinced me I need to step up those efforts so that I can help them compete with LL as much as possible. Having a list of art galleries on avination, inWorldz, and other grids would be very cool and help them grow, not to mention put some products up for sale on those grids as well.

Furthermore I'm taking a hard look at my involvement in such things as the Linden Endowment for the Arts, which I put a rediculous amount of work into for no benefit of my own. I am boycotting any such voluteer work for a company that won't even give me the courtesy of a "hi we got your issue and are working on it, sorry about that" response until such time as this issue is fixed. If that is never, so be it. I don't know why I should work so hard to benefit a company that when the rubber hits the road, really doesn't care, and if they do, they have completely failed to show it.