The only sensible conclusion is that either LL or my web host are blocking these http requests. A quick call to my web host confirmed that they do not block any IP addresses from their shared hosting IP addresses at all, it is up to their customers to block IP addresses on a doman-by-domain basis. In other words, if someone is trying to hack into http://sasun.info (which in fact just happened yesterday, unrelated to this case), I have to block any such offending IPs in my application myself. So... no blocking going on there.
So the obvious conclusion here is that LL is for some reason blocking all http requests from any and all of their servers (which means, the entire grid) to my web app. It may be possible that there is something in between that is blocking these, or some technical issue on LL's end that's not blocking them on purpose but some bad side-effect of the recent OS upgrades (hint, hint, LL) that's stopping http requests from working to my domain (and possibly many others, I have no idea).
All of this is perfecty understandable, though unfortunate. Software is buggy, problems crop up, upgrades cause problems, gremlins sneak into the wires and make things go crazy. I may be just the unlucky victim of bad luck somewhere between LL and my web host.
Or am I? It turns out I am not the only IP address that has been blocked by LL. One person I talked to had a very similar story and their solution was a lot of work on their part to move to a different IP address. For me, that would involve quite a bit of expense and work but is something I am going to investigate today since there is no way for me, or any of the "regular" support people at LL, to even tell me what the status of my issue is.
So here is the frustrating and critical failing of LL in this whole debacle - their unbelievably bad handling of this problem so far. As soon as I filed the case I jumped onto a chat session and was politely told that they would escalate the issue to from a normal "case ticket" to a "technical assistance" issue, meaning that it goes to the real support people that don't solve the stupid questions. Great! It's obviously critical to me and a bunch of other SL business owners so this should get fixed soon, right?
Days go by and I hear absolutely nothing. Every day I start a chat session to find out what is going on, and every time I tell them how critical this issue is and how vital it is that I get some kind of update or acknowledgement that it's being looked into. It turns out that the front-line chat support people have NO WAY to communicate with the "technical assistance" people working on any particular support case. From today's chat session:
You: It has been 4 days and affects well over 6,000 kiosks across the grid. Is there any way to get an ETA when this issue will be fixed? Every single scripted object (approximately 7,000) across the grid that communicates to my web site has not been working for 4 days. I've been in the dark this whole time.Uh, what? Support team A has no way to communicate with support team B? Seriously, really, the live chat support people have no access to support information or any way to communicate with the technical assistance people?
RebeccaH: I do apologize but I don't have access to the technical support operations information.
You: so you guys are cut off from them, there's no way to update customers on what is going on?
RebeccaH: I'm sorry but that is correct, I don't have access to their information or a way to communicate with them.
Furthermore I was told that the technical assistance cases are unprioritized - there is no way to prioritize them. So every non-critical issue is in line with every severe "my business is down" case and is prioritized fist come, first serve. This is a disaster! Any support business in the universe needs to prioritize cases and get to the high-impact, "server down" cases first!
I can guarantee you that LL's own internal IT support department, for example, doesn't prioritize strictly by first-come-first-serve basis. So if an LL intern requests a different colored mouse from the IT department, he's ahead of the CEO whose video card died and is stopping him from getting his job done? I seriously doubt it!!
So why in heavens name would they refuse to prioritize customer cases and make the critical cases that are going to have wide-spread negative effects on the success of Second Life (there are products and content creators that falling into this category, obviously) higher priority? In effect what they are saying is that if LL stops your entire business from functioning, we don't consider that more important than anything else, so you'll need to wait a week for us to get around to it. Great!
I can't begin to express how frustrated I am with LL. I have been sweating over this the whole time, and working like crazy to supply customers with some level of functionality for their broken products. I'm obsessively checking to see if things are working again, since it drives me absolutely nuts to have so many customers unhappy and unable to use my products like this.
I have done some work to port my kiosk network to other grids such as avination and inWorldz, which so far as been very successful. This incident has convinced me I need to step up those efforts so that I can help them compete with LL as much as possible. Having a list of art galleries on avination, inWorldz, and other grids would be very cool and help them grow, not to mention put some products up for sale on those grids as well.
Furthermore I'm taking a hard look at my involvement in such things as the Linden Endowment for the Arts, which I put a rediculous amount of work into for no benefit of my own. I am boycotting any such voluteer work for a company that won't even give me the courtesy of a "hi we got your issue and are working on it, sorry about that" response until such time as this issue is fixed. If that is never, so be it. I don't know why I should work so hard to benefit a company that when the rubber hits the road, really doesn't care, and if they do, they have completely failed to show it.