On Saturday March15th 2014, the Firestorm team hosted a meeting and Q and A session to discussion the recent 4.6.1 release, provide updates on a number of issues, and answer audience questions.
While the meeting was recorded, the Firestorm team are aware that many of their users have hearing difficulties, and / or prefer to read text, so this transcript has been supplied on their behalf.
When reading, please remember:
This is not a word-for-word transcript of the entire meeting. While all quotes given are as they are spoken in the video, to assist in readability and maintain the flow of conversation, not all asides, jokes, interruptions, etc., have been included in the text presented here
In the interests of readability, topics in the transcript are not necessarily presented chronologically compared to the video. For example: questions asked during the various updates, etc., are presented in the Q and A section of the transcript, rather than at the point at which they were asked (unless directly relevant to the topic being discussed). Similarly, topics of discussion which came up during the Q and A session, but which were not tied to specific questions, have been placed under their own subject heading outside of the Q and A section
If there are any sizeable gaps in comments from a speaker which resulted from asides, repetition, questions to others etc,, these are indicated by the use of “…”
Timestamps are provided as guidance should anyone wish to hear the comments in full from any speaker on the video
Questions /comments were made in chat while speakers were talking. This inevitably meant that replies to questions would lag well behind when they were originally asked. To provide context between questions and answers, questions in the transcript are given (in italics) at the point at which each is addressed by a member of the Firestorm team, either in voice or via chat.
Please note: This transcript is provided for informational purposes only. I am not an official member of the Firestorm team, and technical or support issues relating to Firestorm cannot easily be addressed through these pages. Such requests for assistance should be made through the in-world Firestorm Support groups or at the Firestorm support region.
The TL;DR Summary
The following is a brief summary of topics discussed. Timestamps in braces refer to times in the video where the relevant commentary can be heard. All sections are expanded upon in the main transcript – click on the timestamp to go to them.
[0:0015] viewers are often subject to flase flagging by anti-virus programs as carrying a potential virus / Trojan. With the Firestorm 4.6.1, Norton anti-virus in particular had issues with viewer, prompting a positive response from Norton’s support
[0:14:32] Mac issues update: work is being done on some Mac issues within the Lab, but there is no major project to address problems some users are having. Firestorm are somewhat stymied in dealing with issues due to both a lack of developers / developers with free time and because some of the issues are beyond their ability to resolve
[0:31:00] Windows XP officially reaches its end-of-life on Aprial 8th, 2014. What does this mean for users on XP using Firestorm?
[0:38:25] Even running a 32-bit viewer on a 64-bit OS yields stability improvements, although if you have a 64-bit version available, it’s obviously preferable to use that on a 64-bit OS
[0:57:40] Firestorm are often critiqued on the frequency of releases. The team are moving to imporve things to a 3-monthly cycle, and there are reasons why a more frequent cycle may not be feasible
[1:21:05] It remains that Firestorm will not offer nightly or weekly builds, because there are significant support issues
[1:27:32] The team already try to release based on feature sets, however, a time-based cycle offers potentially better management of releases in keeping with the needs of the developers, QA and support
[1:35:21] The target will therefore be a 3-monthly cycle of major releases, with possible interim releases with bug fixes or for special features, such as might be the case with the group ban functionality
[1:58:53] With a target of a 3-monthly release cycle, it is probable that the next 2-3 releases are going to be primarily focused on incorporating features and capabilities coming out of the Lab, simply because there are so many of them: group bans, SSA updates, AIS v3, interest list, voice updates, etc.
[2:01:55] The new download server has performed admirably with not craches or other issues.
[1:55:40] Firestorm classes – with a new release just out, don’t forget there are Firestorm classes which cover all the new features, including things like the updated Contact Sets
Questions and Answers: including information on clean installs / re-installs; using settings back-ups; troubleshhoting issues; the status of voice improvements; why group limits are unlikely to increase in the near future; helping Firestorm support, etc.
With thanks, as always, to North for the video.
Norton Anti-virus and False Positives
0:00:15 Jessica Lyon (JL): Anti-virus false positives is where we’re going to start today; specifically Norton this time around. Every release, we end up seeing from our users when they’re downloading that it’s Avast or it’s Norton or it’s MacAfee, or it’s this or it’s that [anti-]virus that is saying that it’s our website that’s malicious or the binaries that you’re downloading are malicious and sometimes, even as the user is downloading them, the anti-virus sends it straight to their virus vault so they can’t even access it. and it’s not just our binaries, sometimes it’s the binaries which are shipped with our binaries, which is slvoice.exe slplugin.exe …
0:01:13 JL: So, I came in here with the intent to rant, because this release is getting flagged by Norton anti-virus something fierce this time around. but a very unexpected thing happened last night. But before I talk about that, I want to talk about why I think – and believe me, it’s not just Firestorm that gets hit with these false positives; Linden Lab does, Singularity – all of the viewers do. and I’ll tell you what my theory is on that.
0:01:47 JL: If you look at how the heuristics of how a Trojan works, generally a Trojan hitch-hikes on the back of another application and connects to the Internet separately from that application. so if you look at the way the Second Life viewer works, you’ve got Firestorm.exe as the binary, but it also ships with two other executables within it that connect to the Internet as well, separate from the viewer, which are SLvoice and SLplugin. And I think – it’s just a theory – I think that why our viewers often get a false flag is because the heuristics of that kind-of look similar to how a Trojan works, and I think that’s why the false flags happen. And I call the false flags because trust me, Firestorm does not have a virus. Our website is not in any way malicious.
0:02:56 JL: Now I was going to do a big rant towards anti-viruses. We talked a little bit yesterday, after the third-party viewer meeting with Linden Lab, and Latif was there, a developer for Singularity, creator of Radegast client. And I just mentioned that I wanted to just punch Norton Anti-virus in the face, and he mentioned that Singularity has the same problems, and he made a suggestion. He said that something he tried was to flame and shame – it was Avast for them – flame and shame … the anti-virus people through Twitter. So last night, we sent a Tweet – it wasn’t rude [below]
0:04:10 JL: Within just a few minutes, we had a reply on Twitter from Norton support, giving us some suggestions and apologising for the trouble.
0:04:22 JL: And I was pretty surprised as to how quickly that came. And they suggested that the fill-out these forms … Now the problem with getting whitelisted is that we have eight binaries now. We have three Windows, 2 Mac and three Linux. And the way those forms work, is you have to attach the binary in question to the form. Which means you’ve got to fill the form out, I think in this case, you have to fill-out the form for each binary, and I’d have to do this every time we do a release for Norton alone. And then I’ve got to do it for Avast and God knows what other anti-virus programmes are out there. I have to go around to all of them and do this form. And on top of that, it takes time before they actually approve it, because they get lots of these, and they don’t have like a billion people to go through and verify that these things aren’t whitelisted. and it takes something like 2-3 months.
0:05:25 JL: By that time, we’ve got another release out, and we’ve got to go through the whole process again because they’re only white listing that binary. So once you have an update, you’ve got a new binary, and you’ve got to start the process all over again. And so the solution to it is just not really feasible. You’d have to have somebody who is just dedicated to doing that all the time; filling-out these forms and applying for white listing.
0:05:53 JL: So even if they do white list it right away, their customers, the people who use these anti-viruses, have to download the new definitions; so that white list has to wait until Norton or whoever has an update available for their users and those users have to go in and download the update, apply it, probably reboot their machine. So it’s a pretty ugly process and really frustrating.
0:06:21 JL: So I was prepared today to have a lot of nasty things to say about Norton this time, but it’s been Avast in the past, and other programmes. Anyway, so we politely flamed and shamed Norton on Twitter and we got a response pretty quick, and the suggestion was to fill-out these forms. And I had to step out for a few minutes, and when I came back, Ed had posted into one of the support chats that an avatar by the name of Norton Support had entered the region, this region, I kid you not.
0:07:04 JL: The Norton representative created a Second Life account, logged-in to Second Life for the first time, on Firestorm, never been in Second Life before, found this region, teleported here hoping he could find one of us. I kid you not. Oh my gosh!
0:07:25 JL: So Ed was like, “You need to get here. Now!” … and I thought maybe this is just a hoax; other people watch Twitter, they probably saw it. anyway, sure enough, it was Norton, in Second Life. Twitter works. And I have to give credit where credit is due, Norton is now on top of my respect-o-metre in regards to anti-virus programmes, even though I don’t use Norton. That kind of customer service is remarkable. Especially considering we don’t use Norton. We are not a customer of Norton. They didn’t need to do that, but the guy came in, he pointed me to the forms, I filled them out, because he was helping me, I only had to submit one binary, which was the one that’s been hit the most, which is our Windows 64-bit.
0:08:24 JL: He analysed himself, on the spot, he analysed all of our binaries to try to give us some tips on why they’re being flagged falsely, at least with Norton. Obviously, he can’t help with the other programs. And so then I filled-out the forms, he gave me his e-mail address, I e-mailed him the tracking numbers for those submissions and as of [Saturday March 15th] every future binary ever that we put out is not whitelisted. We will never, ever have another problem with Norton going forward.
0:09:00 Ed Merriman (EM): So you can say what you like about Norton anti-virus folks, but damn their support is good.
0;09:05 JL: That’s incredible. I was surprised they’d reply on Twitter. I had no idea they would actually come into Second Life. I mean this guy was a newbie. He actually took it upon himself to find our website get the viewer, log-in, all that stuff. Incredible. Really impressed.
0:09:30 JL: So that fixes the issue with Norton going forward. It does fix the issue with Avast, Bitdefender, AVG, and Kaspersky, and Avast … So at the very least you have my explanation as to why all viewers in Second Life frequently get false-flagged by anti-virus programs. I think it’s because of what the heuristics of these anti-virus software look for sort-of matches the way our viewers work with having other binaries that piggyback and connect to the Internet behind the scenes, like voice and SLplugin …
0:10:39 JL: I can’t speak unequivocally that no viewer in SL is going to contain a virus. But Firestorm won’t contain a virus, or Singularity … if we were to transmit a virus to our users … it wouldn’t be one person out of three hundred, everybody would be flagged, and it would be all over the Internet that Firestorm is transmitting a virus … If your anti-virus is flagging it and you look in the groups and you don’t see other people en masse saying it’s been flagged, it doesn’t have a virus.
0:11:30 JL: And we have a page, we have a wiki page on white listing. so basically, if your anti-virus is flagging us or other viewers, you’re going to have to go into your anti-virus program settings and find where it is you can add exceptions and add the viewer as an exception. And you may have to do that with each release as well, because the binary is named differently from one version to another. And I might also suggest that you complain to your anti-virus company … And I should add, if you’re on Norton, you are going to have to do an update to get the most recent definitions, whenever they release them, in order for Firestorm not to be flagged. The e-mail says they’ve added it to their definitions, but the definitions haven’t actually been released … they may put a whole bunch together and I don’t know how long that will take, and the e-mail had no suggestion of what kind of timeframe that would be.
0:13:29 JL: Anyway, Kudos, Norton Anti-virus. Good job Tim – that’s the name of the rep.
[VirSCAN.org is a good place to upload to check against the latest a/v signatures.]
A Firestorm meeting with more of the team (stock)
Mac Issues and Firestorm Version Blocking
0:14:32 JL: What’s the future of Mac going forward since Cinder is no Longer with us? We do have Tonya, she’s a Mac developer, but Tonya’s availability is limited right now. But even if we still had Cinder, the issues that are plaguing Mac are more-or-less beyond our realm. most of them are related to Cocoa merge, were was after 4.4.2.
0:15:15 JL: We’ve brought this up many times now, almost every TPV meeting we have with Linden Lab we bring it up.
0:15:25 JL: So we’re going to be blocking version 4.4.0 in the next couple of weeks because of our stated rule that we’re only going to have three supported version on the grid at any given time. So 4.4.0 will get blocked and that will leave 4.4.2 as the next victim when we do our next release. And so Mac people who are plagued with these problems related to Cocoa, most of them are still using version 4.4.2 … it also doesn’t see Fitted Mesh and some other things… Materials. I don’t think it can see Materials, either.
0:16:08 JL: So some of the problems with Mac are, stability is a biggie, it’s crashing a lot. There’s a camera glitch with your ALT-camming, and various other problems. So we’re kind-of depending on Linden Lab to fix these, because they’re over our head and we don’t actually have a Mac developer who has the time to put into trying to fix these things. And the bad news is that all the viewers that have upgraded to Cocoa, and I think that’s all of them; I think Singularity has to, I could be wrong, I don’t know for sure. They all have the same problems.
0:17:01 JL: So Linden Lab has Mac developers, but their Mac developers are busy with other projects. I don’t think Linden Lab sees the Cocoa issues as being that bad. Oz [Linden] personally uses a Mac, he’s using the latest Mavericks operating system, and I think for most of the Mac issues, if you upgrade to Mavericks, are gone, except for the ALT-camming bug. I also understand that there are either people who cannot update their operating system or don’t want to for one reason or another.
0:17:44 JL: So I did some number crunching last night, and I took a look at our Google code page, which has all of our downloads prior to this one, and I looked at the 4.4.2 download numbers, how many downloads of Windows, Mac, Linux. It works out to about 9% of you use Macs. And that’s not a lot. And when it comes to prioritising with a big company like Linden Lab, they’re going to prioritise on the users they have the most of, which is Windows. So Mac is not going to be a huge priority. I wish I had better news for you guys.
0:18:35 JL: So I don’t know what else I can say to you guys, other than to file your bugs on Linden Lab’s JIRA. That JIRA is open now [but only for new bugs, unless the reporter of older bugs has opted to make their reports visible].
0:18:59 JL: This is one of those things that I think, that if it’s going to be fixed, Linden Lab need to see that there’s enough people having problems that justifies pulling a Mac developer off of whatever and focusing them on the Mac issues.
0:19:23 JL: How is [Firestorm] Mac development going with Cinder leaving? It’s not. Like I say, we’ve got Tonya, but Tonya’s availability is limited, she’s busy RL … As far as development goes, we don’t develop features specifically for Mac, features specifically for Windows, it’s more of a compatibility thing. We add-in a feature – we don’t even think about the operating system – we just add-in a feature and then we make sure that feature works on Windows, Mac and Linux. And so it’s not a matter of Mac development, it’s a matter of Mac compatibility with Firestorm. And as far as that’s concerned, we’re in sync with Linden Lab. All of our features work on Mac, aside from the bugs that you guys are suffering.
0:20:22 JL: I’ve noticed with this release, looking at statistics, I think it was less than about 5% of Mac people are downloading 4.6.1 – that’s even just downloading it to try it, less than 5% is actually trying 4.6.1. Probably most of you may be going back to 4.4.2 again because of the bugs. I just don’t know what to say to you guys. It’s a small percentage on Mac, which makes Mac not a priority with a big development company. So fingers crossed that things will get better. I’m sure they will.
0:21:10 JL: I did state during the [TPV Developer] meeting with Oz that we’re not going to block 4.4.2 until these major Cocoa issues are addressed. The unfortunate thing about that is doesn’t really solve much for you Mac folks, because you’re still not going to see Fitted Mesh, which I think will become popular over time, you’re not going to see Materials and whatever else is in the pipeline in the future. It’s not going to be an ideal thing, but I can tell you we’re not going to block 4.4.2 until the major issues with Mac are dealt with.
0:21:56 Lette Ponnier (LP): And also just to underline, it is not 4.4.2 that is being blocked at all right now, it is 4.4.0 [which is next to be blocked], which doesn’t actually have a lot of users anymore because 4.4.2 is the same but better. So 4.4.0.
0:22:18 JL: And I’m not promising when we’re going to that. I’ll say that if you’re using 4.4.0, I’ll say “why?” Because 4.6.1 is actually pretty good, 4.5.1 is pretty good … so not many people on 4.4.0, if you are on 4.4.0, get off, update to something newer. If you’re Mac, you may want to say with 4.4.2, we aren’t going to block 4.4.2 … until the major Cocoa issues are fixed.
What about Mac 64-bit?
0:23:49 JL: That was in progress, Cinder was working on that, she came across a few roadblocks. The challenge with making a 64-bit viewer is that you have to recompile all the libraries in 64-bit. Boost is a library that any developer working in Second Life hates with capital letters: HATES. It can be a real pain-in-the-butt. Cinder ran into some roadblocks with Boost and she wasn’t able to get past those roadblocks; we’re still hoping in the future we can get past them.
0:24:40 JL: Tonya is our only current Mac developer, and I’m sounding like a broken record, but she doesn’t have the availability she would like to have to continue on that Mac 64-bit path that we were hoping we would get to eventually. I will tell you that I’ll never say we’re never going to do Mac 64-bit anymore; it’s still on our planning board, it’s still something we absolutely want to do. But it’s not going to happen until we can get more availability from Mac developers or something like that.
0:25:19 JL: Linden Lab is very interested in 64-bit, by the way. They’ve been watching statistics and Singularity has 64-bit [Windows and Linux], we have it now; they’ve been watching both viewers very closely. They are very interested in it, Oz is pushing to get 64-bit support through Linden Lab. Vivox is apparently working on 64-bit voice, which Oz said Linden Lab would share with us even if Linden Lab doesn’t have 64-bit yet. They’ll share with us, so we’ll be able to have 64-bit voice for the Windows version, anyway … probably not Linux; Vivox isn’t putting much effort into Linux.
0:26:36 JL: So Linden Lab is interested in 64-bit and there’s a fair probably that Mac 64-bit won’t become available until Linden Lab does it unless things change with the open-source third-party viewers. Maybe somebody on Singularity will be able to do it, and maybe they’ll share those libraries with us, or maybe some other viewer; maybe they’ll have a Mac developer who can do that. Maybe Katharine Berry will be able to do it; she’s in university, I understand and busy with exams. but if there’s a person who can do that, Katharine Berry would be it, and Kat works on Exodus, for those of you who don’t know her. She’s a genius.
So don’t give up on it, it’s still going come, but it’s not going to come from us any time in the immediate future.
The 64 bit version seems really good except my frame rate is lower compared to the 32 bit version
0:27:34 JL: Some people report that, some people report higher frame rates, I think it depends on various things. Ansariel actually spotted something the other day in 64-bit through fast timers which might explain why some people might have a lower frame rate in windows 64-bit. I don’t know if she was able to look any further into that.
Ansariel Hiller (AH) via chat: Nicky had to disable some optimizations because of crashes and endless loops in 64-bit.
Gibson Firehawk, via chat: Nicky said on a JIRA ticket that it probably had to do with compiler flags she had to change for 64bit in certain parts of the viewer.
Question about voice user for Mac, you can’t do the same things as with Windows, replace the Vivox files with older versions? I always need to relog when I change my headset, and I’m not happy when I need to do that.
0:28:46 JL: Linden Lab has a release candidate with a new version of Vivox. We’ll probably make it available in the next week or two; we’ll make a wiki page for folks who want to try the new one. But Linden Lab hasn’t released it yet, so we’re not going to release it because it’s still being tested. Maybe the new issue will address what you’re talking about.
0:30:03 JL: If there’s one suggestion I could make to you Mac folks, it’s if you can update your operating system, update your operating system. Don’t be on 10.6 or 10.7. The older the operating system , the more issues you’re going to have.
Windows XP
0:31:00 JL: So I don’t know what the actual numbers are of users on Windows XP. I know that there’s enough of them on Windows XP 64-bit to complain that our Windows 64-bit doesn’t work for them. I’ll get back to that in just a moment.
0:31:33 JL: However many people there are on Windows XP, there’s not many of you, it’s going to be a pretty small number. But even small number, a small percentage in relationship to the amount of users we have still translates to quite a few people. And XP people are kind-of loud. You make yourselves heard when things don’t go your way.
0:32:05 JL: For those of you who don’t know, Windows XP support [from Microsoft] officially ends in April … And I do get that some people have incredible stability on Windows XP. I have not stats that I can look at myself, but Linden Lab assure us that by far, people on XP have the highest crash rate of any operating system.
0:32:41 EM: And I have something to say about that. I’ve had my alt on in excess of 200 hours more than once. so that’s a week at a time. So say what you want about the stability, it’s not true for everybody.
0:33:16 JL: So Microsoft is no longer supporting Windows XP. Are we going block Windows XP? Some of you may have noticed in the release notes the possibility of blocking Windows XP. I thought that might get your attention. If Linden Lab blocks XP, we’re going to block XP. But Linden Lab have said they’ve got no real interest in blocking Windows XP. They’re just going to let you guys suffer. The installer will still probably work for you, but it doesn’t mean the viewer is going to run well, it doesn’t mean that you’re not going to crash.
0:33:54 JL: By the way, if you’re running Firestorm on Windows XP and you’re crashing, you are hurting our crash rate. I don’t appreciate that, just sayin’!
0:34:09 JL: So we’re not going to block Windows XP unless Linden Lab does. We’re going to follow their suit in terms of what operating system they support and what they don’t, and so that way we can blame them, instead of being blamed.
0:35:09 JL: Firestorm 64-bit does not run on XP 64-bit. I am to blame, not Tank[master] Finesmith. Tank was pushing very hard that Windows XP 64-bit should be able to run Firestorm 64-bit. I wasn’t. And I’ll tell you why. You’re on a very old operating system. Please don’t expect new shiny to be delivered to you on an old, old, old operating system. I mean, how old is Windows XP? … Thirteen years? Upgrade your computer, guys. I know formatting and all that is a pain, but you can’t tell me that Windows XP is working that good for you.
0:36:37 JL: The Firestorm 32-bit will work on XP 64-bit. Always has, always will, unless Linden Lab decide to prevent that through the installer. Which is a possibility. It could happen. If you want to know for sure without a shadow of a doubt that you’ll always be able to use Firestorm, upgrade your operating system and you’ll never have to worry about it.
0:37:19 Tankmaster Finesmith (TF): I was just going to say Linden Lab have stated that of all the operating systems that they support, XP is by far the least stable of all of them, even [compare with] Mac and Linux. So updating really would help most people.
0:37:43 JL: I’ll tell you want else to. Don’t you dare come into our support groups or our blog post comments saying “Firestorm is really crashy and unstable for me and it sucks, and it’s useless and nobody should use it” if you’re running Windows XP. If you’re running Windows XP and saying all that stuff, you’re hurting us – and it’s your fault … please don’t do that.
0:46:06 JL: Here’s a link to the Third-party Viewer Meeting recording [at 24:42 and 28:00 in the recording - written notes here], and if you’re interested in Linden Lab’s point of view on windows XP, it’s there for those of you who want to look.
The Viewer and 64-bit Operating Systems
0:38:25 TF: Another thing that they say is that just using a 64-bit operating system even with a 32-bit viewer does offer significantly lower crash rates compared to the same version of Windows in 32-bit.
0:38:46 JL: Yeah, that’s interesting. Oz mentioned that as well. Just in our preliminary crash results from before we released 4.6.1, when it was in a preview group. Enough people in the preview group were on it that it showed-up in last week’s stats and the 64-bit has a significantly lower crash rate, really low. Although with the small number of people that was being counted, we expect that to go up. But it was significant. And Oz was suggesting that it may not be because the 64-bit version itself is stable, it’s that 64-bit operating system are stable. And now that we have a 64-bit version of Firestorm, people are downloading it, so we’re starting to see stats that are coming in that are separate from the typical Windows stats. Always before it was just “Windows”, how stable is Windows? But now we’ve got two versions, we’re setting two sets of numbers. We’re seeing Windows, and we’re seeing 64-bit.
0:40:04 JL: So it isn’t actually that the 64-bit Firestorm is more stable. It’s more a matter that 64-bit operating systems are more tolerant to things that would normally crash the viewer. So for example, all the viewers have memory leaks and whatnot, and we know the 32-bit has an out of memory crash, that at some point, you’re doing something and suddenly for some unknown reason the viewer requests more memory than the operating system can offer, and it goes Boom! It’s gone.
0:40:44 JL: With a 64-bit operating system and 64-bit Firestorm, that bug still exists, but you don’t actually run out of memory. so it’s not that we fixed the bug in 64-bit, it’s just that it’s more tolerant and won’t necessarily reach that crash point. And so it seems more stable, and for all intents and purposes it is more stable, it’s not that it’s any different. It’s just that 64-bit is more tolerant. And 64-bit operating systems are more tolerant to the memory crashes that the viewers have.
0:41:27 JL: So you can use 32-bit viewers on a 64-bit operating system, but I would suggest that if you have a 64-bit operating system, use the 64-bit viewer. Unless you need Havok for pathfinding or mesh with physics.
Firestorm Release Cycles
0:57:40 JL: This is a sensitive topic within our team. There are two sides to this, and I’m going to try to be fair to both sides.
0:58:17 JL: We do get people that complain that our release times are too long. And for those of you who don’t know, we’re averaging about four months between releases. That’s what a release cycle is, it’s the time between each update. We’re averaging every four months. And a lot of people are complaining, what’s the matter with us, why would we want to do release cycles so far apart…
0:58:53 JL: I can tell you unequivocally, no matter what side we’re on in the team – there’s a side that would like to do a release every week, and there’s a side that would like to do releases less frequently than every week. But I can tell you that all of the sides agree that every four months is too long. Our release cycle of every four months is not by choice. We’re not aiming to do a release every four months. It would be nice to do them – my side of the fence is every two months; if we can do a release every two months, that is the happy medium between four months and no months.
0:59:38 JL: And I’ll tell you why. And I’ll tell you that Ansariel, and there are people like Ansariel, wants to do a release every week or two … So let me make it clear: none of us are happy that our releases are taking four months. None of us; not even our support team. nobody is happy that releases are taking four months. It’s is too long, we all agree … so if you’re one of those people who have been complaining, please don’t come to us complaining why we’re choosing to do a release every four months. We’re not. It’s just working out that way. and it’s interesting that it happens to work out almost exactly four months every time. I don’t know how that happens.
1:00:28 JL: So there’s only so much we can do. Yeah, we’ve got some 80 people on the team. probably ten or twelve of them are developers. Of those developers, how many do you think are actually developing? If we had all twelve of our developers developing, we could be doing releases really quickly.
1:00:59 JL: We’ve got Ansariel, who does a ridiculous amount of work, just look at our repository. We’ve got Tank, who handles merges and compatibility issues and things like that. We’ve got Nicky, who handles server stuff, crash fixes, major, major issues that are too complicated for most of us, we’ve got Zi Ree who does some work, we’ve got Tonya who does some work; Tonya’s not been available, Zi Ree’s not been that available. And Tech, and Cinder was doing tonnes. We don’t have cinder now.
Some of the Firestorm dev team from top left): Nicky Dasmijn, Ansariel Hiller, Tonya Souther; (lower) Zi Ree, Techwolf Lupindo
1:01:49 JL: How much do you think we can get done with so few developers, in a period of time? So I’ll tell you what happens. When we put out a release, generally the devs take a little bit of a break. They’ve worked hard and they deserve a break … so that’s two or three weeks where nothing is happening; we’re not getting anything done.
1:02:23 JL: During those 2-3 weeks our support team is hammered, crazy busy, going out of their minds, rage quitting because they’re burnt out, they can’t handle it any more – keep in mind support have to deal with all the people who are pissed off and feel entitled and all these different things … I was called some pretty nasty names in an offline because we didn’t provide the opportunity to install into a different directory for 64-bit. Which, by the way, yes we did. You’ve just got to look under the little button that says “Options”, because installing it into a different directory would be an option.
1:03:13 JL: Anyway. Support deals with people like this all the time. And it’s really bad when we do a release. Really, really bad. It’s when support gets hammered and we lose people on support because they just can’t take it anymore. Can you imagine if we did a release every two weeks? Or month?
1:03:37 EM: Every two weeks, and you wouldn’t have a support team.
1:03:40 JL: Exactly. And that’s my point. So, I have to balance out, we have three main departments on the team. We’ve got developers, we’ve got QA and we’ve got support. and all three of these departments have a different idea of what would work best for them, and my job is to choose one that works best for everybody but nobody’s actually fully happy with.
1:05:11 JL: If we did a monthly release, developers are going to get three weeks to work, QA is going to get one week to test. Is that going to be enough? … No … And that’s assuming QA find nothing wrong with the three weeks of work that the devs did, but I can assure you they will. so what happens is that in that one week of testing, they find these issues, and then we’ve got to get the developers to go in and fix the bugs. Let’s say that takes a week. Then another build has to be sent to QA, and they have to test those fixes. And quite often they find problems with those fixes, so then the next week the devs have to do some more work, and then QA has to do … you guys starting to see the problematics between development and QA? It gets complicated, it gets very difficult to get a release out without bugs. and I’ll get back to the fact that I’m really picky and want to do perfect releases, because that’s going to come up, and it’s certainly a valid point against me.
1:06:58 JL: So there’s the fact that there’s development time needed, and then there’s QA time, and as far as development time goes, with the few devs that we have who are active, we’re not going to get much done. How much do you guys like the idea of having updates, say, once a month with very few changes? How many of you would update if you didn’t really have anything cool to update to? We’re talking about maybe a feature and some fixes. Kind-of like an Agile process, little changes. Because shorter release cycles means you get less bang for your buck to download and install.
1:08:14 JL: This is a complicated topic because there’s a lot of different sides and a lot of different valid opinions that go against each other, and I’m certainly not going to say I’m right with my opinion.
1:08:44 JL: Let me give you Ansariel’s argument, which is a valid argument for more frequent releases: with less work done, there is less to test, and less impact on the users, and therefore less support. And that’s a valid argument … I don’t know if it’s true or not; to be honest, we’ve not tried it. I personally disagree to an extent, but I can’t say it from experience … Let me ask support, if we did monthly releases, how would they feel about that?
1:09:35 EM: They’d be burnt-out within two releases. When it comes to release time, we enter what I call the two weeks of hell. We have two weeks of total insanity, and it really is nuts … Let me ask you this, folks. How would you like to answer the same question 16 times in a hour, and they deal with someone who is really upset because there’s a bug affecting them terribly … so support goes through two weeks of hell after a release.
1:10:39 EM: People complain when we send them to the wiki, which has got all the answers; we get complaints every week, “Why do your support staff just send me a link to the wiki?” I’m sorry, folks, with the chat lag in the groups, we throw the links out, you can read it there; if we were to post it into the group, it would be a wall of text and it might not go through.
1:11:13 JL: So let me ask you this. Ed and Lette are our two main support folks who manage the rest of the support team, which is massive. How do you feel about a release every four months, currently what we’re ending-up averaging… I think you guys think every four months is too long as well. Am I right?
1:11:55 EM: Yes and no. It’s a lot of work leading up. For those who do the documentation, there’s a lot of work to be done, and we can’t do it until we’ve got a final build because we don’t want to go back changing everything over and over again.
1:12:16 JL: But again .. Ansariel’s argument, and the argument on that side of the fence, that if we do more frequent, then there’s less documentation changes that you have to do.
1:12:27 EM: That would cut down on the documentation. The other side of me would like more frequent releases, simply because it would be less work that way and less changes and fewer questions from the users. The problem being, every release we get, “Oh, what’s the piece of crap you’ve put out? The last version worked so much better!” Then the next person comes in and says, “Oh my god! this is absolutely wonderful!” How many of you remember the blocked release [4.4.1, June 2013] that was out for a day? Remember that? There were two changes [with the 4.4.2 release]. A bug fix and a debug setting switched off.
1:13:29 JL: And I’ll tell you what, those two things have no impact on anything.
1:13:35 EM: People said [of] 4.4.1, “oh it’s great!”. They get 4.4.2, and they say, “What’d you do?! It’s a piece of crap!” It was the same viewer, but going from one release to the other, “this one’s a piece of crap!” or “That one was a piece of crap, this one’s great!”
1:14:10 LP: The clean install topic is one of the reasons … we could probably be more frequent than every four months, but it’s a reason why a super-fast release cycle would create chaos. Because folks would come in, they’d have a problem that could have been caused by an unclean install and might be fixed by a clean re-install, but if they don’t know what that entails, and honestly, many people who think they’re doing clean installs actually aren’t. So they have this issue, and they think they did a clean install, they decide to roll-back to the previous version and after they do that, they still find that they still have the problem and they’re very upset.
1:15:04 LP: If we had a faster release cycle, people would probably be bouncing back and forth between different versions a whole lot more, trying to figure out which one works best for them because they’re experiencing a problem and not realising that their issue was largely caused by the way that they’re doing the installs. So we’ve got settings back-up now, which ought to make clean installs a lot easier and faster, but we still see these issues come up. So that would be one of my main concerns.
1:15:39 LP: Another one is trying to keep track of the versions. Firestorm is already a very complex and versatile viewer. You can make it work in so many different ways. you’ve got different log-in modes, you’ve got skins that put things in different places, and some don’t have the same things … We try our hardest to keep track of all those different things, and then when we’ve got different version and we’re trying to remember which bugs exist in which version so that we can tell people, “oh, you should be on this one, it’s a bug caused by the one that you’re on,” and so forth; it’s just a lot to keep track of. And we’re not going to be able to provide the best support we can possible give if we have more that we are actually responsible for remembering in that area.
1:16:47 JL: So, on the side of the fence that feels we can do much more frequent releases because there’s less work going into it, there’s fewer changes that need to be tested with less impact to the end user. With Chrome, Firefox, QuickTime updates, Flash updates, that is true. I mean, do you even notice when Chrome makes an update? It’s bam! It’s done, You just restart it, you don’t even think about it … when it comes to applications like that, that’s a different story. Second Life viewer updates are far more complicated than they should be … they should not be as complicated and varied as they are. Changing that, though, is quite non-trivial and I don’t know how updating your Second Life viewer could be make easier … like updating your Chrome or Firefox or whatever it is. But it isn’t. It’s complicated, it’s convoluted, it’s a royal pain in the butt for the user to do.
1:18:17 JL: Because it’s so difficult and such an awkward process, people get the placebo effect that the previous version was better than the current version or vice-versa; that whole example that Ed explained about 4.4.1 is a perfect example of placebo effect … you only thought it was different, it really wasn’t different. So every month just isn’t reasonable and it isn’t really fair to support. Every four months is not reasonable to developers, because developers want to get their stuff out sooner. Son my job in everything we do, is to try to find some comfortable middle ground.
1:19:08 JL: I would like to do a release every two months … That’s what I would like us to be able to put out; I think that’s fair. Fair for support, fair for devs, fair for QA, and fair to the users, because updating is a pain in the butt for you to. So I think every two months.
1:19:32 JL: Believe it or not, we’ve been aiming for that for a year now, and we still manage to hit an update every four months. Why is that? Well, I don’t have pay cheques to give QA, I don’t have pay cheques to give beta testers and I don’t have pay cheques to give to our developers and I don’t have pay cheques to give to our support people. we are all volunteers, and as I say, after a release, developers want to take a break and, generally speaking, they deserve to take a break, they’ve worked hard; especially with the few that we have that are active. But that break that they’re taking means that nothing is getting done.
1:20:16 JL: Now if this was a paid business, and everyone was being paid, you don’t get to take a break. You get a vacation every year. But if you’re making 60-80 grand a year, you’re expected to be working from eight in the morning until five at night and get your job done and have results. Frequently. And lots of them. And if we ran in a model like that, they we could do like Linden Lab does, have a new release out every [two] week[s], and we could force our users to update to that release.
Firestorm Nightly Build Offerings
1:21:05 JL: So … Public nightlies … The problem with that. Singularity does that. I think they have weeklies, don’t they? Weekly alphas? Singularity doesn’t have a support team. Linden Lab doesn’t do that, and if they did, Linden Lab has a support system in which the only way you can really get a hold of them is through support tickets, and support tickets can be quietly ignored. What other viewers do nightlies or weeklies? I can assure you that whoever they are, they don’t have a dedicated support team. We are the only third-party viewer in Second Life that provides live support, in group, unfiltered, ad-lib, we don’t know who’s coming next, we don’t know who’s going to ask what question. And that puts us in a different situation to any other kind of viewer project in Second Life. Because while we can say, “here’s a nightly, it’s completely unsupported, please don’t come to our support team asking for help with it, because we can’t support it, because it’s an alpha. we’re not going to answer your questions.”
1:22:35 JL: We can say that until we’re blue in the face. we can have bars across the screen when they’re running that viewer, which say, “Do not come and ask us questions or for help, because this is a nightly, it’s unsupported, it’s probably broken” – and it would be probably broken, I can assure you, if it’s nightly – do you think that’s going to stop people coming for help? I can assure you it won’t.
1:23:20 JL: So in our situation, we can’t really do public nightlies. We actually do nightlies internally. and so today Ansariel will commit some code that she’s been working on – speaking hypothetically -but this does happen all the time. A developer will commit half the work – because it’s not finished – but she wants to get it into the repository so that other people don’t write over it, for example. but it’s not finished. And if you go and download that nightly, you’ll have some floater panel that has some really cool-sounding feature, but it doesn’t actually work. and if that’s a public nightly, people are going to download it and say, “oh, what’s this really cool feature….oh, it doesn’t work… I’ve got to ask somebody why it doesn’t work. Where can I go? Who can I ask? Oh, I know, Firestorm has this public support group anybody can go in and ask!” And they’re going to come in and ask, and I can guarantee you that support are going to say, “What Are you talking about? … Where is that in the viewer?” And they’ll say, “It’s under this menu”, and support will say, “Well, I don’t have that in my viewer. Where are you seeing that?” And they’ll end up spending ten or fifteen minutes trying to work out what the hell you’re talking about before they realise you’re on an alpha that is not supported.
1:25:00 JL: And you have all the good intentions. You don’t mean to come in and cause headaches for support. you just have this question about this feature. You’ve got nobody to ask if you’re on a nightly. Don’t ask. But people will ask, and that’s why we don’t do nightlies. We just can’t support them, and saying we can’t support them does not stop people from coming in en masse and asking questions.
1:25:42 JL: Nightlies in the beta group. Beta is not an unmoderated group. Out QA runs it strictly, and rightly so. And they do a fantastic job, because more-or-less, we want beta testers to be testing specific things. If we just give them builds, and don’t get them to test anything specific.
1:26:20 JL: Let’s say for example, we’ve got half-finished code in the viewer and we give the beta testers a nightly. The beta testers are going to go in and say, “what’s this?” And they’re going to ask in group … and probably, QA won’t have a clue, they won’t know the answer, and all the beta testers are going to come in … all asking the same question to which we have no answer. You can imagine how frustrating that can become. It’s kind-of the same situation. Our model just doesn’t work that way. I’m not saying it’s the right model or the best model … but it’s just not going to work for public nightlies or even weeklies.
Releases Based on Features?
1:27:32 JL: Instead of doing releases every four months, what about doing releases based on the number of changes that get done in the viewer, once we hit a threshold of new stuff. that should be how we determine whether we do a release or not. Actually, to be honest, that is kind-of what we actually do. It just turns out to be every four months! It should be quicker.
1:28:06 JL: And how do we measure new features? For example, let’s use the SLShare really cool feature. There are some people in this audience who feel that is absolutely worth a release. And there are people in this audience who absolutely believe that it is completely and utterly useless to them. SLShare, by the way is the ability to link your SL avatar to your Facebook account…
1:26:45 JL: So how do we weigh that? How do we weigh what is and isn’t worth a release? It’s not even much of a question. We pretty much know what most people want in a release and we can more-or-less weigh-out what’s worth releasing. What we had a month ago was worth releasing. But we wanted to wait for HTTP, because that is really good, and had we not waited for it, and had a release without HTTP and then HTTP would have been out, and everybody wants HTTP. And then something, and something else, and something else.
1:29:23 JL: Why does it take us so long to keep up with Linden Lab? I’ll tell you why, in part. Because Linden Lab works on these things in secret. when they start to get a little bit closer, we might found out they’re working on it, and we might find out a little bit about it. But we don’t see the code, we don’t know what’s in the code, we don’t know how much code there is going to be. Linden Lab only makes the code available to us right before they put it out to release candidate, more-or-less. Sometime sooner, sometimes later. but we don’t see that code until Linden Lab is ready to release it into the wild.
1:30:07 JL: That doesn’t give us a whole lot of time to get it into our viewer and test it, and work out some of the bugs. no offence to Linden Lab, but Linden Lab … has a different threshold on what they think is ready for release versus what we think is ready for release. and that’s nothing against Linden Lab; Linden Lab has fewer users than we do, Linden Lab doesn’t have the kind of live support that we have. so they can release with big bugs and it won’t impact them as much as it will if we release with big bugs. We release with a little bug that affects one percent of the user base and that’s more than 10,000 people. We have to think about that.
1:31:01 JL: And so I’m really strict about what passes QA. And I can assure you that Takoda and Kate are even more strict on what passes QA. I have to supersede them sometimes and say, “OK that’s an annoying but, but we’re going to have to release with it.” And so again I have to find the comfortable middle ground that nobody is happy with, but they’re a little bit more happy than if I went to one extreme or the other.
1:31:38 JL: So QA is really important, we can’t just push out stuff untested … so that’s where that is.
1:32:42 EM: There’s been a suggestion made that we could do a full release every two to three months, and throw out a bug fix in between … no new features, just bug fixes on the in-between … that’s more work for the devs, though.
1:33:19 JL: It look like most people are OK with every three months.
1:33:24 EM: If it was bug fixes plus the regular stuff going on, they’d have to have two separate repro going on, one for the bug fixes … it’s a great suggestion, but do we have the manpower to do it?
1:33:43 JL: Well, the interest. Again, our devs aren’t being paid; I can’t tell them, “do this, or you ain’t getting paid…” I have to entice the developers with catnip or small mammals if they’re werewolves, depending on what creature they happen to be.
1:34:21 JL: the point I’m trying to get to is that every four months is definitely too long … and we are aiming for every two months. And if we aim for two months, we’ll probably manage to get a release out every three months. So that gives our devs two months, well maybe a month and a half and it give QA about a month … I think that’s fair. and support deals with a month and a half of hell, another month of recovery, and then preparing for another release. I think three months is suitable.
1:35:21 JL: So I’m going to tell you on the record, I try not to make statements that I regret. We are going to try to do a release every three months. That is going to be our target. We may have little minor releases in between, even.
1:35:38 JL: So, for example, we’re working a little bit with Linden Lab on group bans, the ability to moderate who can enter an open enrolment group so you can get rid of spammers and stuff, permanently without closing your group. So this is a really big feature, and it’s one of those features that everyone is going to kind-of need to adopt at the same time or thereabouts. Linden Lab is trying to get that done as soon as possible and released, and when they do, our plan is, providing it suits our quality standards and works the way we feel it should work, which is just to say that it’s useful, and not circumventable, we’ll issue an update which will probably be 4.6.2, that will have 4.6.1 code plus group ban code, and maybe a couple of little blocker fixes that came up in the 4.6.1 release.
Likely Features for Upcoming Releases
1:58:53 JL: If we’re going to succeed in a release every three months, say in the next year, we’re going to have to focus most of our efforts on merging-in the labyrinth of new stuff that Linden Lab is pushing out. And it’s all going to start coming out at the same time, most likely. And a lot of it is unstable still, even according to Linden Lab. So probably our focus for the next 6-8 months is just going to be trying to keep up with Linden Lab’s things.
1:59:30 JL: We’re talking the second part of the sunshine work, the avatar appearance stuff. We’re talking the interest list stuff, which is huge. We’re talking group bans, which is epically awesome as a feature. So don’t count on a lot of our own new features, unfortunately. I’m sure we’ll have some on the planning board, I’m just not sure we’re going to be able to get any included.
The Firestorm Download Server
2:01:55 JL: I am, you may have noticed, pretty happy today. And it’s not just because I think 4.6.1 has been relatively a good release aside from some very loud minority, it’s been a really solid release, I’m really proud of the team. My biggest concerned about this release was the download server, and it performed flawlessly. Absolutely beautifully. It was under a pretty heavy load for a while, 600Mbps, something like that. It was taxed for a while, but considering it’s not actually a download server … it’s not built specifically to send files. I didn’t actually get an e-mail from Hetzner, who we’re getting it from, saying, WTF are you doing to our server? Which I kind-of expected, as well.
[At the time of the meeting there had been over 102, 500 downloads from the server, with over 50% of them the windows 64-bit version, with the windows 32-bit version running a "close second", with Mac at approximately 9% and Linux some 2-3%.]
2:03:05 JL: Everything went flawlessly, and the download server is there to stay. Well worth the money.
Firestorm Classes
1:55:40 Ed issues his usual encouragement for people to attend Firestorm classes, which are held at the Phoenix Firestorm Support region classroom and at Junkyard University. Class times are scheduled throughout the week at different times to meet the needs of different time zones, and recordings of classes can also be obtained on the Firestorm YouTube channel. These include classes on new and updated features, such as the <a href="h