AusCERT 2010 in Review

Well its the day after AusCERT 2010. I thought I would post some of my thoughts and feedback on the event since I now have my own soapbox upon which to rant. Having said that, its been three years since my last one and round #4 by my count. So I think I'm an AusCERT veteran by this point. But I would definitely welcome feedback/thoughts from others who attended.

The Good
There seemed more sponsors to AusCERT than ever before. Yay for AusCERT! Also, the Royal Pines is a great venue and the catering as always, is awesome. Please keep it here!

Several talks need to be shouted at as big wins in my book. I realise at least one or two of these may be controversial to some, I'd like to shout them out:

Stranger in a Strange Land: Reflections of a Linux Guy in Microsoft Windows , Cyber Intrusion: A Government Case Study , US Secret Service: Cell Phone and Embedded Technology Forensics , Black Hat, White Hat, Gray Hat, RedHat: What Dr. Seuss Forgot to Tell You About the Computer Hacker Community 2.0 , The Rules of the Internet, and the Browsers That Break Them , A history of Microsoft exploit mitigations , Professional Vulnerability Research and Analysis  (my personal favorite), John Stewart (CSO for Cisco - Closing Address).

These guys had really good presentations - everything from showing things previously unseen, treated as dark arts, providing unique insights into particular issues or even for just inspiring people to look at things a little differently.

Of particular interest to me was the Professional Vulnerability Research and Analysis talk. I also wished I made Setting the scene in vulnerability work but for some reason I wound up in the wrong room listening to a McAfee talk about APT (again.... APT... sigh...). Vulnerability Research is regarded as the blackest of black arts within infosec IMHO - and its really refreshing seeing industry professionals peeling back the hood and showing everyone what makes it tick. Chris, if you're reading this - I want to see a one day training seminar. Thirty minutes didn't do your presentation justice. Also, I'd like to call out Daniel Grzelak and Paul Theriault for their talk which highlighted some of the newer and evolving web security threats. With HTML 5 still being thrashed about, this talk gave a good highlight of things to come.

The Bad
The speakers were shafted. Straight out shafted - the end. Their "gifts" given very crappy looking metal wallets that were completely impractical for anything beyond storing cigarettes, with the AusCERT logo. I can't even say that they'd function as paperweights because they seemed too light. AusCERT need to lift their game and provide speakers decent presents. Would a bottle of wine and/or a voucher for a security book from the book stand be too much to ask?? On this count, I thought AusCERT were very, very cheap. Without the speakers, they have nothing. They need to recognise this and pay due recognition.

Secondly, the talks are way, WAAAAY too short. It seemed each slot was 35 minutes, with 5 minutes allocated for questions. When I see talks like Chris Spencer's (which is one of the most detailed talks given at the event) on a topic of great interest then it seems to me that they do the speakers and the audience an injustice. Black Hat goes for approximately an hour and 15 minutes, allowing plenty of time for setup and Q&A. I would suggest that AusCERT needs to find a happier medium.

Thirdly, the transport situation was unacceptable. As someone that didn't stay at the Royal Pines, there were only two buses that were departing from our hotel. A whole group of us (at least 8 from DD) were waiting at the bus at the correct time. When I tried to board I was promptly kicked off (presumably because it was full but I was never told directly why) at which point they drove off - with no bus forthcoming. This was a very crappy handling of the situation. I seem to remember a time when buses used to run regularly and this was never an issue. Why was it an issue now? I feel they should pay the $30 cab fair we incurred as a result.

Fourthly - and this is something that hasn't been vocalised before but I will come out publicly and state this. There has been an ongoing 'rumor' that AusCERT has always shown particular favor to talks given by sponsors. This has never been more prevalent than it was this year when five talks from Stratsec were accepted and one training seminar. SecureWorks were accepted for three talks. Seriously - WTF?

Now I will be the first to shout out how awesome the Stratsec crew are, having worked with a good chunk of them for a good length of time and proud to call at least two of those that presented my friends. They know their stuff and they had a wide array of topics, all timely, relevant and illuminating. But seriously, FIVE talks from the one sponsor? When talks are being knocked back by non-sponsors and five talks are selected from one, it is clear there is a problem. I would like to stress that this is a failing on AusCERTs behalf and certainly no reflection on their (Stratsec's) capability. In the past I know of people who have had their talks knocked back and yet, sponsors given the slot for presenting almost the same talk in a non-vendor slot. Also, I'd like to add that this criticism isn't because I didn't get to speak (so don't think it sour grapes). There were a number of factors at work on that front (and mostly my own fault):

1. I didn't present my synopsis by the due date (I was told by staff in Dimension Data I could use our vendor slot which would be guaranteed),
2. Dimension Data cancelled their sponsorship this year,
3. By the time I submitted my presentation to AusCERT, I was talk they'd already found someone to talk on my topic.

I also know of people whose talks were not formally accepted until the last minute and then given the worst timeslots. I personally don't have a problem with one talk in a sponsor slot and another in a non-sponsor slot. But to throw out too many slots to multiple sponsors in non-sponsor slots reeks.

AusCERT organisers need to lift their game, be more objective and more fair in assigning their talks. They also need to consider variety and people other than sponsors. While I could name names I certainly won't, but they know who they are. If they're reading, they should be vocal and voice their concerns to the powers-that-be.

The Indifferent
Attendees seemed to consist of mostly Queensland local councils and businesses. I did not see many businesses from Sydney or Melbourne unless they represented security vendors, consultancies or professioanl services firms. Some did come, and I did see a lot of the usual education, federal government, defence and law enforcement crowd, I just didn't see a lot of businesses. Is AusCERT losing its relevancy to these audiences? That said, the attendees were around the 1000 mark - which was about where it was when I last attended. Wierd.

Too many talks just plain sucked. In some cases, it was blatantly nerves/first time speaking. Can't blame that, those people deserve kudos for having the balls to step up and do something.

However, some people presented talks that blatantly did not match the description. Richard Steinnon's talk 'Titan Rain, the inside story of Shawn Carpenter' was a prime example. The talk stated (amoungst other things): 'Attendees will learn the methodologies the Chinese used to steal critical data on the Mars Lander and military data from US research labs. They will also learn the techniques the Shawn employed to back track the hackers and use their own tools against them.' This is a blatant lie. Certainly the author needs to share a measure of blame for an ill fitting description for his talk, but again, the blame rests on AusCERTs shoulders. Why wasn't this picked up in QA? Did any QA of the talks take place? I know that sponsor talks aren't usually QAed - but this only led to further questions - like, are sponsors that present in non-vendor streams QAed? What is the QA process? Again, yet another thing that must be fixed on the AusCERT side.

Part of my view is that too many of the talks are now becoming non-technical in nature. While I am certainly not as technical as some, I like to have a balanced view of things. In years prior there used to be a technical and non-technical stream. This is no longer the case. Now the talks seem to blend and in many ways, become more watered down. Is this a result of popular demand and market forces? Quite possibly - I simply don't know, but I certainly hope this trend isn't going to continue.

Also, while I like Max Kilger's talks and found him animating and engaging and the topics highly interesting, I found that he threw around numbers far too whimsically. He often did not clearly cite his references, how he arrived at these numbers and started jumping straight to conclusions. At the very minimum, he should have pointed out where a copy of the research could be found so the audience could see for themselves. While this is a nitpick of an otherwise brilliant view of things, I felt it smacked a twinge of bad science.

The Gala Dinner was apparently a let down (thankfully, I didn't attend). Dinner at the Royal Pines pales when previous years have seen it held at the Australian Outback Spectacular, Seaworld, Movieworld, etc. While Jimeon as the entertainment and the meal got the thumbs up, the entertainment afterwards got cained. While I must stress I wasn't present for this, I don't know anyone that did attend that liked it. However, take that for what you will.

Lunch on the last day was at a different timeslot to every other day. Umm... why??

Vendor sponsored events for two nights in a row also wound up at the same venue. Again, why?? This is not AusCERT's fault and I'm sure it was coincidence, but this was disappointing. Then again, it might have been that I was drinking there on the Sunday night, making Tuesday night day number #3 at the same venue...

Finally, the SMS feedback system - I don't think it worked out well over the paper system provided in years prior (although I hear they had a different electronic system last year). I say stick with pen and paper, or otherwise provice an online URL that people can access. Sod the SMS system.Charging people for their opinion is the height of rudeness.

Parting Thoughts
Seems like the sponsorship dollars were down this year on most. Yet, while there were seemingly more sponsors, the money didn't seem to go as far. Transportation was terrible, the entertainment lacklustre, quality of many of the talks seemed rather weak. Overall, the organisation just didn't seem up to scratch from previous years. I don't know whether it is a case of being spoiled in years prior or I'm just dreaming. Some of this I'm definitely not - infact most of what I've written here I can say is based on concensus feedback of numerous attendees I've spoken with. While I wouldn't say it was a failure or a waste of money (as there was value to be had) - there are many areas for improvement that simply need to be looked at urgently. 

I would go so far as to say that if my experience repeated itself again, I would probably not attend. Based on the attendees I saw, I also feel that there is a very real risk that AusCERT is losing its relevance to private industry. More effort needs to be maintained to ensure its relevancy, stronger support in local security professional groups (AISA, OWASP) and actively drumming up support for talks long before the official request for papers goes out.

I welcome feedback from anyone who attended on this - sorry for taking so long.

- J.