Tuesday, March 30, 2010

Australian Government Spindoctoring on Internet Filter

Its is interesting reading Conroy's blustering with Google and defiance that the US Government never contacted the Federal Government, despite it being a matter of public record.

There are two possibilities as I see it - either he's truly ignorant, which truly paints a ghastly picture of the Federal Government's competence, or he is lying.

I'll let you pick.

- J.

Monday, March 29, 2010

Wall of Shame: Logging

Firstly, I wanted to call this Log Management - but then it occured to me that the point of this post is that few people actually log, let alone manage the logs at all. So I felt it was more prudent to identify the failure specifically. This is almost worse than than patching.
Lets assume for a second that the worst has happened. Your website owned and defaced. Your data stolen and sold to some state sponsored entity or criminal syndicate. You are in the front page news for all the wrong reasons. Your CIO is screaming down the phone and you have to sit infront of a governance board to explain what went wrong within 30min.

However there is one tiny problem (yes - over and beyond the above).

You have no logs. Nothing.

You thought you were logging - I mean most applications do right? Only as you investigate you uncover the following:
  • the Apache logs were never enabled due to "performance issues" (at least your Application Support team tells you).
  • the system logs were suspiciously clean.
  • even error logs, IDS, network logs (netflow, ACL/firewall logs, etc) all come up null. Again, suspiciously clean.
You have been told to start giving answers, only you have none to give.  

This is NOT a position you want to find yourself in!!!

Yet, you'd be amazed how often it happens. Or more specifically businesses (big and small) leave themselves susceptible to this sort of incident. Most people accept a bare minimum in their logging standards and yet don't realise their true purpose.

Despite being rooted in providing an audit trail of potential security events, they also help provide audit trails in general. In turn, fulfil audit and compliance requirements. Sarbanes Oxley is built on ensuring these sorts of checks are in place. However I'm not even talking about the level of compliance often required under Sarbanes Oxley. I'm talking basic logging of security events that is either never done or done so poorly as to be laughable.

I often see the centralised log collector used for all sorts of things. Shameful examples I have seen:
  • central log collectors used as bastion hosts by various admins (very common, mostly due to where these boxes are positioned on the network based on existing network segments);
  • the log collector used as a miscellaneous dumping ground (e.g. router configs, torrent files, etc),
  • a large portion of the IT infrastructure teams with full administrator access to the log collector;
  • directory services that did ZERO logging due to performance impacts and proprietary vendor lock in forcing the client to purchase a commercial product if they wanted security event logs;
  • ERP billing systems that did zero audits (total breach of SarOx) due to performance constraints and lack of vendor know how on what to implement let alone how.
This is just off the top of my head.

Everyone, logs are your friends. If you can't be stuffed patching, implementing proper application security, identity management, etc - then for gods sake, do one thing right: get you logs in order.

Some advice, in order of evolving maturity:
  1. Any logs are better than none. 
  2. Build a dedicated log collector, and nail that sucker down.
  3. Build dedicated processes around the management and response to those logs.
  4. Do you want more logs? 
  5. Think strategically
For anyone starting on this journey, I recommend you start with the NIST guide 800-92.

1. Any logs are better than none.
Yep. Its lame. But when we're talking dragging an organisation into the 20th century (yes, 20th) this is often a huge hurdle. Who will view the logs? How? Who will action them? Will the task be rotated to prevent staff boredom and lapse in response?

Yeah it sucks. I know. I've spent 8 hr shifts in the past reviewing logs. Believe me when I say I know the pain. You'd be amazed at how date/time stamps, IP addresses and port numbers blur in that time. I swear you can see ASCII art like Neo. But if you play your cards right, you can get through this phase early and quickly if you jump to phase 3 fast.

2. Build a dedicated log collector, and nail that sucker down.
You want a dedicated log collector for several reasons:
  • if the host is compromised, all data retained on it is suspect. however if logs are sent to a dedicated collector real time then the potential for them to be compromised is virtually nil.
  • in turn, this helps to preserve the body of evidence - particularly if legal proceedings are required.
  • it enables a more strategic approach in future (will get there later).
  • it makes security operations easier as there is a single repository for investigations.
Hardening is absolutely required because you need to ensure that chain of evidence. By that I mean:
  • The system should be sufficiently hardened in line with best practise (duh!),
  • It should reside in a fully segmented network zone, firewall, dedicated VLAN, etc. This could be a consolidated security operations DMZ.
  • User access should be restricted to security operations staff only. 
  • Backup/Archiving options should be in alignment with your data retention requirements as provided by legal council. I usually say 18 months as the 7 years is generally only required for financial transactions. I keep getting flip-flop advice from security professionals and legal eagles on this one. At the end of the day, I'll defer to the lawyers on this.
  • If you're worried about the hard drive filling up, just get as big a disk as you can and set the logs to rotate at a certain volume. If you're even more worried about clobbering your logs, see point 1.       
This host is going to be relied upon heavily and built upon over time. You may decide to do all sorts of funky things. Data replication, multiple nodes due to log volume, high availability

3.  Build dedicated processes around the management and response to those logs.
Ok, you are now collecting logs, have a central interface to view and search, what now? Its time to start thinking about what you want to do them.

What logs are you collecting? What events are you collecting?

In the early days you may lack any form of automation of any kind. You might configure syslog to give you the bare minimum (e.g. SU events, SSH logs - success/fail, that sort of thing). Some may disagree but I say that's fine. Anything more at this point and you're going to be overwhelmed and probably do nothing with them.  Start light and build up over time.

Get the events for Unix hosts sent to the Unix team. Windows events to the Windows teams. Don't know how to get syslog events for Windows hosts? Look at tools like Snare. Not sure what Windows security events are or where to start? Try here

Getting to phase 3 I think is the "bare minimum" when it comes to logging. And there's no shortage of open source tools to help either. If you do no logging today (or poor logging) this is your target end state for the moment.

Being the early stage, seriously don't worry about logs filling up too much. Try to undertake some capacity planning and forecasting but don't sweat it if you get it wrong. Eventually you'll have the data you need to make more accurate estimates but not likely at this initial stage. That said, this is early days and disk being as cheap as it is (assuming you're not using SAN just yet) then you'll be fine. If in doubt, remember rule #1!

4. Do you want more logs?
Finally you're at a point you can be more bold with your logging. You've covered your critical hosts? Check. What about IDS/IPS? All your hosts (Unix and Windows)? Web server/application logs? Authentication services such as Active Directory, Novell, RADIUS, TACACS+? Remote access services (Citrix or VPNs)?
I haven't even discussed integrity logging yet! If you can look at integrity monitoring/logging, please do so. Prioritise your critical hosts and go from there. This is something that really is neglected in a lot of environments yet I'm loathe to advocate it (despite how important I think it is) simply because of the number of businesses that aren't even here yet (i.e. still at stage 0!).

If you have finally reached this phase however, then this the part where you get to start to slowly and incrementally build upon your foundation.

Use a risk based approach. What aren't you logging that you really should? Equally important, how much more information can you take in without being overloaded? Are the processes currently working? If not, how can they be changed or otherwise optimised?

This is the point where you start hitting the upper limits of procedural and technical capabilities of your current environment. If you cannot take on any more events but know you need to, its time to take the next step.

5. Think strategically
Time to start thinking about how you want to handle security events on a larger scale. This involves a lot of discussions with various stakeholders - senior management, technical leads (network engineers, system admins, etc). How can you take in more events, ensure your compliance objectives and stay ontop of them as new compliance regimes come to the fore.

Here's some questions for you to consider:
  • How big is the security team at present? Do they have room to grow? Is there budget to allow it? If not can you build a business case to justify it? (Business cases for security staff are worthy of a post in their own right - but another day perhaps). If not, can other teams handle these events?
  • What be done to automate the response of security events? Can suspicious events be automatically responded to - further eliminating the need for manual processes? (e.g. you see 10 failed SSH login attempts to your SAP server at 1:04am and it isn't within a dedicated change window. you could argue its safe to auto-block and deal with the fallout later rather than having to manually investigate the host in the morning or being paged at ungodly hours).
  • Is event correlation a functional requirement (e.g. connect the dots between disparate logs). Are you willing to invest in an security information event management appliance to achieve this? This is a BIG question and perhaps starting to drill towards the heart of point 5. If you are thinking about it, you need to be mindful of what key technologies that a SIEM appliance must support? Also are you able and willing to spend the time learning how to use it, configure the appliance, tweak it and support it? This can be a 6 month+  journey for a given organisation. Tuning these things is often the most ghastly part of their implementation (but it does pay off in the end).
  • If you can't or won't move towards a SIEM, can you manage a reduced set of interfaces at the bare minimum with the logging information you require (e.g. anti virus, syslog events, IDS/IPS represents three consoles that would have to be checked regularly if not constantly). Try to factor in the cost of administering these machines on a daily basis as well into your justification of whether or not you can afford a SIEM. Reduction in staff overhead, licensing and admin costs is a cost benefit, I don't care what anyone else says!
  • Once you have implemented the solution of your choice, always review your processes and tech. Is it working? Are you reasonably sure you are capturing events? Are your security incidents showing events that simply are not being tracked by your SIEM (or newly optimised log management) appliance? Is there a new compliance regime with new auditing/logging requiremetns over and beyond your existing practises? If so, this is a sign that things need a change. However, if you are at this phase you are well poised to make the adaptations required.
If you hit step 5, pat yourself on the back. You're now 99.99% better than everyone else!

Apart from NIST 800-92 I also wanted to point people to Dr. Anton Chuvakin. He's done some awesome work on loggigng is also a champion of the security metrics space. If you're wanting some additional guidance and thought leadership on this subject, I suggest you check his material out as he's definitely the "go to" man on this subject.Even SANS says so. Also, since we're on the subject of logging, no post would be complete without a reference to the securitymetrics.org site. I haven't been as diligent in following this site as I would like - but I believe there is good work in progress here that we should all be paying mindful attention to.


- J.

Friday, March 26, 2010

Hitler & Cloud Security

This is a really accurate description of the problem with security in the cloud.

I know this has alrady done the rounds in the infosec community but for people who want to understand what the cloud fuss is about, go here.

- J.

PS: Next part in the Wall of Shame series is coming soon.

Wednesday, March 17, 2010

Are users right in rejecting security advice?

Pretty controversial topic. See here and here.

Reading the summary at first, I started getting angry. I mean lets face it, a user who ignores sage advice and continues to open every attachment blindly deserves whats coming. I mean its like a person with a trashed liver continuing to drink. It will be the death of them one day.

But then I realised that perspectives that challenge our own encourages growth. We examine our values and beliefs and what underpins them. Are our foundations made of bricks and mortar or are they a house of cards - built up on delusions, misinformation and misunderstanding?

Read on:

"The advice o ffers to shield them from the direct costs of attacks, but burdens them with increased indirect costs, or externalities. Since the direct costs are generally small relative to the indirect ones they reject this bargain. Since victimization is rare, and imposes a one-time cost, while security advice applies to everyone and is an ongoing cost, the burden ends up being larger than that caused by the ill it addresses."

Several thoughts come to mind on this.

Firstly, I have touched on in the past about pragmatic approaches to security. I immediately picture my 'holy grail' brethren preaching to the masses about the most strictest defenses, completely best practise and clearly not commensurate with the level of risk. Part of me admires their bluntness and ardour, but I digress.

However, comments like this clearly articulate that this approach is doomed to failure.

But I am oversimplifying  - Cormac Herley isn't saying that these recommendations are inappropriate. He's gone beyond that -- any expectations for the user to change their behaviour is doomed to failure.

Secondly, the equations in the document and the risk calculations are actually missing vital information. They make no reference to the actual impact to the user - purely a figure based on cost. Cost of remediation, cost on user time, lost productivity. Actually he

Mr Herley proceeds to explain the inconvenience on the user to change their password and make it longer for the perceived short term benefit, multiplied by the number of users. He clearly hasn't spoken to people who have had their accounts ransacked, lost hundrends of thousands of dollars or had their identity stolen, lost their business, etc. I bet you any money those people sure as hell learned from their mistakes.Whats more, I dare say they would place a much higher weighing to the risk calculation.

More than the actual calculation, the arguments are built on a faulty premise. The cost of security is high because of the need to provide assurance - trust - in our communications and electronic transactions. If everyone across the world tomorrow thought nothing they did could be trusted, e-commerce as we know it would die. Sure - many don't - but it isn't a universal belief. But it is human nature to trust. So playing devil's advocate, does this mean that the cost of providing that assurance is a waste of time? Is that the point he's making? Interesting possibility. Alas, these are intangibles aren't even acknowledged. To not consider or at least identify unquantified variables or areas for further research smacks of bad science to me. Conversely, trying to quantify benefit is also delusional. I mean lets face it - what is the benefit of security? Zero incidents? No investigative clean up costs? Wow. Ok. How is a zero benefit greater than a cost? Maths ain't my strong suite but clearly the quantification of a benefit needs as much work as the definition of cost.

Thirdly, with the above point in mind, if users are not doing the correct cost-benefit analysis then they do not understand the risk! Period - and guess what? This is an education problem!! Ok, so your users don't understand what could happen if they get infected with malware. Show some horror stories. I do believe that security education is a necessity - and I do agree with the argument that it does have diminishing value. But when it comes to articulating risks, clearly pointing out the full breadth and depth of what could happen - however rare - that is where security education really counts. And it this is where security education really matters.

If someone still chooses to accept the risk, knowing this and more importantly - you know they understand, then that is a personal decision. You - as their advisor - have done all that can be done. If however a decision is made in ignorance, not understanding, not caring or more importantly, dismissing your advice - then we must look at ourselves and ask "where did we go wrong? was it us or them? what could we do better?".

Fourthly, the examples are poor. His use of spammers neglect to mention that even moderately effective spam filters will prevent most spam from being delivered to an inbox. What is the direct cost on a user to delete it, as they would any other email they didn't care about?? A hundred percent of SSL errors are false positives>? Please! He didn't even cite where he obtained these figures or how they were formed. While MITM attacks are exceedingly rare in the overarching scheme of things (being honest - a drive-by compromise is far easier), they do occur. To put the blinkers on to this reeks of ignorance. The argument on phishing URLs being an imposition on user time?? This is analgous to telling people to look both ways before crossing the road is a waste of time.

"A main nding of this paper is that we need an estimate of the victimization rate for any exploit when designing appropriate security advice."

So in other words, he is really saying that we need evidence of true likelihood in order to quantify risk (which will enable us to define appropriate controls). Fair call. Unfortunately, this the hardest part of any risk assessment. Any anyone who tells who actually does this stuff in InfoSec will tell you - risk assessments - while vital - is the art of guestimation. Literally. It is the art of sticking your finger into the air and trying to figure out which way the wind is blowing and determine whether it will continue to blow that way. You provide me with stats on what the likelihood a desktop user will be hit with Conflicker, and I'll give you the true value of an asset and present you with a one hundred percent accurate risk assessment. Har har.

"Second, user education is a cost borne by the whole population, while o ffering bene fit only to the fraction that fall victim. Thus the cost of any security advice should be in proportion to the victimization rate."

So really, he's saying we need to include the net impact on the userbase of a control to calculate its true cost for implementation. Ok, I jive. What about benefit? Hrmm.

"Third, retiring advice that is no longer compelling is necessary. Many of the instructions with which weburden users do little to address the current harms that they face. Retiring security advice can be similar to declassifying documents, with all cost and no bene fit to the person making the decision."

Such as... ? There's a difference between old advice and just plain old bad advice. I've seen more of the latter than the former. Give us an example!

"Fourth, we must prioritize advice. In trying to defend everything we end up defending nothing. In attempting to warn users of every attack, and patch every vulnerability with advice we have taken the user (our boss's boss's boss) o ff into the weeds. Since users cannot do everything, they must select which advice they will follow and which ignore."

This is irritating on numerous fronts. It presupposes that users have a buffer that will overflow or clobber existing information if you tell them too much. While do I agree that users can make some dumb decisions, it persumes that that they really are lemmings - which is offensive. I don't argue that we need to take a risk based approach on prioritising our time and resources in protecting the threats that matter. However, users can learn. It might be an uphill battle and there will always be outliers or people who refuse. But that doesn't make it a waste of time.

My approach to security isn't to focus on user education. It is to make security so simple, so embedded, automated and ingrained into everyday processes that users have to go out of their way to do the wrong thing.

To be fair, the conclusion I found myself agreeing one hundred percent. Hey, at least they're reasonable. But are they practical? We do need to articulate the risks better. But we can only do that with the information at hand. Quantifying the likelihood of malware infection is certainly more difficult than calculating the risk of natural hazards (fire, flood, etc). Will we ever get there? Hard to say.

In any case, while the conclusion was sound, the math supporting it was terrible. The paper was rifle with poor conclusion, questionable methods and unsubstantiated claims with no regard for additional, influencing factors which could spur on further research.

Hey, at least I know my foundations are solid.

- J.

Friday, March 12, 2010

Wall of Shame: Patch Management

I like to compare patch management to car maintenance. Nobody questions the importance of car maintenance and it serves as a good metaphor. The car is a series of finely engineered, interwoven, interdependent parts that must run in perfect harmony in order to function. If something breaks down, wears down, doesn't run properly, then it can cause a chain reaction leading to a car breakdown if neglected. Usually if it is tended to early enough, it is a simple enough procedure to fix, a few hours in service and then its back on the road..

Now there are excuses why people don't perform it -
"I'm too busy to get it serviced right now."
"I can't afford it right now."
"I'm not paying a mechanic to do it when I can do it myself"
Blah blah blah.

But everyone knows the net effect if they do not get it serviced. Think of a cab. That car is on the road every day, usually 8hrs+ (most cabbies I've met work much longer hours btw) no less than 5 days a week. Do you think they get their car serviced? I'm sure they stretch out the times between maintenance as much as possible in some cases, but most would appreciate that the car is their livelihood and they cannot function without it. Cabs can have 300,000 kms, 500,000kms or even more on the clock. They don't last that long without a single car service.

But let's be real - that is why they get serviced. Its almost forced tax - extortion in a way - you pay this money regularly or your car dies. Everyone understands the consequeneces of not servicing the car.

Nobody has made that association yet with patching. Why not? Because nobody understands the full consequences of not patching.

Your IT systems are exactly the same - they are comprised of a series of finely engineered, interwoven, interdependent parts. They must run in total harmony in order to function properly. Likewise, so is a robust security architecture comprised of many controls to provide depth of coverage. If one of those controls breaks then it can lead to a total compromise of the directly affected system or application, other interconnected systems and applications and all your data. Security is only as good as the weakest link in the chain - after that it all collapses like a house of cards.

Here's what I've seen:
  • A Windows desktop environment for 1,000+ users remain unpatched for an 'indeterminate period'  because two Windows system administrators assumed the either had been doing it and never communicated it with each other.
  • In most cases though, its 50/50 whether Windows desktops get patched.
  • Windows servers are usually patched but again, flip a coin.
  • A Windows Group Policy that won't allow end users to enable Automatic patching of their desktops because central IT divisions want to "control the patch management process" - in this case an XP image with over a year of missing patches and service packs).
  • Solaris Servers when entered into Production are almost never, ever patched (in some instances 2 years+). Many of these Internet facing. I see this almost everywhere I look I might add.
  • Internet Explorer 6 used as the primary web browser within the enterprise for years without many several security patches because they broke with core company applications. Risk permanently accepted..
  • Network devices (routers, switches, firewalls, proxies) patched haphazardly. In some cases never.
  • Applications (e.g. desktop applications or even larger scale enterprise business applications installed on servers) are typically ignored.
The state of play now is that attackers are targeting applications increasingly so and targeting the desktop applications. Not just the operating system, not just the browser. Desktop applications.

See here.

This gives you an indication of what applications are being targeted TODAY. Lets take your operating system out of the equation for a moment. Are you running Adobe Acrobat? If the answer is yes, and you are not patching your applications then there is a strong chance of being compromised. Even if you are patching, guess what - even the tools you use for patching are experiencing issues.

If you want to talk real world examples and research on what are the practises of patch management, check out Project Quant on Patch Management as reported by Securosis (very illuminating btw):

The key findings from the survey are:
  • Most companies were driven by compliance regulation, usually
    more than one regulation applied
  • Process maturity was generally high for operating systems, but low
    for other asset types such as applications and drivers (see chart)
  • Companies tend to utilize multiple vendor and 3rd-party tools in
    their patch management process
  • 40% of companies depend on user complaints as one factor for
    patch validation
My interpretation of this report (bearing in mind that it was mostly US based, and certainly not representative of the security landscape of Australia!) -
  • It is not even the fear of their "car" (IT infrastructure) breaking down - it is the strong arm of the law and financial consequences that drive people to patch (i.e. when you have been mugged, beaten and left lying in a ditch somewhere because your car has broken down).
  • Acknowledgement that not enough attention is being paid to application patching.
  • Companies that are patching are using multiple tools to patch (duh).
  • Testing involves shoving in patches and waiting for user complaints - possibly into a limited test bed if you're lucky!
Being fair, I read this report and was somewhat suprised at how upbeat the document sounded. I firmly believe if it was done in Australia we would come across far, far worse. Our compliance landscape is not the same as the US. Likewise, Australians adopt a laissez-faire attitude ("She'll be right mate!") meaning that nobody really takes things seriously until the shit literally hits the fan.

If you think your patch management process is robust enough, ask yourself:
- Does your patch management include all operating systems and devices within your environment?
- Are applications both for server and desktop in scope for your patching regime?
- Can you say that all of the patches you pushed were applied?
- When patches fail (and they do) do you follow up on each instance to rectify the situation?
- Do you have proper asset management in place and are you able to ensure beyond a shadow of a doubt that no assets are left untracked and unmanaged?
- When a patch breaks a critical business application do you have a plan beyond rolling back the patch?
- Do you have a testing strategy that goes beyond user complaints when a patch breaks something?

If the answer is no to one or more of any of the above, you need to lift your game.

Lift it quickly too I might add before its too late.

- J.

SecurityFail series - the Wall of Shame

I was talking to a peer over at an OWASP chapter meeting recently about proposed talks I've got in mind. And since I got my soapbox right here, I've deciced I am going to start posting about my big gripes about basic security controls which I see as not in place or broken. Nothing tricky like reverse engineering of malware or 0 days.

Oh no.

Basic stuff that we are still doing wrong 20+ years on. Stuff that tells me that 0 days stuff means nothing when people don't keep basic logs, still use default admin passwords, haven't patched production systems for 2 years+. That sort of stuff.

It may sound boring to you but guess what, you better read it - your half a million firewall refresh, your new beaut enterprise anti virus and integrating vulnerability management solution - it means nothing. Sorry. Likewise, I appreciate reading about 0-days but it doesn't fascinate me when 99.9999% of all people are never going to be exploited with them. No. Most people are going to be exploited by their own laziness or ineptitude. Nearly everywhere I look, it is always the same and its time people start paying attention.

Risk management is intended as a way to prioritise our time and efforts, not fob off our responsibilities for running a clean ship indefinitely. But too often I see risk acceptance used as a corporate catch-cry for "can't be f**ked" or "TL/DR".

Yes, I use strong language. The molly coddling has gone on too long. If you are capable enough to read this blog then you are capable of hearing the truth.

Well guess what - my blog will soon become a Wall of Shame very, very soon. Not just to security folk like myself who haven't done our jobs right when we could have but to IT managers and admins everywhere. To the senior managers who ignore it. To the CIOs who fail to ask the right questions or honestly don't care to hear the truth. To people who make up excuses why they can't get shit right. For fear of breaking production, for insufficient budget, for overwhelmed staff, or because they just can't be f**ked.

Watch this space.

- J.