Dear Readers,
today we present you an interview with Victor Julien who is one of the creators of Suricata tool- an open source network IDS. He told us how does the tool work and about its new version. Enjoy reading!
[PM] Can you please tell us something about yourself and your team?
[VJ] I live in Amsterdam working from my home office. My team is spread across the western world currently. They are in Paris, Sweden, Germany, Canada, Boston and Indiana. In the past we’ve had people in many other countries as well. Like Mexico, India, Brazil, England, etc. We’re quite international, which I feel is great. We’re all learning a lot from each other. Not just about work, but about our cultures as well.
I’ve been in open source development for about 15 years, although the first years I didn’t make my first project, Vuurmuur, public yet. Vuurmuur (Dutch for Firewall) is a frontend to iptables. It was the project I created to learn programming and everything around open source and managing my own project.
From my firewall project I moved into Snort_inline, which was an attempt to turn the Snort IDS into a intrusion prevention system. Portions of that work, of which I just did a small part, were later integrated in to the main Snort. For some years I’ve been working as a contractor on Snort/Snort_inline related projects. That was also when slowly I started dreaming about doing my own project around IDS.
Nowadays I work full time on the development of Suricata.
[PM] What are your thoughts about cyber security nowadays?
[VJ] We’re in a pretty bad place. The technology we all use has many fundamental flaws and there are countless actors trying to take advantage to them. As law enforcement is still quite ineffective at this time, the bad guys are often getting away with it. It also means that for the time being, people and companies are mostly on their own. IT has spread into every part of our society, and as all IT is connected, it all has a security aspect. Where traditionally Infosec was about servers and desktops, these days it’s also about industrial control systems, cell phones, medical devices, cars, etc. The IoT revolution that is underway is going to expand this to almost everything, with refrigerators and children’s dolls as some of the recent examples. I think one of the larger issues is that networks are getting so complex that even small and medium sized companies are losing control. As most of the future of IT is about networking all those devices, I think tools like Suricata can be of great value. They can help give insight into the network and into the connected devices. At the same time we see governments slowly step up the efforts to help address the situation. In the Netherlands for example, we now have obligatory reporting of data leaks, including possible fines on leaks happening. It will mean companies and organizations will have higher incentives to try to prevent them and if they happen, detect them early.
[PM] Can you introduce Suricata to our readers?
[VJ] Suricata is an open source network IDS, or intrusion detection system. Even though the ‘system’ suggests a complete solution, it’s really an engine. A building block in a larger ecosystem, where 3rd party tools are used to manage the engine and it’s output. Suricata tries to detect bad and unwanted traffic by sniffing and inspecting network traffic using a rule language. The rule language is both a subset and superset of the Snort rule syntax. In addition to the rules language Suricata features a powerful lua based scripting language that can be used to express much more complex logic. The rules are not created by the Suricata team. There are various companies and open source communities working on creating those rules and detect logic.
[PM] How does it work exactly?
[VJ] Suricata reads packets from the network. It uses libpcap or more specialized and higher speed versions of the same concept, such as PF_RING or Netmap. Those packets are raw, so the first step is to decode them. Then we know things like ip addresses, ports, etc. The next steps are flow and stream tracking and reassembly. Packets that are part of a single connection share a common data structure. In it we track details of the TCP connection, reconstruct the TCP data. With this data we can decode and reconstruct the higher level protocols such as HTTP and DNS. The raw packets, TCP stream data, interpreted high level states are then all available to the detection engine. Various parts of the rule language inspect the different types of data. On the output side, a rule match leads to an alert. An alert can be written out in various textual and binary output methods. Next to the alerts, we log protocol information. For example for HTTP we can log each request and response, containing URL, User-Agent, etc. In 2.0 we introduced ‘Eve’, which is a unified JSON log for both alerts and the protocol logs. This is a very powerful feature, especially when combined with ELK or Splunk.
[PM] How did you came up with idea of creating it?
[VJ] I had been involved in the open source security community for quite some time, around projects like Snort, ModSecurity, my own Vuurmuur project, etc. It’s where I worked with and became friends with Matt Jonkman and Will Metcalf. While working and playing in this community I wanted to learn more about IDS and IPS. Next to this we wanted to try out some new ideas, like multi threading and protocol detection. At some point I simply decided to get started. So I started to create a multi threaded packet forwarder. At the time I called it ‘VIPS’, or Victor’s IPS. Then in 2009 Matt managed to get us funding, and suddenly it turned serious. We created OISF and came up with the meerkat logo and the Suricata name. I’ve been working on it pretty much full time ever since.
[PM] Have you got any difficulties with creating it?
[VJ] While creating it we’ve had plenty of technical issues, bugs, and design choices that we regretted later. I suppose that is just how development goes. The most challenging part of the development is the funding. Since we wanted the project to be open source, and to stay open source, we decided to organize it around a non-profit foundation: the Open Information Security Foundation (OISF). The foundation is based in the US as our initial seed funding came from the US government. After the government funding ended Suricata has mostly been supported by the industry. We’ve formed a consortium, where organizations donate to keep our development going. The current top supporters are FireEye and Proofpoint. Next to this, we’ve started a training program. We offer both user and development trainings. This gives us additional resources, although it also takes some development time away from the team. We’re looking at hiring dedicated trainers in the future.
[PM] Suricata is an open source project. How do you feel about sharing your work with others?
[VJ] Sharing is great in general. It’s one of the main reasons why I got excited about open source. I think it’s amazing how many people are together moving the development of tools and ideas forward. It’s also great to see how many tools are developed in the ecosystem. From frontends to honeypots, post processing pipelines, full distributions, etc.
That being said, as our project is gaining traction it’s sometimes also a bit frustrating to see how many people and especially companies are not contributing back. It’s something most licenses don’t require, but imagine how much more progress there could be if everyone contributed. I sometimes wonder if people realize how hard it is to get stable funding for something like an open source Suricata.
[PM] I’ve seen that you’ve just released new version of the tool. How much does it differ from the previous one?
[VJ] Yes, we just released Suricata 3.0. It’s been almost 2 years since our last major release, and we’ve improved it in many ways. We extended the JSON output in many ways. It can go into redis for example. We also added (net)flow logging, so that Suricata can now also be used in a flow analysis system. Performance was improved a lot, both raw throughput and scalability. With the help of the industry and community we were able to address many corner cases. Support for protocols like TLS, DNS and SMTP was extended, with file extraction support added for SMTP. The Lua detection support was much improved. Lua output support was added as well. So now you can create your own output formats using lua scripts. We also added support for multi-tenancy. This makes it much easier to use Suricata in more heterogeneous environments. For FreeBSD users we now support Netmap, a feature that was contributed by the community. We’re already seeing that being used by OPNsense.
[PM]Have you got any final thoughts? Is there anything you would like to add?
[VJ] One of the best things about running an open source project is the community. It’s great to see how people are helping each other, developing tools, etc. Last November we have brought the community together for the first time at our first user conference. It took place in Barcelona and it was a great success. We were planning to start with something like 40 people, but in the end over a hundred people signed up! We had a very exciting two days of talks, discussions and of course also food and drinks. The whole team came away from it with many new ideas and lot of energy.
So we’re already planning for the next edition. It will take place in Washington, DC, November 9 to 11. [http://oisfevents.net]
Social media:
Twitter: @inliniac
Blog: http://blog.inliniac.net
Github: https://github.com/inliniac
Suricata: http://suricata-ids.org
OISF: http://oisf.net/