A year ago, I started itching. Having accumulated a bunch of great logs on an online payments project, I wondered – just what useful information I could extract from them?
Working on a consulting gig, I wanted a quick way to get something useful in front of my client. “Quick” then meant minutes – not days or weeks. It had to be Windows-friendly, on premises and useful even if the only installation went onto my development laptop.
Coming away from that search unsatisfied, I started thinking about the fundamentals of the problem. Did I really need a log parsing and storage tool at all? Most of the challenge seemed to be in taking ugly log data and reliably extracting information. If the log data could be made less ugly, perhaps I could just use one of the tools I was already familiar with, like CouchDB or even SQL Server?
Somewhere down that line Serilog was born. It’s been out in the wild for a little while, and feedback has been very positive. Serilog solves the problem of log-ugliness for me, while maintaining the low up-front investment characteristic of successful logging tools. I’ve been less satisfied by the available back-ends, though. NoSQL databases are a good fit for storing structured events and have a gentle learning curve, but don’t really satisfy my second “enterprise consultant” requirement: being general-purpose, it is still necessary to invest time in learning their APIs and actually writing some code before getting at the juicy data inside.
Seq (as in ‘sequence’), released today as a very early preview, is my effort to scratch the itch for myself.
Seq is just getting started, but it assembles what I think are the right ingredients to build on:
Runs on-premises, as a console app without installation, or as a Windows service (no IIS to configure)
Provides queries over structured data using a simple C#-like syntax, with full-text search when it is needed
Takes exactly one line of code to configure via Serilog, which we’ll see below
Is built with .NET from the ground up
What can you do with Seq today?
Seq is a fun way to centralize logs from networked systems. You can set up views (“The eCommerce Site”, “Intranet Apps”, …) based on properties embedded in log events. You can write and save queries with multiple structured- and text-based filters, and you can export the results in CSV format for analysis in Excel (my preference!) or your traditional BI tools.
And where is Seq going?
In the short term, there are a lot of performance, usability and feature requirements to implement before it will be ready for prime time. I hope Seq will stay simple as it grows, perhaps one day into the “developer’s tool of choice” for managing application events in .NET. First-class support for .NET programmability, and integration with the tools we use every day, are high on my list.
What Seq is not:
To clarify where Seq fits in, Seq is not an analytics tool. There are plenty of those out there, and chances are you can find one that meets your needs already. Seq is better described as a ‘staging ground’ for feeding those tools with quality data generated by applications.
Seq is also not “big data”. Big data capabilities come with trade-offs and usability issues that don’t make sense for everybody. Seq can theoretically handle terabytes of event storage, but takes a while to search more than a few hundred thousand events at a time (especially when ‘cold’). The number of useful information-level events emitted by an enterprise app in a week are generally orders of magnitude less than that; if you have big data you probably already know it, and have the motives and the resources to invest your time in learning something else.
Setting up – the client
To use Seq, you need to log events with Serilog. If you’re not using it already, you’re missing out! The Serilog site has some useful material to help you set up.
Seq provides a “sink” for Serilog, which we release via NuGet. It currently targets .NET 4.5 but support for .NET 4.0 is planned.
At the Visual Studio Package Manager console type:
PM> Install-Package Seq.Client.FullNetFx
Then, configure the logger and write some events:
using System;
using Serilog;
using Seq;
namespace SeqSimpleExample
{
class Program
{
public static void Main()
{
Log.Logger = new LoggerConfiguration()
.WriteTo.ColoredConsole()
.WriteTo.Seq("http://localhost:5341")
.CreateLogger();
Log.Information("Hello, {Name}!", Environment.UserName);
Console.ReadKey(true);
}
}
}
Run it now – there’s no Seq server listening, but the app will behave: as much as I love event data, collecting it can’t come at the expense of app stability.
Setting up – the server
The server requires .NET 4.5 and is tested on Windows 8. You should be fine with any .NET 4.5 machine. The web UI is tested with the latest Chrome, Firefox and IE10.
After downloading the Seq zip file from the site, extract it to a handy location – e.g. C:\Seq – and double-click the included Run.bat file. That’s it!
Open a web browser at http://localhost:5341 to see your (empty) event stream.
By default Seq will create a Data folder next to the executable. You can change where data is stored and the URL that Seq will listen on, as well as install Seq as a Windows service, by running seq.exe from the command-line. Type seq help for a list of options.
Now, go back and run your client application again. Refresh the event stream with the little ‘o’ button in the toolbar, and voila! If everything went well your event will be displayed.
(If nothing showed up, make sure you didn’t terminate the console app “hard” with the close button before the event was published – the Windows console kills apps hard.)
Using Seq
The Seq interface is pretty simple and should be self-explanatory in most places. The best way to get started is to type some text in the Filter box and hit ‘Enter’ or press the ‘Filter’ button. Once you’ve found an event of interest, click on it. If you hover over a property value, you’ll see a selection of actions like the ‘Mark’, ‘Filter’ and ‘Exclude’ buttons shown here.
(If you haven’t used Serilog before, look closely – not just static properties tacked onto the events, but the tokens from the message itself like CustomerName are available as properties for searching on.)
Chances are the first thing you’ll use on “real” event data is the ‘Exclude’ button – there’s always more signal than noise! In this case though, press ‘Filter’ to drill down to events with matching property values.
Seq queries use a syntax that should feel comfortable to C# developers and natural to almost everyone else. For example, big checkouts might be found with a query like:
"checkout" && TotalSpend > 500
To keep typing to a minimum, strings floating around on their own like "checkout" are treated as if they’re part of a Contains() function call. Pop over to the ‘doc’ tab in Seq to see some examples and a bit more information.
To export tabular data like one of the early screenshots show, use the ‘Mark’ button to pick out properties from events of interest.
Limitations
There are a lot! For example, event retention is fixed at 15 days (a proper retention policy implementation is in the works) and only one user account is supported (it is called ‘admin’). But, it is more fun to iterate in the open. Depending on the time available I hope to put an increment out every 2-4 weeks.
Licensing and all that stuff…
Unlike many of my other software obsessions, Seq is not an open source project. To follow through with the project is going to take time, so make that a reality a commercial release is more realistic.
The each preview build can be used for 30 days, after which I hope you’ll grab an updated preview build. At ‘RTW’ there will be a variety of licensing options that I hope will fit everyone who wants to use the app.
Feedback
I’d love to hear from you if you try Seq! Experiences good and bad are best shared on the issue tracker.
Thanks for help reaching this milestone go out to Jesse Beard, Jaben Cargman, and David Pfeffer who kindly provided feedback on the very first usable builds.