Jordi Boggiano
@seldaek
http://nelm.io/


Application monitoring with Heka and statsd

Belgian living in Zürich, Switzerland

Building the internet for 10+ years
 http://seld.be

Symfony2, Composer and other OSS contributions
 http://github.com/Seldaek

Working at Nelmio
 http://nelm.io
 Symfony2 & performance consulting

Application Monitoring

Why?

Complex applications

Many moving parts

Complete system is hard to test

Spot problems before they occur

What do you use now?

Nothing?

System monitoring?

collectd, cacti, nagios, ganglia, munin, ..

Performance/Business metrics?

statsd, sentry, ..

Error reporting?

newrelic, ..

Log aggregation and routing?

graylog, loggly, splunk, logstash, syslog, ..

Stats rendering?

graphite, nagios, ..

Too many tools

http://www.flickr.com/photos/stankuns/8738770989/

What would jesus do?






Jesus don't care, so what would mozilla do?

Enter Heka

Egyptian god of magic

«We've spoken to a number of folks, both inside Mozilla and at other well known tech companies, and everyone is doing pretty much the same thing: wiring up a number of different open source and paid service provider options into one pseudo-integrated system held together with duct tape and baling wire.»

Heka?

Inspired by logstash

Written in go: much faster and memory efficient

Self contained binary

Enables new use cases (no java, low footprint)

It's mostly glue

Architecture

Inputs: Reads in, receives, or polls for information

Decoders: Decodes info into a Message instance

Filters: Parse, process and aggregate Messages (emit new, absorb or passthru messages)

Outputs: Send out data, write out to logs/db, forward to other heka

Plugins: Go or Lua

Plugins

StatsdInput

ElasticSearchOutput

AMQPIn/Output

NagiosOutput

ProcessInput

...

Usage

TOML

Configuration

[packagist_log]
type = "LogfileInput"
logfile = "/var/www/packagist.org/app/logs/prod.log"
decoder = "monolog_decoder"

[monolog_decoder]
type = "PayloadRegexDecoder"
# Parses: [2013-11-21 12:34:56] request.WARNING Something funky!
match_regex = "^\\[(?P<Timestamp>[^\\]]+)\\] (?P<Channel>.+?)\\.(?P<Severity>[A-Z]+): (?P<Message>.*)"
timestamp_layout = "2006-01-02 15:04:05"
timestamp_location = "UTC" # optional, default value
            

Configuration

[monolog_decoder.severity_map]
DEBUG = 7
INFO = 6
NOTICE = 5
WARNING = 4
ERROR = 3
CRITICAL = 2
ALERT = 1
EMERGENCY = 0

[monolog_decoder.message_fields]
Type = "monolog_log"
Logger = "packagist.org"
Hostname = "composer"

Channel = "%Channel%"
Message = "%Message%"
Payload = ""
            

Configuration

[aggregator_output]
type = "TcpOutput"
address = "aggregator.example.org:5565"
message_matcher = "Type != 'heka.counter-output' && Type != 'heka.all-report' && Type != 'test'"

[debug]
type = "LogOutput"
message_matcher = "Type == 'test'"
            

ElasticSearchOutput + Kibana

Inspecting data

VersionEye

SilverStripe

Schweizerische Teletext AG ..?

statsd

liuggio/statsd-php-client

liuggio/statsd-client-bundle (Symfony2)

statsd

$statsManager->increment('packagist.search');
$statsManager->timing('packagist.request_time', $duration);
            
packagist.search:1|c
            

=> StatsdInput + StatAccumInput

=> CarbonOutput

Heka's Dasher

Composer Update Stats (Jan '14)

Composer Update Stats (Jan '14)

PHP Versions (All)

Total         4649683  100.00%
PHP 5.3.10     475979   10.24%
PHP 5.3.3      468551   10.08%
PHP 5.4.4      397824    8.56%
PHP 5.5.6      364280    7.83%
PHP 5.4.21     235730    5.07%
PHP 5.5.7      233908    5.03%
PHP 5.5.3      214623    4.62%
PHP 5.4.20     206360    4.44%
PHP 5.4.17     202147    4.35%
PHP 5.4.22     160533    3.45%
PHP 5.3.27     158662    3.41%
            

Composer Update Stats (Jan '14)

PHP Versions (Minor)

Total      4649683  100.00% (vs Nov 2013)
PHP 5.4    2160169   46.46% (-3.85)
PHP 5.3    1519506   32.68% (-4.6)
PHP 5.5     969927   20.86% (+8.45)
PHP 5.6         42    0.00% (=)
PHP 5.7         39    0.00% (=)
           

Composer Update Stats (Jan '14)

Operating Systems

Total      4649766     100.00%
Linux      3735847      80.34%
Windows     476825      10.25%
Darwin      425563       9.15%
FreeBSD      10134       0.22%
SunOS          580       0.01%
CYGWIN         422       0.01%
OpenBSD        136       0.00%
AIX            119       0.00%
OS400           32       0.00%
NetBSD          25       0.00%
            

Browser Visits (Jan '14)

Total: 231,725

Windows     99.517   42.95%
Macintosh   70.767   30.54%
Linux       53.396   23.04%
iOS          4.178    1.80%
Android      3.384    1.46%
            

Quick logging rant

Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\HttpException: "Media field is not one of ["electricity","heat","cold","water"] nor ["temperature","other"]"


Uncaught PHP Exception UnexpectedValueException: "Untrusted Host"


[LogicException]
A meter with existing precomputed measurements should have raw measurements before that date
            

Quick logging rant

throw UnexpectedValueException WITH the value

[LogicException]
A meter with existing precomputed measurements should have raw measurements before that date
            
[LogicException]
Meter #969 with existing precomputed measurements should have raw measurements before 2013-09-26 00:00:00
            

Back to Heka

Current State

Rough around the edges but getting there

Regular progress from the Mozilla team

Effective replacement for many things already

Upcoming Features

SmtpOutput (0.5)

LogstreamInput to improve LogfileInput (0.5)

TcpOutput reliability and security

Composable Dashboards

OpenTSDB / tcollector integration

UDP output with signing

JavaScript sandbox (maybe)

Find Out More

Thank you.

Questions?

jordi@nelm.io

@seldaek

slides.seld.be


Feedback:

http://joind.in/10263