Jordi Boggiano
@seldaek
http://nelm.io/


Application monitoring with Heka and statsd

Belgian living in Zürich, Switzerland

Building the internet for 10+ years
 http://seld.be

Symfony2, Composer and other OSS contributions
 http://github.com/Seldaek

Working at Nelmio
 http://nelm.io
 Symfony2 & performance consulting

Application Monitoring

Why?

Complex applications

Many moving parts

Complete system is hard to test

Spot problems before they occur

What do you use now?

Nothing?

System monitoring?

collectd, cacti, nagios, ganglia, munin, ..

Performance/Business metrics?

statsd, sentry, ..

Error reporting?

newrelic, ..

Log aggregation and routing?

graylog, loggly, splunk, logstash, syslog, ..

Stats rendering?

graphite, nagios, ..

Too many tools

http://www.flickr.com/photos/stankuns/8738770989/

What would jesus do?






Jesus don't care, so what would mozilla do?

Enter Heka

Egyptian god of magic

«We've spoken to a number of folks, both inside Mozilla and at other well known tech companies, and everyone is doing pretty much the same thing: wiring up a number of different open source and paid service provider options into one pseudo-integrated system held together with duct tape and baling wire.»

Heka?

Inspired by logstash

Written in go, pretty fast

It's mostly glue

Architecture

Inputs: Reads in, receives, or polls for information

Decoders: Decodes info into a Message instance

Filters: Parse, process and aggregate Messages (emit new, absorb or passthru messages)

Outputs: Send out data, write out to logs/db, forward to other heka

Plugins: Go or Lua

Plugins

StatsdInput

ElasticSearchOutput

AMQPIn/Output

NagiosOutput

ProcessInput

...

Usage

TOML

Configuration

[packagist_log]
type = "LogfileInput"
logfile = "/var/www/packagist.org/app/logs/prod.log"
decoder = "monolog_decoder"

[monolog_decoder]
type = "PayloadRegexDecoder"
# Parses: [2013-11-21 12:34:56] request.WARNING Something funky!
match_regex = "^\\[(?P<Timestamp>[^\\]]+)\\] (?P<Channel>.+?)\\.(?P<Severity>[A-Z]+): (?P<Message>.*)"
timestamp_layout = "2006-01-02 15:04:05"
timestamp_location = "UTC" # optional, default value
            

Configuration

[monolog_decoder.severity_map]
DEBUG = 7
INFO = 6
NOTICE = 5
WARNING = 4
ERROR = 3
CRITICAL = 2
ALERT = 1
EMERGENCY = 0

[monolog_decoder.message_fields]
Type = "monolog_log"
Logger = "packagist.org"
Hostname = "composer"

Channel = "%Channel%"
Message = "%Message%"
Payload = ""
            

Configuration

[aggregator_output]
type = "TcpOutput"
address = "aggregator.example.org:5565"
message_matcher = "Type != 'heka.counter-output' && Type != 'heka.all-report' && Type != 'test'"

[debug]
type = "LogOutput"
message_matcher = "Type == 'test'"
            

ElasticSearchOutput + Kibana

Inspecting data

VersionEye

SilverStripe

Schweizerische Teletext AG ..?

Heka's Dasher

Composer Update Statistics (Nov 2013)

Composer Update Statistics (Nov 2013)

PHP Versions (All)

Total         4112760  100.00%
PHP 5.3.10     490350   11.92%
PHP 5.3.3      419871   10.21%
PHP 5.4.20     387450    9.42%
PHP 5.4.4      274741    6.68%
PHP 5.4.9      198343    4.82%
PHP 5.4.16     180150    4.38%
PHP 5.4.19     167416    4.07%
PHP 5.5.3      166317    4.04%
PHP 5.4.17     160754    3.91%
PHP 5.4.21     144939    3.52%
PHP 5.3.26     131497    3.20%
            

Composer Update Statistics (Nov 2013)

PHP Versions (Minor)

Total      4112760  100.00%
PHP 5.4    2069021   50.31%
PHP 5.3    1533073   37.28%
PHP 5.5     510596   12.41%
PHP 5.6         62    0.00%
PHP 5.7          8    0.00%
            

Composer Update Statistics (Nov 2013)

Operating Systems

Total      4112819     100.00%
Linux      3263686      79.35%
Windows     441016      10.72%
Darwin      393557       9.57%
FreeBSD      13476       0.33%
SunOS          511       0.01%
CYGWIN         407       0.01%
OpenBSD         61       0.00%
NetBSD          35       0.00%
OS400            8       0.00%
AIX              3       0.00%
            

Browser Visits (Nov 2013)

Total: 141,882
Windows 59,877      42.20%
Macintosh 44,154    31.12%
Linux 33,638        23.71%
iOS 2,255            1.59%
Android 1,714        1.21%
            

Statsd

liuggio/statsd-client-bundle

Quick logging rant

throw UnexpectedValueException WITH the value

Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\HttpException: "Media field is not one of ["electricity","heat","cold","water"] nor ["temperature","other"]"

        
Uncaught PHP Exception UnexpectedValueException: "Untrusted Host"


[LogicException]                                                                              
A meter with existing precomputed measurements should have raw measurements before that date  
            

Quick logging rant

[LogicException]                                                                              
A meter with existing precomputed measurements should have raw measurements before that date  
            
[LogicException]                                                                                           
Meter #969 with existing precomputed measurements should have raw measurements before 2013-09-26 00:00:00  
            

Back to Heka

Current State

Rough around the edges but getting there

Regular progress from the Mozilla team

Effective replacement for many things already

Upcoming Features

SmtpOutput (0.5)

TcpOutput reliability and security

Composable Dashboards

OpenTSDB / tcollector integration

UDP output with signing

JavaScript sandbox (maybe)

Find Out More

Thank you.

Questions?

jordi@nelm.io

@seldaek

slides.seld.be


Feedback:

http://joind.in/10365