mozilla

Mozilla Monolith

Note

monolith is still under heavy development

Mozilla Monolith is a project to provide statistic gathering, aggregation, a web-service API and a dashboard.

The first consumer of Monolith is the Firefox marketplace. Statistics include amongst others public global page views / hits, application specific downloads and even payment related information.

This aggregator part deals with gathering data from multiple sources, bringing them into a common format and storing them.

Currently data is stored in MySQL for durable archival and ElasticSearch is used provide the actual data access for the web-service and dashboard.

The web-service and dashboard are implemented in monolith.

There’s also a Python client library: monolith-client.

Here’s the high-level overview of the whole system:

https://raw.github.com/mozilla/monolith-aggregator/master/docs/monolith-big-picture.png

monolith-aggregator adds data to a MySQL database, using a command-line script called monolith-extract.

This script that grabs data from various sources and adds them in the SQL Database.

The script also indexes all the content in Elastic Search, which can be used to do time-series queries on the data.

The SQL database is the single source of truthe of Monolith. The Elastic Search can be recreated at anytime for any date range, using the SQL database.

A typical setup is to run monolith-extract every day as a cron.

Source Code

All source code is available on github under monolith-aggregator.

Further information

You an read up on some background and motivation.

There is also a detailed product specification available to the public.

We also keep some progress information in an etherpad.

License

monolith-aggregator is offered under the MPL 2.0.