Showing posts with label debugging. Show all posts
Showing posts with label debugging. Show all posts

Monday, July 6, 2009

My Guess on Symfony 2

After I read this tweet from Fabien I was left thinking on how Symfony 2 will be. Then I
remembered this presentation from Fabien where he talks about the new framework. I took a look at it and then I decide to glue the pieces together to get something working out of the
code given in slide 28.

So to start with it, I needed an application to build. Since some days ago I'm playing with MongoDB, a document oriented database. To learn how to use it I built a centralized logger for symfony applications. The idea is that if we have in production twenty machines serving symfony and then we need to parse the logs to find possible errors, etc., it will be nice to have a tool that centralizes the logs in one place. Since this database is lightweight and fast, I wrote a simple logger to store the messages in a MongoDB database instead of using the normal file logger.

After I got the logger working I needed a way to display and search through the logs. Initially I built a symfony application that was able to filter the logs by priority and by some words in the log message. The feeling I got was that a full symfony 1.2 project was too much for such a
simple web app. This was the perfect excuse to experiment with Symfony 2.

The Logger

The MongoDB logger is just a simple symfony logger that stores for every symfony log an array with this structure:

 

$log = array( 'type' => $type, 'message' => $message,
'time' => time(), 'priority' => $priority) );

The idea is to provide a form to issue queries to the database to filter the logs by any of those fields. i. e.: I type sfRouting and I should see only those logs that contain that word in their messages.

Here's a screen shot of the final application:

So, how to build that using Symfony 2?

First we create the folder structure like this:

 
-/
--/ apps
--/ config
--/ lib
--/ web

Inside web we place the index.php file which has the following content:


define('ROOT_PATH', dirname(__FILE__).'/..');

require_once ROOT_PATH . '/config/sf_requires.php';
require_once ROOT_PATH . '/config/app_requires.php';

$app = new LogAnalyzer(); $app->run()->send();

There we define the root path, and then we include two configuration files, one which will take care of requiring the Symfony libraries and the other that will require the application files.

Then we instantiate the application class and we run it.

Now let's check what's inside the sf_requires.php file:

define('SF_LIB_PATH', ROOT_PATH . '/lib/vendor/symfony/lib');

require_once SF_LIB_PATH . '/utils/sfToolkit.class.php';
require_once SF_LIB_PATH . '/utils/sfParameterHolder.class.php';
require_once SF_LIB_PATH . '/event_dispatcher/sfEventDispatcher.php';
require_once SF_LIB_PATH . '/request/sfRequestHandler.class.php';
require_once SF_LIB_PATH . '/request/sfRequest.class.php';
require_once SF_LIB_PATH . '/request/sfWebRequest.class.php';
require_once SF_LIB_PATH . '/response/sfResponse.class.php';
require_once SF_LIB_PATH . '/response/sfWebResponse.class.php';

First we define the location of the Symfony libraries and then we proceed to include the required files.

The only new class here is the sfRequestHandler which Fabien describes in his presentations, the other ones I just took from a symfony 1.3 distrubution.

With those files included, we are done with what refers to symfony, then we have to include the application files. So the contents of app_requires.php will be:

require_once ROOT_PATH . '/lib/dba/MongoLogReader.class.php';
require_once ROOT_PATH . '/lib/dba/CollectionModel.class.php';
require_once ROOT_PATH . '/apps/LogAnalyzer.class.php';

Besides the LogAnalyzer class we include two classes that will take care of querying the log database. As we can see, the LogAnalyzer class will reside under the apps folder and then others under lib/dba.

So now let's check what's inside the LogAnalyzer class.

 
public function __construct()
{
$this->dispatcher =
new sfEventDispatcher();
$this->dispatcher->connect('application.load_controller', array($this, 'loadController'));
}

On instantiation we create a new instance of the sfEventDispatcher and we connect our application to the application.loadController event which will be fired by the sfRequestHandler::handleRaw() method. There we tell it that the loadController method of our application will process the request.

Then we have the run method:

public function run()
{
$request = new sfWebRequest($this->dispatcher);
$handler = new sfRequestHandler($this->dispatcher);
$response = $handler->handle($request);
return $response;
}

Here we initialize a sfWebRequest object to start parsing the request parameters. Then we instantiate our sfRequestHandler and we call the handle method. The handle method will return a response object, which is the one where we call send() in the index.php file to output the response to the browser.

When the sfRequestHandler start to do it's job it will fire the application.load_controller event for which we set up the following listener:

public function loadController(sfEvent $event)
{
$event->setReturnValue(array(array($this, 'execute'), array($this->dispatcher, $event['request'])));
return true;
}

There we say that the method execute of the LogAnalyzer class will take care of generatiing the response data out of the request.

And finally the code for the execute method:

public function execute($dispatcher, $request)
{
$response = new sfWebResponse($dispatcher);
$response->setContent($this->render($this->getTemplateValues($request)));
return $response;
}

There we instantiate a sfWebResponse. The content of this one will be the result of the proteced method render. –Here I must say that is possible to create our own View class to
handle this part, but for this example I preffer to build it like this–.

protected function render($values)
{
extract($values);
ob_start();
ob_implicit_flush(0);
require(ROOT_PATH . '/apps/template.php');
return ob_get_clean();
}

This method expects an array with all the variables that will be used in the template. This values are extracted from the array and inserted in the current scope by calling the extract function. As we can see, the template is a simple php file that is required from the apps folder. This file is plain PHP code embedded into HTML.

The getTemplateValues method take will get the data out of the database, plus interpreting the request:

protected function getTemplateValues($request)
{
$values = array();
$values['sf_request'] = $request;
$values['collections'] = $this->getCollections();
$values['priorities'] = $this->getPriorities();
$values['cursor'] = $this->getLogs($request);
$values['pageNumber'] = $request->getParameter('page', 1);
$values['cursor']->skip(($values['pageNumber'] - 1) * $this->maxPerPage)->limit($this->maxPerPage);
$values['hasMore'] = $values['cursor']->count() > ($this->maxPerPage * $values['pageNumber']);
$values['filterParams'] = $this->buildFilterParams($request);
return $values;
}

And that's it! Which such a simple structure we can leverage the power of the sfRequestHandler class which will be at the core of the new Symfony version. We know that symfony does very well for complex projects, but sometimes I felt like it was too big for a simple application like this one. With this new component I think that this distinction will be gone.

THE CODE:

The application code and the sfMongoDBLogger class can be found here.

You will need to setup a virtual host in order to run this application.

RESOURCES/REQUIREMENTS:

To learn more on MongoDB refer to this website: http://www.mongodb.org
If you setup MongoDB as explained here http://www.mongodb.org/display/DOCS/Getting+Started, you should be able to run this project without problems –Give it a try, MongoDB is pretty easy to setup and the documentation is very good–
For the installation instructions of the PHP native driver go here

IMPORTANT:

Even if this should be implicit, keep in mind that this are my personal views on the subjects. This is by no means an official statement from the Symfony project. Is just what I believe this new component will be based on Fabien presentation.

Wednesday, January 28, 2009

Integrating Facebook Hive with Symfony

A symfony feature that has been really helpful while working on projects with it is the debugging capabilities of the framework. Almost every day I see my self going to the command line an doing a tail -f log/frontend_dev.log in my symfony project. This helps me see what is going on behind the scenes, spot bugs, find places where I can improve the code, etc. If something goes wrong there is always a developer on the team that shouts “check the symfony logs”, showing that they had became an essential tool for development. 


But not everything shines under the sun. Sometimes it happens that we would like to have a tool that allows us to filter the logs according to specific criteria. A tool that goes beyond a simple cat logs/frontend_dev.log | grep SELECT. We wanted to perform some analysis on the website usage, basically to help us improve it performance.


A colleague talked about Facebook Scribe, that was not actually related with this dream tool but later lead me to learn about the existence of the Apache Hive project (which was started by Facebook).


The Hive project allows to load a logs file and then filter it with SQL like commands. -Hive is more than this raw description, you can read more about it here.


My idea was to adapt the file logging format of symfony to make easy to import those files inside a Hive database. Because Hive support table partitioning by date, it should be easier to load the data from the logs and then perform the analysis with the SQL like syntax provided by Hive.


After some fights with Ant and Java 1.6 It was possible for me to get Hive running in my Mac. Then I just created a shameless copy of the sfFileLogger and renamed it to sfHiveLogger. I did small changes here and there and got it ready to log in a format that it’s easy to load later into Hive. I browsed through my testing symfony project to generate some logs and then I moved to the command line to start the fun with Hive.


There I created a table to hold the logs with the following command:


CREATE TABLE sflogs(

  logTime STRING, 

  priorityName STRING,

  message STRING

  COMMENT 'This is the sflogs table' 

  PARTITIONED BY(dt STRING) 

  ROW FORMAT DELIMITED

    FIELDS TERMINATED BY '\011' 

    LINES TERMINATED BY '\012';


Everything was working smoothly. The next step was to load the data from the logs file into the sflogs table to start issuing queries to it. I did it with this command:


LOAD DATA LOCAL INPATH '/path/to/myproject/log/frontend_dev.log.2009-01-28' INTO TABLE sflogs PARTITION (dt='2009-01-28');


With the data loaded I started to issue some commands against the Hive console like:


SELECT * FROM sflogs WHERE message LIKE "{sfRequest}%";


SELECT DISTINCT priorityname FROM sflogs;


SELECT COUNT(1) FROM sflogs;


The results were similar as when we work on a mysql client which was awesome. What amazed me the most was how with some easy changes it's possible to adapt symfony to our needs. But which were this changes? Here they are:

First we need to enable the sfHiveLogger in the logging.yml file of your symfony project under the sf_file_debug entry


Then do the shameless copy of the symfony class to your lib folder and rename it to sfHiveLogger.


Then change the code of the initialize method to look like this:


    if (!isset($options['file']))

    {

      throw new sfConfigurationException('File option is mandatory for a file logger');

    }


    $dir = dirname($options['file']);


    if (!is_dir($dir))

    {

      mkdir($dir, 0777, 1);

    }

    

    $logFileName = $options['file'] . '.' . date('Y-m-d');


    $fileExists = file_exists($logFileName);

    if (!is_writable($dir) || ($fileExists && !is_writable($logFileName)))

    {

      throw new sfFileException(sprintf('Unable to open the log file "%s" for writing', $logFileName));

    }


    $this->fp = fopen($logFileName, 'a');

    if (!$fileExists)

    {

      chmod($logFileName, 0666);

    }


Basically the change there is to append the current date in “Y-m-d-” format at the end of the logs file name. This will make easier to import the logs into Hive -only if you want them partitioned by date-.


Then on the log method change the line with:


$line = sprintf("%s %s [%s] %s%s", strftime('%b %d %H:%M:%S'), 'symfony', $priorityName, $message, DIRECTORY_SEPARATOR == '\\' ? "\r\n" : "\n");


to:


$line = sprintf("%s\t%s\t%s%s", strftime('%b %d %H:%M:%S'), $priorityName, $message, "\n");


Here what we do is to apply a little formating there. The most important part is to have the tab and then new line characters as delimiters because this was what we specified in the CREATE TABLE command above.


With this easy steps we can have a Hive enabled log file. If you want to learn more about Hive and the supported commands and the theory behind it please refer to the wiki.


Conclusion:


Besides that I’m still comparing Hive with other solutions to parse and analyze the logs, I think that this tool has a lot of potential to help debugging and profiling symfony applications. If we polish the log format and refine the table structure, then it’s shouldn’t be hard to setup some cronjobs that generate reports of the website usage, improving the usability of the symfony logs.