Integrating Facebook Hive with Symfony

A symfony feature that has been really helpful while working on projects with it is the debugging capabilities of the framework. Almost every day I see my self going to the command line an doing a tail -f log/frontend_dev.log in my symfony project. This helps me see what is going on behind the scenes, spot bugs, find places where I can improve the code, etc. If something goes wrong there is always a developer on the team that shouts “check the symfony logs”, showing that they had became an essential tool for development. 

But not everything shines under the sun. Sometimes it happens that we would like to have a tool that allows us to filter the logs according to specific criteria. A tool that goes beyond a simple cat logs/frontend_dev.log | grep SELECT. We wanted to perform some analysis on the website usage, basically to help us improve it performance.

A colleague talked about Facebook Scribe, that was not actually related with this dream tool but later lead me to learn about the existence of the Apache Hive project (which was started by Facebook).

The Hive project allows to load a logs file and then filter it with SQL like commands. -Hive is more than this raw description, you can read more about it here.

My idea was to adapt the file logging format of symfony to make easy to import those files inside a Hive database. Because Hive support table partitioning by date, it should be easier to load the data from the logs and then perform the analysis with the SQL like syntax provided by Hive.

After some fights with Ant and Java 1.6 It was possible for me to get Hive running in my Mac. Then I just created a shameless copy of the sfFileLogger and renamed it to sfHiveLogger. I did small changes here and there and got it ready to log in a format that it’s easy to load later into Hive. I browsed through my testing symfony project to generate some logs and then I moved to the command line to start the fun with Hive.

There I created a table to hold the logs with the following command:

CREATE TABLE sflogs(

  logTime STRING, 

  priorityName STRING,

  message STRING

) 

  COMMENT 'This is the sflogs table' 

  PARTITIONED BY(dt STRING) 

  ROW FORMAT DELIMITED

    FIELDS TERMINATED BY '\011' 

    LINES TERMINATED BY '\012';

Everything was working smoothly. The next step was to load the data from the logs file into the sflogs table to start issuing queries to it. I did it with this command:

LOAD DATA LOCAL INPATH '/path/to/myproject/log/frontend_dev.log.2009-01-28' INTO TABLE sflogs PARTITION (dt='2009-01-28');

With the data loaded I started to issue some commands against the Hive console like:

SELECT * FROM sflogs WHERE message LIKE "{sfRequest}%";

SELECT DISTINCT priorityname FROM sflogs;

SELECT COUNT(1) FROM sflogs;

The results were similar as when we work on a mysql client which was awesome. What amazed me the most was how with some easy changes it's possible to adapt symfony to our needs. But which were this changes? Here they are:

First we need to enable the sfHiveLogger in the logging.yml file of your symfony project under the sf_file_debug entry

Then do the shameless copy of the symfony class to your lib folder and rename it to sfHiveLogger.

Then change the code of the initialize method to look like this:

    if (!isset($options['file']))

    {

      throw new sfConfigurationException('File option is mandatory for a file logger');

    }

    $dir = dirname($options['file']);

    if (!is_dir($dir))

    {

      mkdir($dir, 0777, 1);

    }

    

    $logFileName = $options['file'] . '.' . date('Y-m-d');

    $fileExists = file_exists($logFileName);

    if (!is_writable($dir) || ($fileExists && !is_writable($logFileName)))

    {

      throw new sfFileException(sprintf('Unable to open the log file "%s" for writing', $logFileName));

    }

    $this->fp = fopen($logFileName, 'a');

    if (!$fileExists)

    {

      chmod($logFileName, 0666);

    }

Basically the change there is to append the current date in “Y-m-d-” format at the end of the logs file name. This will make easier to import the logs into Hive -only if you want them partitioned by date-.

Then on the log method change the line with:

$line = sprintf("%s %s [%s] %s%s", strftime('%b %d %H:%M:%S'), 'symfony', $priorityName, $message, DIRECTORY_SEPARATOR == '\\' ? "\r\n" : "\n");

to:

$line = sprintf("%s\t%s\t%s%s", strftime('%b %d %H:%M:%S'), $priorityName, $message, "\n");

Here what we do is to apply a little formating there. The most important part is to have the tab and then new line characters as delimiters because this was what we specified in the CREATE TABLE command above.

With this easy steps we can have a Hive enabled log file. If you want to learn more about Hive and the supported commands and the theory behind it please refer to the wiki.

Conclusion:

Besides that I’m still comparing Hive with other solutions to parse and analyze the logs, I think that this tool has a lot of potential to help debugging and profiling symfony applications. If we polish the log format and refine the table structure, then it’s shouldn’t be hard to setup some cronjobs that generate reports of the website usage, improving the usability of the symfony logs.

Custom Views in Symfony 1.0

After seeing this feature request for symfony 1.3 I decided to write a tutorial explaining how to use custom views with symfony 1.0. Yes symfony 1.0 has this feature, a little bit hide inside, but is there since the ol’ good days.

So, there are three ways to have a custom View class for your actions. To show how this work, we will need a project with a module called example. 

First example:

After you have your module ready add an action called FirstExample which will have this code: 

public function executeFirstExample() {}

Then on the example module add a folder called views. Inside it we will add our custom view class. The name will be firstExampleSuccessView. Create a firstExampleSuccessView.class.php file with the following code inside:

class firstExampleSuccessView extends sfPHPView

{

  

}

As you can see, it extends the sfPHPView provided by symfony,  to get all the functionality inside. Please note that custom views in symfony need to extend sfView, which is the parent class of sfPHPView. 

At this point I’m going to tell you that we can customize a lot of features of the symfony views. For the tutorial I will just show you how to have new shortcut variables inside the templates like the $sf_request or $sf_user already provided by symfony. Feel free to comment about your experience extending the sfView.

To add a new shortcut to the template we need to extend the following method:

sfPHPView::getGlobalVars(). We will add a new shortcut smartly called my_view_data. with a dummy array inside. This will be our code:

protected function getGlobalVars()

{

  $context = $this->getContext();

   $shortcuts = array(

     'sf_context' => $context,

     'sf_params'  => $context->getRequest()->

getParameterHolder(),

     'sf_request' => $context->getRequest(),

     'sf_user'    => $context->getUser(),

     'sf_view'    => $this,

     'my_view_data' => array('foo' => 'this data came from firstExampleSuccessView')

  );

 if (sfConfig::get('sf_use_flash'))

 {

   $sf_flash = new sfParameterHolder();

   $sf_flash->add($context->getUser()->

  getAttributeHolder()->

  getAll('symfony/flash'));

     $shortcuts['sf_flash'] = $sf_flash;

   }

    return $shortcuts;

}

As you can see there is a new my_view_data key with a one element array.

To see this in action we add in the example/templates folder a file called firstExampleSuccess.php with the following content:

<h1>First Example View:</h1>

<?php var_dump($my_view_data); ?>

Clear the cache and point your browser to http://yourapp.com/example/firstExample

There you will see the output of our new shortcut.

As long as our action returns sfView::SUCCESS it will use our new custom view class. Who said that symfony had no custom views? :-)

Remember to follow the naming convention of: <actionName><viewName>View. In our example we had firstExampleSuccessView. firstExample is the action, Success is on of the possible values that our actions can return, View is the default suffix. 

Second Example

In this case we will create a class called secondExampleView inside a file secondExampleView.class.php located in the lib folder of our application. The contents of this class will be almost the same that four our first view.  We will change the class name to secondExampleView and the my_view_data entry will be declared like this:

'my_view_data' => array('foo' => 'this data came from secondExampleView')

Then we create a new action for our module with the following code:

public function executeSecondExample(){}

Then we add the template secondExampleSuccess.php with this code inside:

<h1>Second Example View:</h1>

<?php var_dump($my_view_data); ?>

Now, how do we use our new view with this action? In case we don’t have one, we add a module.yml file in the config folder of the example module. There we add the following content:

all:

  view_class: secondExample

This will tell symfony to use our custom view class for this module. As you can see here,  we can specify different views per environment :-) Yes,  symfony is configurable :-) 

If we point our browsers to the secondExample action we will see the contests of our shortcut.

Third Example:

For this example we will create a class called thirdExampleView inside the lib folder of our application. As we did with our secondExampleView we will have the same code, except for the class name and the my_view_data shortcut that will have inside the following: 

array('foo' => 'this data came from thirdExampleView')

We add a template called thirdExampleSuccess.php with this content:

<h1>First Example View:</h1>

<?php var_dump($my_view_data); ?>

and we create an action like this:

public function executeThirdExample()

{

    $module = $this->getRequestParameter('module');

    $action = $this->getRequestParameter('action');

    $this->getRequest()->setAttribute($module.'_'.$action.'_view_name', 'thirdExample', 'symfony/action/view');

}

What we do here is setting in the 'symfony/action/view' namespace of the sfRequest attribute holder a value with our view class without the View suffix. The key of the attribute should be composed by <module_name>_<action_name>_view_name  as you can see from the code above.

Then we can point our browsers to the thirdExample action to see the results.

With this three ways we can set up your own views and customize the rendering process of our applications. 

As final note I want to add how symfony knows which view class to pick.

• It checks inside the module/views folder to see if it can find our custom class as explained on the first example.
  
• If there is no user class there, it will check if we have set up one in the sfRequest as explained on example three.
  
• Then it will try to see if we defined a view in the module.yml file.
  
• As last resort it will load the default sfPHPView class.
  

Thanks for reading this far and I hope this will help you on your projects.

Firebug Framework: CSS helper functions

In a previous post I started talking about the functionality that is already implemented in Firebug. While I was doing some reworking on FireSymfony I had the chance to use some of the CSS methods that are part of the Firebug lib.js source file:

hasClass = function(node, name)

This function expects a node, that for example can be obtained with Firebug "$()" function, and the CSS class name that we are searching for. Returns boolean as expected.

setClass = function(node, name)

Adds the CSS class provided by name to the node

removeClass = function(node, name)

Removes the class specified by name from node

toggleClass = function(elt, name)

If elt has class name then it will remove it from the element, otherwise it will add the class to the element.

setClassTimed = function(elt, name, context, timeout)

This will add the class name to the element. When the milliseconds timeout has expired, the class will be removed from the element and the timeout will be cleared. The timeout is added to the provided Firebug context

cancelClassTimed = function(elt, name, context) 

This method will remove a class added with the previous method and will clear the timeout.

All this methods can be accessed from inside our Firebug extensions if we declared them as explained here by Jan Odvarko. Another way is calling them like this: FBL.methodName(...)

There are more utilities inside the Firebug libraries that can help us while we develop our custom extensions, so we don't have to reinvent the wheel. I've plans of keep documenting the Firebug code, so stay tuned.